Distilling Intelligence via Synthetic Data

The Data Bottleneck

In 2026, model architecture is largely commoditized. Data is the new bottleneck. We hit a wall trying to train a 3B parameter model for a highly specific proprietary coding domain because we simply didn't have enough high-quality human-written data.

Synthetic Distillation Pipeline

The solution was to distill knowledge from a frontier model. We used a massive 100B+ parameter model via API to generate 100,000 synthetic problem-solution pairs.

To ensure quality, we implemented a dual-pass verification system. We used a separate Reward Model to score the outputs, filtering out the lowest quality 20%. We also ran static analyzers on the generated code to instantly drop any examples that threw syntax errors.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(api_key="sk-...")

async def generate_synthetic_pair(prompt_template):
    # Generate code using a frontier model
    response = await client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt_template}],
        temperature=0.7
    )
    return response.choices[0].message.content

We open-sourced a subset of this filtered dataset to Kaggle. Using this synthetic data to fine-tune our 3B model (using LoRA) resulted in a model that outperforms general-purpose models 10x its size on our specific internal benchmarks.