If you've ever wondered how a tiny model that runs for fractions of a cent can answer almost as well as a giant frontier model, the answer is usually one word: distillation. It's one of the most important — and least understood — techniques in modern AI, and it's the reason cheap models like DeepSeek V4 Flash, GLM 4.7 Flash, and GPT-5.4 Mini exist at all.
The One-Paragraph Version
Model distillation is a training technique where a small "student" model learns to imitate a large "teacher" model. Instead of learning only from raw data, the student learns from the teacher's outputs — including the teacher's confidence levels and reasoning patterns. The result: a model 10–100x cheaper to run that captures much of the teacher's capability.
Why Not Just Train a Small Model Normally?
You can — people do. But small models trained only on raw data hit a wall. They lack the subtle judgment that emerges from scale: knowing when to say "I'm not sure," how to structure a complex answer, or which of two plausible answers is actually correct.
The teacher model has already paid the "learning tax" — hundreds of millions of dollars of compute went into building its judgment. Distillation is a way to copy that judgment into a smaller container.
How It Actually Works
A simplified distillation pipeline looks like this:
- Generate teacher outputs. Run millions of prompts through the large teacher model, capturing not just its answers but its full probability distribution over every token.
- Train on soft targets. The student learns to match those distributions, not just the final answers. The "soft" probabilities carry extra information — e.g., the teacher was 70% sure the answer is "Paris" but assigned 20% to "Lyon", which tells the student something about the question's ambiguity.
- Filter for quality. Low-confidence or wrong teacher outputs get filtered or corrected, often with human review.
- Iterate. Repeat with harder prompts where the student still fails.
# Conceptual distillation loss (not runnable code)
teacher_probs = teacher(prompt) # full distribution
student_probs = student(prompt) # full distribution
loss = KL_divergence(student_probs, teacher_probs)The key trick is the KL divergence on full distributions — called "soft targets" — rather than plain right/wrong labels ("hard targets"). Soft targets carry richer signal per training example.
Real-World Examples You Use Every Day
Nearly every "mini", "flash", or "nano" model on the market is distilled:
- GPT-5.4 Mini and Nano — distilled from larger GPT-5 family models
- Gemini 3.x Flash — Google's fast tier, distilled from Gemini Pro-class teachers
- DeepSeek V4 Flash — tops global token usage charts at around $0.03/M input tokens
- GLM 4.7 Flash — Zhipu's budget tier at roughly $0.005/M input on Qubax
The pattern is consistent across the industry: flagship model first, then a distilled fast tier that captures 85–95% of the capability at 1–5% of the cost.
Distillation vs. Fine-Tuning vs. Quantization
These three get confused constantly. They solve different problems:
| Technique | What changes | Use case |
|---|---|---|
| Distillation | Trains a smaller model to imitate a bigger one | Making deployment cheaper |
| Fine-tuning | Adapts an existing model to your data | Specializing behavior/domain |
| Quantization | Compresses weights to lower precision | Faster inference, same architecture |
Distillation changes what the model knows (by copying a teacher). Fine-tuning changes what the model is for. Quantization changes how it's stored. Big labs often combine all three: distill, then fine-tune, then quantize for serving.
The Limitations
Distillation is powerful but not magic:
- The student can't exceed the teacher. Distilled models inherit their teacher's blind spots and knowledge cutoff. (Though some research shows students can generalize slightly better on narrow tasks.)
- Capability loss at the tail. Easy and medium tasks transfer well; the hardest 5% of reasoning often doesn't survive the shrink.
- Licensing matters. Many commercial model terms restrict using outputs to train competing models. Open-weight models (like those on Hugging Face — now owned by Nvidia) are the safe foundation for distillation work.
- Reasoning models are harder. Copying a chain-of-thought reasoning process is harder than copying final answers, though newer techniques handle it.
Why Distillation Explains Today's AI Prices
The price collapse at the small end of the market is a direct distillation story. DeepSeek V4.1 Flash outperforms its own flagship on some benchmarks while costing a fraction as much, because open-weight ecosystems let anyone distill frontier capability into cheap architectures. When the teacher's knowledge is one API call away, every lab can produce a competent fast tier — and competition does the rest.
For you as a developer, the practical takeaway is simple: default to the smallest model that passes your evals. The gap between a frontier model and a good distilled one is smaller than ever, and the price gap is still enormous.
Want to compare distilled models side by side? Qubax lists pricing for every mini, flash, and nano model in one place — see qubax.ai/models, or start with the API docs at qubax.ai/docs.
FAQ
Is distillation the same as copying a model?
No. Copying duplicates the weights. Distillation trains a new, smaller model to imitate the teacher's behavior — the architectures are usually completely different.
Can I distill a model myself?
Yes, using open-weight teachers and public distillation frameworks. The main costs are teacher inference for generating training data and student training compute.
Do distilled models hallucinate less?
Usually slightly less on common tasks, because the teacher's outputs are cleaner than raw internet data. But they inherit the teacher's failure modes, so don't count on it.
Which cheap distilled model should I start with?
On Qubax, DeepSeek V4 Flash, GLM 4.7 Flash, and GPT-5.4 Mini are all strong starting points under $0.05/M input tokens — benchmark them on your own tasks via qubax.ai/models.
