Distillation

It is March 2015. Competition-winning networks for image and speech tasks increasingly ship as ensembles — five, ten, sometimes more separately trained models whose predictions are averaged — and averaging ten models at inference time means paying for ten models at inference time. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean publish a short paper arguing that an ensemble’s knowledge can be transferred into one smaller network, not by handing it the ensemble’s final answers, but by handing it the ensemble’s uncertainty about every answer it considered. That idea outlives the ensembles it was built to compress, resurfacing a decade later as the mechanism that turns a closed frontier model’s chain of thought into an open checkpoint a fraction of its size.

Where We Were

By the early 2010s, winning a leaderboard and shipping a product had quietly become two different problems. AlexNet’s 2012 win on ImageNet (Chapter 2) used a single convolutional network, but the years that followed pushed accuracy further by training several large networks from different random initializations and averaging their predicted probabilities at test time. Ensembling reliably improved accuracy — different initializations and data orderings converge to functions that make different mistakes, so averaging cancels out some of that error — but it multiplied inference cost by the number of models in the ensemble, and each individual model in a competition-grade ensemble was already large enough to be expensive to run once. A speech recognizer or image classifier that needed ten forward passes per prediction to hit its reported accuracy was not something most products could afford to serve.

The training objective underneath all of this discarded a specific kind of information. A network trained with the standard cross-entropy loss against one-hot labels pushes the correct class toward probability 1 and every incorrect class toward probability 0, treating all wrong answers as equally wrong. But a well-trained model’s raw outputs, before that objective forces them to extremes, distinguish between a wrong answer that was nearly right and a wrong answer that was nowhere close — a model shown an image of a BMW assigns some non-trivial probability to “car” being confused with “truck,” and almost none to it being confused with “cat.” That relative structure among the wrong answers is not noise; it reflects distinctions the model has learned about how classes relate to each other. Hard 0/1 labels throw it away. Any single network trained directly on hard labels had to relearn those distinctions from scratch, rather than being handed them.

Three years of ensemble inference cost set up the question knowledge distillation answers: keep the accuracy, pay for only one model. PNG ↓

The Breakthrough

Soft targets and temperature. Hinton, Vinyals, and Dean’s proposal, which the field would come to call knowledge distillation, reframed the problem: instead of training a small network directly on hard labels, train it to reproduce the full output distribution of a larger “teacher” — a single big model or an ensemble — including the teacher’s low-probability guesses about wrong classes (Hinton et al., 2015). Read directly off a standard softmax, those low probabilities are so close to zero that they carry almost no usable signal; the paper’s fix was to raise the “temperature” of the softmax, a single scalar that flattens the distribution and makes the relative sizes of the small probabilities visible without changing which class is most likely. A compact “student” network is then trained to match this softened distribution — the soft targets — typically blended with an ordinary loss against the true hard labels, so the student both imitates the teacher’s relative confidences and stays anchored to ground truth.

A student learns the teacher's uncertainty, not just its answer; softened logits carry an opinion about near-miss classes that hard labels alone do not. PNG ↓

The paper’s results were modest in scale but clear in direction. On MNIST, a baseline network made 146 test errors and a large, heavily regularized network made 67; a much smaller network trained with soft targets from the large model, at a temperature of 20, made 74 errors — nearly matching the large model’s accuracy in a fraction of its size. On a commercial acoustic model, a single baseline network reached 58.9% frame accuracy and 10.9% word error rate; an ensemble of ten such networks improved that to 61.1% frame accuracy and 10.7% word error rate; a single network distilled from that ensemble reached 60.8% frame accuracy at the same 10.7% word error rate, which the authors reported captured more than 80% of the ensemble’s improvement over the baseline in a model with the ensemble’s deployment cost divided by ten. The paper also noted a detail that later implementations kept: because the soft-target loss’s gradients scale down with $1/T^{2}$ , using a higher temperature to soften the distribution shrinks that loss term’s contribution unless it is rescaled back up by multiplying by $T^{2}$ when combining it with the hard-label loss.

Task-agnostic distillation at pretraining time: DistilBERT. Hinton’s recipe distilled a classifier trained for one task. Four years later, Sanh, Debut, Chaumond, and Wolf applied the same teacher-student logic to a general-purpose language model itself, before any task-specific fine-tuning happened (Sanh et al., 2019). DistilBERT initialized a 6-layer student by taking every other layer from a pretrained BERT-base teacher — a 12-layer, 110-million-parameter encoder (Chapter 6, masked pretraining) — exploiting the fact that student and teacher shared the same hidden dimensionality, so no new layer had to be learned from a random start. Training then combined three losses rather than one: a distillation loss matching the student’s softened output distribution to the teacher’s, the same masked-language-modeling loss BERT itself was pretrained with, and a cosine-embedding loss that pulled the student’s hidden-state vectors into the same directions as the teacher’s. The result was a 66-million-parameter model, 40% smaller than its teacher, that ran 60% faster on CPU at batch size 1 and 71% faster on an iPhone 7 Plus for question answering, while retaining 97% of BERT-base’s GLUE macro score — 77.0 against 79.5, with individual task scores such as 86.9 against 89.0 on STS-B and 82.2 against 86.7 on MNLI. Because the distillation happened during pretraining rather than after fine-tuning, one DistilBERT checkpoint could be fine-tuned on any downstream task, the way the undistilled teacher could.

Distillation without logits: Orca. By 2023, the strongest available teachers were not open research checkpoints but closed APIs — ChatGPT and GPT-4 — that returned only generated text, never the internal probability distribution over the next token. Hinton’s original method assumed access to exactly that distribution: soft targets are the teacher’s softmax outputs, and there is no equivalent to distill from a model that only ever hands back a sampled string. Mukherjee, Mitra, Jawahar, Agarwal, Palangi, and Awadallah’s Orca answered this by moving the imitation target from probabilities to text (Mukherjee et al., 2023). Rather than training on a teacher’s answer alone, Orca prompted ChatGPT and GPT-4 with system instructions designed to elicit step-by-step explanations — not just “the answer is X” but the reasoning that got there — and trained a 13-billion-parameter LLaMA-based student on those explanation traces with an ordinary language-modeling loss, the student learning to generate the reasoning text itself rather than to match a distribution over it. The training set drew 5 million queries from the FLAN-v2 collection to collect ChatGPT responses, then collected GPT-4 responses for 1 million of those same queries. On Big-Bench Hard, Orca surpassed Vicuna-13B — an open 13-billion-parameter model of the same size trained by ordinary instruction tuning without explanation traces — by more than 100% and reached parity with ChatGPT; on AGIEval it improved on Vicuna-13B by 42%. On professional and academic exams such as the SAT, LSAT, GRE, and GMAT, Orca closed to within 4 points of ChatGPT with an optimized system message while still trailing GPT-4 substantially. The teacher’s reasoning process, not its logits, had become the thing worth distilling.

What a student imitates moves from probabilities to text, as each step removes an assumption about what access to the teacher it actually has. PNG ↓

The Mathematics

Hinton’s softened distribution is an ordinary softmax with one extra scalar dividing the logits before they are exponentiated:

q_i = \dfrac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

Symbol	Meaning
$z_i$	teacher’s raw logit for class $i$
$j$	index ranging over all classes, used to normalize the distribution
$T$	temperature; higher values flatten the distribution
$q_i$	softened probability for class $i$ at temperature $T$

The student’s training loss combines two cross-entropy terms — one against the true hard label, one against the teacher’s soft target — with the soft term rescaled to counteract the shrinking effect of a high temperature:

\mathcal{L}_{\mathrm{KD}} = (1-\alpha)\,\mathrm{H}\big(y,\ \sigma(z_s)\big) \;+\; \alpha\,T^{2}\,\mathrm{H}\Big(\sigma\big(\dfrac{z_t}{T}\big),\ \sigma\big(\dfrac{z_s}{T}\big)\Big)

$\mathrm{H}$ is cross-entropy and $\sigma$ is the softmax function. $z_s$ and $z_t$ are the student’s and teacher’s logits, $y$ is the true hard label, and $\alpha$ is a weight between zero and one balancing the two terms. The first term is an ordinary supervised loss: the student’s own softmax, at the standard temperature of 1, against the true label. The second term is the distillation loss: both student and teacher logits are passed through the same softened softmax at temperature $T$ , and the student is pushed to match the teacher’s softened distribution rather than the hard label. The $T^{2}$ factor undoes an effect of the temperature scaling itself — the gradients this second term produces shrink in proportion to $1/T^{2}$ as $T$ grows, so multiplying the term by $T^{2}$ keeps its contribution to training comparable to the hard-label term’s, regardless of how much the distribution was softened. What this buys you: a single objective that keeps a student honest against ground truth while still pulling it toward the teacher’s more informative, and more forgiving, view of the wrong answers.

Symbol	Meaning
$\mathrm{H}$	cross-entropy
$\sigma$	softmax function
$z_s, z_t$	student and teacher logits
$y$	true hard label
$\alpha$	weight balancing the hard-label and soft-target loss terms
$T$	temperature used for the soft-target term

DistilBERT extends the same two-term idea to a third signal, since a language model has intermediate hidden states a classifier’s output layer does not expose:

\mathcal{L}_{\mathrm{DistilBERT}} = \alpha_{\mathrm{ce}}\,\mathcal{L}_{\mathrm{ce}} + \alpha_{\mathrm{mlm}}\,\mathcal{L}_{\mathrm{mlm}} + \alpha_{\mathrm{cos}}\,\mathcal{L}_{\mathrm{cos}}

$\mathcal{L}_{\mathrm{ce}}$ is the distillation loss from the equation above, matching the student’s softened output distribution to BERT-base’s. $\mathcal{L}_{\mathrm{mlm}}$ is the ordinary masked-language-modeling loss (Chapter 6) computed against the true masked tokens, keeping the student’s own pretraining objective in the mix rather than relying on the teacher alone. $\mathcal{L}_{\mathrm{cos}}$ is a cosine-embedding loss pulling the student’s hidden-state vectors into the same directions as the teacher’s corresponding layers, a signal Hinton’s classifier-only setup had no equivalent for. Each $\alpha$ term is a fixed weight set before training. What this buys you: a student that inherits not just the teacher’s final answer shape but the internal representational structure that produced it, distilled once during pretraining so the resulting checkpoint is fine-tunable on any downstream task, not just the one the teacher was measured on.

Symbol	Meaning
$\mathcal{L}_{\mathrm{ce}}$	distillation loss against BERT-base’s softened output distribution
$\mathcal{L}_{\mathrm{mlm}}$	masked-language-modeling loss against true masked tokens
$\mathcal{L}_{\mathrm{cos}}$	cosine-embedding loss between student and teacher hidden states
$\alpha_{\mathrm{ce}}, \alpha_{\mathrm{mlm}}, \alpha_{\mathrm{cos}}$	fixed weights combining the three losses

Orca’s training signal has no equation of this kind, and that absence is itself the point: with a closed-API teacher, there are no logits $z_t$ to soften and no hidden states to align, so the objective collapses to an ordinary language-modeling loss over teacher-generated explanation text, with the teacher’s reasoning process rather than its output distribution supplying the signal a $\mathcal{L}_{\mathrm{ce}}$ term would otherwise carry.

What It Enabled

Each era’s distillation solved a different constraint on what a teacher could hand over and what form of imitation a student could learn from. None retired the one before it — logit-matching distillation is still how many task-specific classifiers are compressed, and pretraining-time distillation is still how general-purpose small encoders get built — but each opened a use case the earlier method could not reach.

Technique	Year	Teacher signal	Student outcome
Knowledge Distillation	2015	Ensemble or single-model logits, softened at temperature $T$	Single network transferred over 80% of a ten-model ensemble’s frame-accuracy gain on a speech task
DistilBERT	2019	BERT-base logits, hidden states, and MLM labels	66M-parameter, 6-layer model: 40% smaller, 60% faster on CPU, 97% of GLUE macro score (77.0 vs. 79.5)
Orca	2023	ChatGPT- and GPT-4-generated explanation text, no logits available	13B LLaMA-based model beat Vicuna-13B by over 100% on Big-Bench Hard, reached BBH parity with ChatGPT

Distillation sits alongside, and composes with, the other paths to a cheaper model this book covers: pruning and quantization (Chapter 15) shrink an already-trained model’s weights, while distillation trains a smaller model from scratch to imitate a larger one — a distilled student can then itself be quantized, and the two savings stack. However small a student’s parameter count gets, it still has to be served, which is where inference engines — the KV-cache and batching machinery downstream of this chapter — turn a compact checkpoint into low-latency throughput.

The word “distillation” also names a second, unrelated compression axis this book keeps separate on purpose: where this chapter reduces a model’s parameter count, few-step generation reduces a diffusion model’s number of sampling steps, distilling a slow multi-step denoiser into one that produces a comparable image in far fewer steps. Both borrow the teacher-student framing; neither reduces to the other.

Orca’s move from matching logits to matching generated text put distillation on the same footing as a closely related practice: training a model on a stronger model’s outputs is also how Self-Instruct and Alpaca built cheap instruction-following data from a single teacher API, without any notion of temperature or soft targets at all — explanation tuning added the requirement that the imitated text show its reasoning, not just its answer. That same output-imitation logic, applied to reasoning traces at much larger scale, resurfaces in DeepSeek-R1: alongside its main reinforcement-learning-trained reasoner, the DeepSeek team distilled the same reasoning behavior into dense Qwen and Llama checkpoints from 1.5B to 70B parameters, with a 32B Qwen distillation reaching 72.6% on AIME 2024 and a 70B Llama distillation reaching 70.0%, both well above what those checkpoints scored before distillation. Ten years after Hinton’s ensemble of speech models, the teacher had become a 671-billion-parameter reasoning model and the signal being distilled was a chain of thought, but the underlying move — a smaller network trained to imitate a larger one’s output rather than raw ground truth — was the same one.

Key Papers

Paper	Authors	Year	Link	Why it matters
Distilling the Knowledge in a Neural Network	Hinton et al.	2015	arXiv:1503.02531	Introduces soft targets, temperature, and the teacher-student framing that every later distillation method builds on
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	Sanh et al.	2019	arXiv:1910.01108	Extends distillation to task-agnostic pretraining with a triple loss, producing a general-purpose compressed transformer
Orca: Progressive Learning from Complex Explanation Traces of GPT-4	Mukherjee et al.	2023	arXiv:2306.02707	Distills a closed API’s reasoning by imitating generated explanation text, with no teacher logits available

PM Lens

Distillation is the compression lever I reach for when the constraint is a serving budget rather than a training budget: DistilBERT’s 60% CPU speedup and 97% retained accuracy are the kind of trade a latency-sensitive product feature can actually use, where a 3% quality gap is cheaper than the GPU fleet a full BERT-base deployment would need. Orca’s variant matters for a more mundane reason — most teams distilling today are working from an API, not a checkpoint, so there are no logits to soften even if they wanted them, and training on a stronger model’s explanations is often the only lever available. That same logic is now a build-versus-buy line item: DeepSeek’s distilled 32B and 70B checkpoints outperform their undistilled base models by enough that “self-host a distilled reasoner” competes directly with “call a frontier API,” and the choice comes down to a per-request cost comparison I re-run every time a usage estimate changes materially.

Search