Menu
DistillationConcepts

Knowledge distillation

Explainer5 min readIntermediateUpdated 1 Oct 2026
1 · In one line

Distillation trains a small student model to copy the full answer probabilities of a large teacher model, so you can run something smaller and cheaper.

1 · What it is

Distillation passes what a large model has learned into a smaller one. The large model is the teacher, and the small one is the student. The student is trained to give the same predictions as the teacher, as closely as it can. It ends up quicker and lighter, though usually a little less accurate.

The trick is in what the student copies. Picture a photo sorter shown a cat. A plain label says only “cat”. The teacher might instead say 90% cat, 8% dog and 2% fox. Those small numbers are a hint: this animal looks a bit like a dog and hardly like a fox. The teacher’s full output carries this extra information, which a plain label throws away. A setting called temperature spreads the numbers out, so the small ones are easier to learn from. The student then chases two goals at once: match the teacher’s softened numbers and get the true label right. You choose how much weight each goal gets.

Copying the final probabilities is not the only option. Some methods also push the student to imitate the teacher’s hidden states, the in-between numbers it works out on the way to an answer. The hope is that the teacher has found a better way to represent the input than the student would reach alone. Hidden-state copying has a catch, though. A student could hold the same knowledge with its numbers in a different order, so forcing them to line up has no firm theory behind it.

The idea predates today’s chatbots. Earlier work by Rich Caruana and colleagues showed that the knowledge in a whole group of models could be squeezed into one model that is simpler to run. In 2015, Geoffrey Hinton, Oriol Vinyals and Jeff Dean pushed the method further in a paper about distilling knowledge. They used it to make the speech model of a busy commercial product noticeably better. In 2019, DistilBERT used distillation while pre-training a smaller version of the language model BERT, rather than for one task only. It came out 40% smaller and 60% faster, yet held on to 97% of BERT’s language understanding.

Large language models are often distilled in a simpler way. The teacher writes answers to a big batch of prompts, and the student is fine-tuned, or further trained, on those answers. Amazon Bedrock offers this as a managed service: you supply the prompts, and it handles the teacher’s answers and the student’s training. OpenAI added its own version to its platform on 1 October 2024. It stores a large model’s answers as a dataset, trains a cheaper model on them and lets you test the result.

2 · Why it exists

The best models are often too big to run cheaply, and a small model trained on plain labels misses useful hints.

Costly answersLarge language models need a great deal of computing power and energy for every prediction they make.
Hard to deployA big model, or a group of models whose answers are averaged, can be too slow and expensive to offer to many users.
Thin labelsA plain right-or-wrong label carries less information than a score that says how likely each answer is.
3 · How it works

Follow one cat photo through a single training step.

A student model learns to match a teacher's soft probabilities A cat photo and its true label go to a frozen teacher and a small student. The teacher outputs cat 0.90, dog 0.08, fox 0.02. The student guesses cat 0.60, dog 0.25, fox 0.15. The true label is cat 1, dog 0, fox 0. The highlighted key step compares the student's guess with the soft targets and the true label. The combined loss updates only the student, and the step repeats. INPUT cat photo + true label Teacher large, frozen Student small, learning SOFT TARGETS cat0.90 dog0.08 fox0.02 TRUE LABEL cat 1 dog 0, fox 0 STUDENT GUESS cat0.60 dog0.25 fox0.15 KEY STEP Match the soft spread compare guess with targets and label one loss Update student only new weights, next example
The student is pushed toward the teacher's whole spread of probabilities, not only toward the right answer.
  1. 1 · teachA large teacher that has already been trained looks at a training example, and its weights stay frozen.
  2. 2 · softenA setting called temperature flattens the teacher's probabilities, so the small ones get a bigger share and become easier to see.
  3. 3 · matchThe student is scored on how close its probabilities are to the teacher's softened ones, and on whether it predicts the true label.
  4. 4 · updateOnly the student's weights change, and the loop repeats over many examples.
  5. 5 · deployThe finished student is put to work on its own, where it is cheaper to serve than the teacher.

The student learns how sure the teacher is about every answer, not just which answer won.

4 · Where it's used
WhoWhat they askWhat it works with
Moderation team“Can a small model score live comments for toxicity fast enough?”A large offline scorer labels the training comments first
Mobile developer“Can a language model like BERT run on a phone?”A distilled student such as DistilBERT
API customer“Can a cheaper model answer our support tickets as well as the big one?”Answers from a large model, used to fine-tune a smaller one
Speech team“Can one model replace a whole group of averaged models?”The group's combined knowledge, distilled into a single model
5 · What it solves, and what it doesn't
solves
  • The student is smaller, so it answers faster and uses less memory and energy.
  • Soft probabilities give the student a richer signal than labels that are only right or wrong.
  • DistilBERT cut BERT's size by 40% and ran 60% faster while keeping 97% of its language understanding.
  • A teacher can label extra examples, giving the student more training data than the teacher itself was trained on.
doesn't solve
  • The student's answers are usually a little worse than the teacher's.
  • Vendor tools tune the student for a specific task, so it should not be expected to match the teacher on everything else.
  • Distillation changes what the student knows, not how much computing each answer takes. Its own size sets that.
  • If the student is still too heavy, you need another method, such as quantization, to shrink it further.
6 · Go deeper
QuantizationPruningSmall language modelsFine-tuningSoftmax

Sources used

This explainer is written in original language. The links below support its factual claims.

  1. paperDistilling the Knowledge in a Neural Network, arXiv (Hinton, Vinyals and Dean) · read 1 Oct 2026
  2. paperDistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, arXiv (Sanh et al.) · read 1 Oct 2026
  3. docsKnowledge Distillation Tutorial, PyTorch · read 1 Oct 2026
  4. docsKnowledge Distillation, Keras · read 1 Oct 2026
  5. docsMachine Learning Glossary, Google for Developers · read 1 Oct 2026
  6. docsLLMs: Fine-tuning, distillation, and prompt engineering, Google for Developers · read 1 Oct 2026
  7. officialCustomize a model with distillation in Amazon Bedrock, Amazon Web Services · read 1 Oct 2026
  8. officialModel Distillation in the API, OpenAI · read 1 Oct 2026
Report an error