Model distillation: How a small AI learns from a bigger one, and why it is disputed

Model distillation is a way to train a small AI model by having it learn from a bigger, more capable one. The big model is the teacher and the small model is the student. The student copies the teacher’s answers, and often ends up far better than it would have been if it had trained alone.

The technique is back in the news because of a dispute over whether one lab copied another’s model this way. Distillation itself is routine and legitimate in most uses.

A student who studies the teacher’s worked answers

Picture a student preparing for an exam. One way to learn is to read the textbook and work out everything alone. Another is to study a top tutor’s worked solutions, including the tutor’s notes on why a wrong answer is tempting. The second route is usually faster.

Distillation is the second route. The teacher model answers many prompts. The student is then trained to produce similar answers. The student never needs the teacher’s internal numbers, only what the teacher says.

How the copying actually happens

A model trained from scratch learns from raw text, where each example has one correct next word. A distilled student learns from richer material: the teacher’s full set of preferences. If the teacher is torn between two answers, the student sees that doubt. That extra detail is a big part of why the method works.

There are two common ways to collect the teacher’s output. A lab with full access to its own teacher can read its probability scores directly. An outsider who only has a chatbot or an API (the programming interface developers use to send prompts and receive answers) can only collect the text of the replies and train on that.

Written-out reasoning is especially useful. If the teacher explains its steps, the student can be trained on those steps as well as on the final answer. The goal is a student that is smaller and cheaper to run, while keeping as much of the teacher’s skill as possible.

From ensembles of models to DistilBERT

The idea has older roots. In their 2015 paper “Distilling the Knowledge in a Neural Network”, Geoffrey Hinton, Oriol Vinyals and Jeff Dean described compressing an ensemble of models into a single model that is easier to deploy. An ensemble is a group of models whose answers are combined. It performs well but is costly to run. The authors credit earlier work by Caruana and collaborators with showing that such compression is possible. The paper was presented at the NIPS 2014 Deep Learning Workshop.

The authors reported surprising results on MNIST, a classic handwriting-digit test, and a significant improvement to the acoustic model of a heavily used commercial system, achieved by distilling an ensemble into one model.

In 2019, Victor Sanh, Lysandre Debut, Julien Chaumond and Thomas Wolf of Hugging Face applied the idea to language models with DistilBERT. They reported shrinking Google’s BERT model by 40%, making it about 60% faster, while keeping about 97% of its language understanding ability. The key point was that distillation could be done during pre-training, the first large training phase, and not only for one narrow task.

Who uses it today, and where the story comes in

Anthropic’s own description is blunt: distillation is training a less capable model on the outputs of a stronger one. It says labs use it legitimately on their own models, for example to produce smaller, cheaper versions for customers. That is the everyday use, and it is behind many of the compact models that run on phones or serve cheap API plans.

The same technique is now at the center of a political and legal fight. In February 2026, Anthropic published a post titled “Detecting and preventing distillation attacks”. It alleged that three labs, DeepSeek, Moonshot AI and MiniMax, generated over 16 million exchanges with its Claude model through about 24,000 fraudulent accounts. That, it said, violated its terms and regional access limits.

For DeepSeek, Anthropic alleged more than 150,000 exchanges. It said the traffic targeted reasoning and asked Claude to write out step-by-step reasoning for completed responses, which effectively creates training data. Anthropic also said the activity used rubric-based grading that made Claude act as a reward model for reinforcement learning. These are Anthropic’s claims. One critical commentary notes that the evidence was not published in a way outsiders can verify.

Our report on DeepSeek’s reported $15 billion round after a US federal ban notes that China denied claims about distillation. DeepSeek has not been quoted directly in what we read. DeepSeek’s own January 2025 R1 paper, published in Nature in 2025, states that the reasoning patterns of large models can be used to improve smaller ones. The abstract does not describe a method, so it does not settle the question.

Limits, trade-offs and the line that is hard to draw

A student rarely beats its teacher. Distillation passes along the teacher’s strengths and also its mistakes and blind spots. The student is usually a cheaper version, not a better one.

The technique also depends on quantity and quality. A few examples teach little, so large copying efforts need millions of answers. Anthropic’s post suggests that is why it looks for coordinated traffic across many accounts, using classifiers and behavioral fingerprinting, and for prompts that elicit chain-of-thought text.

The hardest issue is where legitimate use ends. Training on your own model’s outputs is ordinary engineering. Training on a competitor’s outputs, against its terms of service, is what Anthropic calls illicit. Whether it is also unlawful is a separate question that this article does not settle.

Defenses carry costs too. Anthropic’s own post says the answer is stronger verification of educational, security research and startup accounts, plus safeguards meant to reduce the usefulness of outputs for illicit distillation without degrading legitimate use. Secondary coverage reports that blocking sometimes hits legitimate requests.

What to watch next

Three things are worth following. First, whether any lab publishes evidence that outsiders can check, since most claims so far come from one side. Second, whether API providers tighten identity checks and change how much reasoning text they show. Third, whether regulators or courts treat distillation as a breach of contract, a trade issue or neither.

For readers, the practical point is simple. Distillation is how many small, fast and cheap models get good. It becomes controversial only when the teacher’s owner has not agreed to the lesson.