Speculative decoding: How a small draft model makes a big AI model answer faster

Speculative decoding is a way to make a large language model produce text faster without changing what it says. A small, cheap “draft” model guesses the next few words, and the large model checks all the guesses in a single step. Correct guesses are kept, wrong ones are thrown away.

The idea is relevant to a recent story: Amazon added a faster Kimi variant to its platform, and Moonshot AI claims it outputs 2 to 3 times faster. Our report on Kimi K3.1 Pro Highspeed on Bedrock does not say how that speedup was achieved, so this guide covers a well-known technique for the same goal, not Moonshot’s method.

Why a chatbot types one word at a time

Think of a senior editor who must write an article word by word, and each word requires a long meeting. Most words are easy, like “the” or “of”, yet every one costs the same meeting.

Language models work like this. They generate text in small pieces called tokens, one at a time, and each token needs a full pass through the whole model. The 2022 paper that introduced speculative decoding describes it plainly: decoding K tokens from a large model normally takes K serial runs of that model.

That wait is the bottleneck, and it is largest for big models.

Letting an intern draft while the editor checks

Now give the editor an intern. The intern quickly drafts the next several words. The editor reads the whole draft at once and approves words from the start until the first one that looks wrong, then corrects that word and the process repeats.

In model terms:

1. The small draft model proposes a short run of tokens, for example a handful.
2. The large target model scores all of them in one parallel pass.
3. Tokens that match what the large model would have produced are accepted. At the first mismatch, the large model supplies its own token and the loop restarts from there.

Checking several tokens at once costs about as much as generating one. DeepMind’s authors, in their 2023 paper, report that scoring a short continuation in parallel takes about as long as sampling a single token from the large model. When the draft is often right, each expensive pass yields several tokens instead of one.

Why the answer does not get worse

A natural worry is that a weaker model’s guesses would lower quality. The method is designed to prevent that. Both published versions use a modified acceptance rule (the DeepMind paper calls it modified rejection sampling) so the output follows the large model’s own probability distribution. The Google authors state the method samples “without changing the distribution” and needs no retraining or architecture change, so it can speed up existing models.

This is the key difference from simply switching to a smaller model. The draft model only proposes. The large model always has the final say.

Where it came from

Two teams published the idea within months of each other:

• Google, 2022: Yaniv Leviathan, Matan Kalman and Yossi Matias posted “Fast Inference from Transformers via Speculative Decoding” in November 2022 (an ICML 2023 oral). On T5-XXL they report a 2X to 3X acceleration with identical outputs.
• DeepMind, 2023: Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre and John Jumper published “Accelerating Large Language Model Decoding with Speculative Sampling” in February 2023. They report roughly 2 to 2.5 times faster decoding on Chinchilla, a 70-billion-parameter model, in a distributed setup.

These figures come from the authors’ own experiments on those models. Gains on other models, hardware and tasks vary.

Limits and trade-offs

The speedup depends on how often the draft is right. Predictable text, such as boilerplate or code with repeated patterns, tends to benefit more. Hard, surprising text means more rejected guesses and less gain. This follows from how the method works, since each rejection wastes the drafted tokens.

There are also costs. The system must run a second model, which uses extra memory and adds engineering complexity. The draft model should be a good match for the large one, because a poor match is rejected more often. And the technique speeds up a single response; it is a different tool from batching many requests together, which improves overall throughput.

The first paper also notes the method rests on a simple observation: hard language-modeling tasks often contain easier subtasks that smaller models approximate well.

Why speed matters for agents, and what to watch

Speed is not only about impatience. In the Bedrock story, AWS argues that agent loops make many sequential calls, so a faster output rate shortens each step of the edit-test-fix cycle. Any technique that cuts the time per token compounds across dozens of steps.

When a provider advertises a faster version of a model, a few questions are worth asking:

• Does the output stay identical to the original model, or is it a retrained or modified model?
• Is the speed measured as tokens per second, and under what load?
• Does the claim come from the vendor or from independent tests?

In the Kimi case, the Bedrock announcement reported Moonshot’s 2 to 3 times figure but, per our reading, gave no tokens-per-second numbers or latency tests, and it describes the model as post-trained rather than naming a decoding method. That is why the claim is best treated as a vendor figure until independent measurements appear.

Speculative decoding is one of several ways to speed up inference. Its appeal is that, in the original papers, it offers a speedup with no quality trade and no change to the large model, which is rare in a field where speed usually costs accuracy.