Speculative Decoding

Picture a magazine newsroom at deadline. The senior editor — brilliant, slow, thorough — can polish one sentence in the time the cub intern can draft four. So you pair them up. The intern races ahead and types out a guess at the next four sentences. The editor reads all four at once, nods at the ones they would have written themselves, and strikes through the first sentence that sounds wrong. That paragraph ships. Everything after the strike is thrown away, the intern starts again, and the editor moves on — having read four sentences in the time it used to take them to write one.

That's the whole lesson. The editor is a big autoregressive decoding language model — Llama-70B, say. The intern is a small cheap model — Llama-7B, or a 160M-parameter toy — doing the same job at a fraction of the cost. The draft is the intern's guess at the next K tokens. The approval is a single parallel forward pass of the editor. And the magic — the part that took a DeepMind paper to nail down — is that the tokens you ship under this scheme are indistinguishable from what the editor would have typed alone. Not approximate. Not “good enough.” Identical in distribution.

Here is the most uncomfortable number in modern ML deployment. When Llama-70B generates a single token on an H100, the GPU spends roughly 97% of the time reading weights out of HBM and about 3% actually doing arithmetic on them. You bought an $30,000 matrix-multiply machine and you're using it as a glorified memory-copy engine.

The reason is mechanical. Autoregressive decoding generates tokens one at a time. To produce token t+1 you need the entire model weights — all 140 GB of them — streamed from HBM into the SMs. You then do a batch-of-one forward pass, which touches those weights once and throws them out. The arithmetic is trivial. The memory read is everything. Inference is memory-bandwidth-bound, not compute-bound.

Which means: if you could somehow do useful work on more tokens per weight-read, you'd be getting that work for free. The compute is sitting there idle anyway. This lesson is about speculative decoding — a beautifully cheeky trick from Leviathan and Chen (both 2023) that turns idle compute into a 2-3x wall-clock speedup, with zero quality loss, and it's become the default in every serious inference stack.

Speculative decoding's answer is: guess. Keep a small, cheap draft model — the intern — next to the big one — the editor. Every step, let the intern cheaply draft K tokens autoregressively (say K = 4). Then run the big target model — the editor — once on all K tokens in parallel. That single forward pass gives you the editor's probability distribution at every position simultaneously. Compare. Approve the draft tokens the editor would also have produced. Reject and resample at the first disagreement.

The numbers are what make this work. Running the editor on one token: one weight read, one token produced. Running the editor on K tokens: one weight read, up to K tokens approved. If the intern's guesses agree with the editor ~70% of the time, you average ~3 approved tokens per editor pass — and the editor's pass cost barely budged because it was memory-bound all along.

Back to the newsroom for a second. The part that surprises people — even people who write inference code for a living — is that the editor reads all four intern-drafted sentences simultaneously. Not sentence one, then sentence two, then sentence three, then sentence four. All four, at once, in a single pass of the eyes. That's only possible because reading is cheaper than writing. Writing one sentence forces the editor to produce a softmax over the vocabulary, pick a token, condition on it, do it again. Reading four drafted sentences just asks: “for each of these positions, what probability would I have assigned to this token?” That's one parallel forward pass — the same shape a training batch has. The editor is getting four answers for the price of one weight read.

Watch the animation. The intern (small, fast) sprints ahead and drafts four candidate tokens. The editor (large, slow) takes those four tokens and runs one forward pass that produces predictions at all four positions in parallel — the same way a training batch works. If the first three match, we approve them. The fourth disagrees, so we reject it, resample that position from the editor, and discard anything after. Net gain: three tokens produced in the time it would have taken to produce one, plus a small intern-model tax.

How much speedup does this actually give you? It depends on the per-token acceptance probability α. If each draft token is approved independently with probability α, and the intern guesses K tokens, the expected number of approved tokens per verification step is a simple geometric-series calculation.

Translate that to wall-clock speedup. Let c be the cost ratio intern-over-editor (say c = 0.1 for a 10x smaller intern). Each speculative step costs K·c + 1 editor-equivalents and produces on average E[accepted] tokens. The ratio is your speedup over plain autoregressive decoding:

Slide the acceptance rate up and down. Two things to notice. First, the curve is very sensitive to α in the 0.5-0.9 range — an intern that goes from mediocre to good doesn't linearly improve your throughput, it improves it dramatically. Second, cranking K past 4 or 5 rarely helps: once the geometric series has mostly converged you're just paying more draft cost for diminishing marginal approved tokens. The sweet spot for most deployments is K ∈ [3, 7].

Now the part that sounds too good to be true. Every sentence you just read is about speed. What about correctness? An intern who guesses wrong 30% of the time is shipping drafts into your output stream. How is the final text the same as what the editor would have written alone?

Go back to the acceptance rule. Look at it as an editor would. Two cases. If the editor's own probability for the drafted token p(x) is already at least as high as the intern's probability q(x), the editor approves unconditionally — of course they would have written that. If the editor's probability is lower than the intern's, the editor flips a biased coin with probability p(x) / q(x): approve sometimes, reject the rest of the time. And on the first rejection, the editor resamples from the leftover mass max(0, p − q), normalized. Two lines of algebra on the cases give you the miracle:

That last line is the whole ballgame. The probability that speculative decoding emits token x is exactly p(x) — the same as sampling from the editor directly. The intern's distribution q vanishes from the final answer. It only ever affected speed, never the output. If you would have sampled the string "the cat sat" from the editor at temperature 0.7, you get the same string with the same probability under speculative decoding. An intern who guesses well makes you faster; an intern who guesses badly makes you barely faster than the editor alone. Neither one ever makes you wrong. That's why this isn't a quality/speed tradeoff, and that's why every production inference stack shipped it the moment the paper landed.

Three layers, as always. Pure Python on a toy bigram editor to make the acceptance rule itself totally concrete. NumPy to verify the distribution is preserved by sampling a million times. PyTorch with HuggingFace's assistant_model argument, which is how you'd actually ship this.

Now the payoff. Run the above many times, collect the output distribution, and compare it to a million draws from the editor directly. They match. This is the correctness guarantee — speculative decoding's entire reason for being welcomed into production stacks.

PyTorch / HuggingFace has shipped this since transformers 4.30. Hand it an assistant_model — the intern — and generate() does the rest: draft rollout, editor verification, KV-cache splicing, early-exit on the first rejection. You never touch the acceptance math; what you do touch is the model-pair selection and the tokenizer compatibility check.

You will not implement any of these from scratch in 2026. vLLM, TGI (Hugging Face's Text Generation Inference), TensorRT-LLM, and ExLlamaV2 all support some mix of vanilla speculative decoding, Medusa, Eagle, and lookahead. Your job as a deployment engineer is choosing which one, measuring acceptance on your traffic distribution, and sizing the intern model correctly.

What to carry forward. LLM inference is memory-bandwidth-bound, so any trick that increases useful work per weight-read is essentially free speed. Speculative decoding is the cleanest example: the intern drafts K tokens, the editor approves or rejects all K in one parallel verification pass, and a careful acceptance rule preserves the editor's output distribution exactly. Typical speedup is 2-3x on real workloads with no quality loss. Medusa and Eagle push this further by making the intern a head of the editor itself. Every production inference stack (vLLM, TGI, TensorRT-LLM) supports this — you will configure it, not implement it.

Next up — Continuous Batching. The intern/editor trick fills idle compute inside one request. But a real production server isn't handling one request — it's handling dozens at once, each at a different point in its own decode. Naive batching forces the whole group to wait for the slowest conversation to finish before starting a new one. Continuous batching is the serving-system trick that slots new requests into an ongoing batch at every step, and when you combine it with speculative decoding you unlock the throughput numbers that make LLM APIs economically viable. That's the lesson after this one.