Speculative Decoding: Faster Tokens, Identical Answers

Key takeaway: Speculative decoding speeds up text generation by letting a small, fast "draft" model guess the next few tokens and having your big model verify all of them in a single forward pass — keeping the guesses it agrees with and correcting the first one it doesn't. The remarkable property is that the final output is mathematically identical to what the big model would have produced alone: same tokens, same distribution, no quality change — just delivered 2-3x faster. The decision this post helps you make is not whether to turn it on — if you serve your own models and latency matters, you almost certainly should — but understanding what it does and does not buy you: it changes when your tokens arrive, never which tokens, which means a faster answer grounded in stale context is just a wrong answer that now arrives sooner.

If your LLM feels slow and you serve it yourself, speculative decoding is one of the highest-leverage latency cuts available, and it has an unusually honest guarantee attached: it does not trade quality for speed. A large model generates text one token at a time, and each step is dominated not by arithmetic but by the time it takes to load the model's weights from memory — so the expensive hardware sits mostly idle, waiting. Speculative decoding fills that idle time by having a small model propose several tokens ahead, then verifying the whole batch at once, because checking a handful of tokens costs almost the same as generating one (NVIDIA, Introduction to Speculative Decoding).

So the "so what": speculative decoding is a rare free lunch — a 2-3x speedup with provably identical output, no retraining, and no quality risk — but it is a generation-speed technique, and generation speed is not where your worst answers come from. The original method demonstrated 2x-3x acceleration on a large model with outputs identical to standard decoding (Leviathan et al., ICML 2023). This post is the operator's map of that lever: what it actually does, why it is safe, where the speedup evaporates, and the one thing it leaves completely untouched.

Previous post: Quantization Is the Cheapest Way to Cut Your AI Inference Bill.

What this post covers

By the end, you should be able to decide whether to enable speculative decoding in your serving stack — and recognize the trap of speeding up a system whose real cost is bad context, not slow tokens.

• What speculative decoding really is — draft, verify, accept, correct, in plain terms.
• Why it is free, and why the output is identical — generation is memory-bound, and the verification step guarantees no quality change.
• Where the speedup disappears — acceptance rate, draft alignment, and a saturated GPU.
• A business-life example, and what it does not fix — the same assistant, faster tokens, same wrong answer, and the two clocks inside an AI system.
• A speculative-decoding readiness scorecard — a worksheet to decide if it will actually help you.
• Where Inherent fits — why the context layer, not the decoder, is the correctness you can't speed up.

Speculative decoding is "guess, then verify" for tokens

Start with the mechanism, because the guarantee follows from it. A big model generates tokens one at a time and slowly; speculative decoding pairs it with a small, fast "draft" model that proposes several tokens ahead, then the big model verifies all of those proposals in one pass — keeping every guess it agrees with and correcting the first one it doesn't. The draft model is cheap to run, so proposing is nearly free; the expensive model runs once per batch of guesses instead of once per token.

The technical explanation: autoregressive generation is sequential — token n+1 depends on token n — so you cannot normally parallelize it. Speculative decoding breaks that by using the draft model's guesses as a candidate future, which the target model can score all at once in a single forward pass. Hugging Face's implementation describes exactly this: the assistant proposes candidate tokens and the main model verifies them together in one forward pass (Hugging Face assisted decoding). If the draft guessed k tokens and the target agrees with the first three, you just produced three tokens for roughly the price of one, and you resume drafting from there.

The business explanation you can picture in a day: it is a senior editor working with a fast junior writer. The junior drafts the next sentence quickly; the editor reads the whole sentence at a glance and either signs off on it or stops at the first word they'd change and fixes it. When the junior is good, the editor approves long runs untouched and the pair moves several times faster than the editor writing every word alone — but the finished text is exactly what the editor would have written, because the editor checks everything.

Why it is free: generation is waiting, not computing

Here is the pattern that turns speculative decoding from a trick into a predictable win: single-token generation is bottlenecked by memory bandwidth, not compute — the GPU spends most of each step loading weights, not doing math — so it has spare compute to verify several tokens at almost no extra cost. You are not adding work; you are using capacity that was already being wasted.

• Generation is memory-bound. To produce one token, the hardware streams the entire model's weights through the chip. That data movement dominates the time; the actual arithmetic is small. This is why one token and several tokens cost nearly the same to verify — the weights only get loaded once.
• Drafting is cheap. The small model streams far fewer weights, so proposing candidate tokens adds little latency.
• The net effect is throughput for idle silicon. vLLM frames speculative decoding as a win precisely under memory-bound, medium-to-low-QPS conditions, where the GPU is not already saturated (vLLM speculative decoding docs).

Why the output is identical — and why that matters

The answer first: the verification step is a real check, not a rubber stamp, so any draft token the target model would not have produced is thrown away and corrected — which makes the final output provably identical in distribution to running the big model alone. This is the property that separates speculative decoding from every "cheaper but slightly worse" optimization: there is no quality dial to get wrong.

The original paper's central result is exactly this — it accelerates decoding "without any changes to the outputs" (Leviathan et al.). That has a sharp practical consequence: you never need to re-run your evals after enabling it. Quantization can shave a few points off hard queries and demands you measure; speculative decoding cannot, because the target model has the final say on every token. The only thing at risk is the speedup, never the answer.

Where the speedup disappears

Speculative decoding is safe but not universal. The gain is entirely a function of how often the target model accepts the draft's guesses, and a few conditions quietly kill acceptance:

• A poorly aligned draft model. If the small model's guesses rarely match the big model, the target rejects constantly, you pay for drafting and verifying, and you can end up slower. The draft must be a good, cheap approximation of the target — same tokenizer, similar behavior.
• A saturated GPU. Under high QPS, the hardware is no longer idle-waiting on memory; it is compute-bound serving many requests at once. vLLM's own guidance frames speculative decoding as a technique for reducing inter-token latency specifically under medium-to-low-QPS, memory-bound workloads (vLLM docs) — outside that regime, the extra draft work can reduce throughput instead of improving it.
• Highly unpredictable text. Creative, high-entropy generation is harder to guess than boilerplate or structured output, so acceptance — and speedup — falls.

The mitigations are well-trodden: use a proven draft strategy rather than a random small model. Self-speculative methods draft by skipping layers of the model itself (Draft & Verify, ACL 2024); Medusa adds extra decoding heads so no separate draft model is needed (Cai et al.); and n-gram / prompt-lookup drafting cheaply reuses tokens already in the prompt. All preserve the identical-output guarantee; they differ only in how good the guesses are.

The business-life example: faster tokens, same wrong answer

Picture a customer-support assistant. You enable speculative decoding and answers that took three seconds now stream back in one and a half. Users are happier, the interface feels alive, and you did not touch the product, the prompt, or the model's quality. That is a genuine win, and it is the right thing to do.

Now watch what did not change. The assistant is asked about your refund policy and confidently quotes last quarter's terms — because the help-center article was updated but the retrieval index still holds the old chunk. Speculative decoding delivered that stale answer faster. The expensive event — an escalation, a compliance question, an angry customer — was never about how quickly tokens streamed. It was about which context the model was handed. Speculative decoding optimizes the delivery of the answer and does nothing about whether the answer is right. That is not a reason to skip it. It is the reason it is necessary but nowhere near sufficient.

What speculative decoding does not fix

Here is the pattern that reorganizes how you think about inference: there are two different clocks in a production AI system — how fast the model emits tokens, and whether those tokens are grounded in correct, current, replayable context — and speculative decoding only moves the first one. Speed up the decoder all you like; you have not changed which documents retrieval pulled, whether they are fresh, whether the same query returns the same context twice, or whether you can explain afterward what the model was shown.

The relevance bridge, stated directly: because speculative decoding produces output identical to the base model, it can neither improve nor harm correctness — it faithfully speeds up whatever answer your context was going to produce, right or wrong. The failure modes that generate real business pain — an answer grounded in a deleted policy, an eval you can't reproduce because the index mutated underneath it, a disputed answer you cannot replay — all live in the context layer, which no generation-speed technique touches.

A speculative-decoding readiness scorecard

Use this to decide whether speculative decoding will actually help you — and to catch the case where you are speeding up the wrong clock. Each row is a decision; a row you cannot clear is a specific, named gap, not a vague "we should optimize inference."

The pattern the scorecard exposes: the top five rows are about generation speed — do you control serving, is latency the real pain, does your load profile fit, is the draft aligned, is the output truly identical. The bottom two rows are about context, and they are the ones a speculative-decoding project silently skips, because they are not what it is for. You can clear every speed row perfectly and still ship a system whose expensive failures come entirely from the two rows it doesn't cover.

Where Inherent fits

Only now, with the mechanism clear, does the product framing earn its place — and it lands on the bottom two rows of the scorecard, Context freshness and Replayability, because those are the rows a decoder speedup cannot touch and most teams therefore never close. Making tokens arrive faster is the easy, well-trodden win; keeping the context the model reasons over fresh, deterministic, and auditable is the half that decides whether your fast answers are also right.

That is precisely the layer Inherent provides: it sits above your vector storage and below your orchestration, making ingestion managed and retrieval governed rather than best-effort — on a different axis from the decoder entirely. The truth layer version-stamps and hashes every source at ingestion, so the context feeding your (now faster) model reflects the current state of the world, not a stale snapshot. The memory layer makes retrieval deterministic and version-pinned, so the same query returns the same context — the property that makes even a sped-up model's answers reproducible enough to trust. The audit layer issues a retrieval receipt per request — which sources, versions, and chunks produced the context — so when a fast answer is disputed, you can replay exactly what the model was shown.

To be clear about where we are: Inherent is early, and this is an architecture argument, not a claim that context infrastructure makes speculative decoding unnecessary. You should absolutely enable it — it is the best latency lever with no quality cost. The point is narrower and it holds: speeding up the decoder and correcting the context are two different jobs, and doing the first one perfectly does not do the second one at all.

The bottom line, and where to start

Speculative decoding is a rare free lunch: a small draft model guesses tokens, your big model verifies them in one pass, and you get 2-3x faster generation with output mathematically identical to the base model — no retraining, no quality risk, no evals to re-run. Turn it on if you control your serving stack, latency is your pain, and your GPU has idle memory-bound headroom to reclaim. But do not mistake faster tokens for a correct system: because the output is identical, a faster answer grounded in stale context is simply the same wrong answer, sooner.

Small task for today: take the single AI feature where users complain most about speed, and answer two questions. First — do you control its inference stack, and is speculative decoding already enabled (many serving frameworks support it in a config flag)? If not, that is likely a fast, safe latency cut sitting unclaimed. Second — of the last ten complaints about that feature, how many were about speed versus wrong or stale answers? If most were about correctness, you just learned that the clock speculative decoding speeds up isn't the one costing you. Fix the latency where it helps — then close the context axis: start with the Inherent Public API — get started in the docs. Sped up your stack and the wrong answers didn't budge? DM Flow on X with what broke — that's the gap we're building against.

Next read: Batch Ingestion Is Why Your RAG Answers Go Stale — the context axis no decoder speedup can touch.