Prefill and Decode: Why One Model Has Two Bottlenecks

Key takeaway: Every LLM answer is produced in two phases that stress the GPU in opposite ways. Prefill reads your whole prompt in one parallel pass to produce the first token — it is compute-bound, limited by how fast the chip can do math, and it sets your time to first token. Decode then generates the rest one token at a time, each pass re-reading the model's weights — it is memory-bandwidth-bound, limited by how fast data moves, and it sets your inter-token latency. The decision this post helps you make: when a latency complaint lands, know which phase is the culprit, because the fixes are different — and understand that both fixes are serving-layer wins that do nothing about whether the answer is correct.

If your LLM feels slow, "slow" is two different problems wearing one label. A long pause before anything appears is a prefill problem. A reply that streams out in visible, stuttering chunks is a decode problem. They live in different phases of the same request, hit different hardware limits, and are fixed by different techniques — so a team that optimizes the wrong one spends money and moves nothing.

So the "so what": the single most useful mental model for LLM serving is that one model runs two workloads — a compute-bound prefill and a memory-bound decode — and most production latency pain comes from making one GPU do both at once. Naming the phase turns a vague "it's slow" into a specific, fixable bottleneck. This post is the operator's map of that split: what each phase is, why their bottlenecks are opposite, where they collide, and the two standard fixes.

Previous post: Continuous Batching Is How One GPU Serves a Crowd — where the prefill/decode collision first showed up.

What this post covers

By the end, you should be able to look at a latency complaint and say which phase owns it — and recognize that speeding up either phase changes when answers arrive, never whether they are right.

• What prefill and decode actually are — the two phases of every generation, in plain terms.
• Why their bottlenecks are opposite — compute-bound math versus memory-bound weight loading.
• Why running both on one GPU spikes latency — prefill/decode interference, and the two metrics it hits.
• A business-life example and a fix scorecard — a support assistant, chunked prefill, and disaggregation.
• Where Inherent fits — why a faster first token is not a correct one.

Prefill and decode are two different computations

Start with the mechanism, because the whole post follows from it. Generating text with a transformer happens in two stages: first the model ingests the entire prompt in a single parallel forward pass — the prefill — to emit the first output token; then it generates every subsequent token one at a time — the decode — each step feeding the previous token back in. Same model, same weights, two very different shapes of work.

The technical explanation:

• Prefill processes all N prompt tokens together. Because they are all present up front, the GPU can compute their attention and feed-forward math in parallel, in one big matrix-heavy pass. This is where the KV-cache (the stored keys and values the model reuses later) gets built. Prefill produces exactly one token — the first — and its cost grows with prompt length.
• Decode is autoregressive: token t+1 depends on token t, so the passes are strictly sequential and there is no way to parallelize within a single request. Each pass does a small amount of math but must re-read the model's weights and the growing KV-cache from memory. Decode runs once per output token, so its cost grows with response length.

The business explanation you can picture in a day: prefill is reading the whole question, decode is writing the answer one word at a time. Reading can be skimmed all at once; writing has to happen in order, word after word, and each word requires flipping back through everything already on the page. That asymmetry — read-in-parallel, write-in-sequence — is the entire reason the two phases behave so differently on hardware.

The terms come straight from the serving literature that now splits the two on purpose: DistServe (OSDI 2024) and Splitwise (ISCA 2024) both formalize the request as a prefill phase and a decode phase with distinct performance profiles (DistServe, Splitwise).

Why the two phases hit opposite bottlenecks

The answer first: prefill is compute-bound and decode is memory-bandwidth-bound, so they are limited by different parts of the GPU — and a chip tuned to keep one phase busy is, by construction, underused by the other. This is not a tuning detail; it is why a single GPU can look 90% utilized one millisecond and nearly idle the next.

• Prefill saturates the math units. Processing many prompt tokens at once is a large, dense matrix computation — exactly what a GPU's tensor cores are built for. During a prefill burst, compute utilization is high; the chip is doing real math.
• Decode starves the math units. Generating one token means a tiny amount of math wrapped around a large, unavoidable cost: reading the full model weights (and the KV-cache) out of memory for every single token. The bottleneck is memory bandwidth, not arithmetic, so the tensor cores sit mostly idle waiting for data (DistServe).
• The two limits don't average out. Because prefill wants compute and decode wants bandwidth, no single hardware configuration is optimal for both. Optimize the GPU for throughput-heavy prefill and your decode is bandwidth-starved; size it for decode and your prefill compute is wasted.

The practical read: "GPU utilization" is a misleading single number here. A serving GPU alternating between the two phases is oscillating between compute-bound and memory-bound regimes, and the average hides that neither phase is getting a chip shaped for it.

Why running both on one GPU spikes your latency

Here is where the abstraction earns its keep, because it explains a specific production symptom: when prefill and decode share one GPU, a newly arrived request's big prefill pass blocks the token-by-token decodes already in flight — so admitting new users makes existing users' streams stutter. This is prefill/decode interference, and it shows up as the two latency metrics moving against each other.

• Time to first token (TTFT) is dominated by prefill. A long prompt, or a queue of prefills ahead of you, means a long pause before the first token appears.
• Inter-token latency (TPOT / time per output token) is dominated by decode. Every time a heavy prefill jumps the queue, the in-progress decodes wait, and users watching a live stream see it hitch (Sarathi-Serve).
• You cannot optimize both by "adding a bigger GPU." The phases compete for the same device at the same instant. More raw power raises the ceiling but does not stop prefill from stalling decode; the conflict is structural.

The reason the last three serving posts kept mentioning "prefill" in passing is that this collision is the hidden tax under continuous batching: batching packs the GPU efficiently, but the moment it mixes a fresh prefill into a decoding batch, the interference reappears. Naming the phases is what lets you target the fix instead of guessing.

The business-life example: the pause and the stutter

Picture a support assistant at peak. A customer pastes a 2,000-word error log and asks what went wrong. That long prompt is a heavy prefill — the model must read all 2,000 words before it can say a single thing, so the customer stares at a blank box for a beat. That pause is TTFT, and it is a prefill problem; a faster memory bus would not fix it, more compute would.

Now the answer starts streaming — but it arrives in visible stutters, a few words, a hang, a few more. That is decode, and the stutter is inter-token latency. It got worse the instant three more customers hit "send" with their own long logs, because each new prefill elbowed ahead of the decode already streaming. Same feature, same GPU, two completely different slownesses — and a team that "optimized latency" without separating them would have tuned one and left the other untouched.

Now watch what neither phase touched. That first customer's error log was real, but the assistant answered it using a troubleshooting article that was deprecated last month — the fast first token was a confidently wrong first token. Prefill and decode decide when the words arrive. They have nothing to say about whether the words are right.

A prefill/decode fix scorecard

Use this to route a latency complaint to the phase that owns it — and to catch the case where the real cost is not latency at all. Each row is a decision; a row you cannot clear is a specific, named gap.

The two standard fixes both live in the top four rows. Chunked prefill splits a big prefill into small pieces and interleaves them with ongoing decodes on the same GPU, so no single prefill monopolizes the device — the single-node mitigation (Sarathi-Serve). Disaggregation goes further: it runs prefill and decode on separate GPU pools, each sized for its own bottleneck, and hands the KV-cache from one to the other. DistServe reported this can serve 7.4× more requests or hold a 12.6× tighter latency target than a colocated baseline within its SLOs (DistServe); Splitwise built the same split into "prompt" and "token" machines (Splitwise).

The bottom two rows are about context, and a phase-scheduling project silently skips them — because they are not what it is for.

Where Inherent fits

Only now, with the two phases clear, does the product framing earn its place — and it lands on the bottom two rows of the scorecard, Context freshness and Replayability, because those are the rows no phase-scheduling fix can reach. Splitting prefill and decode is a genuine, well-trodden serving win: it lowers your time to first token and steadies your stream. But both phases operate on whatever context they were handed, and neither one checks it.

That is the layer Inherent provides: it sits above your vector storage and below your orchestration, on a different axis from the phase scheduler entirely. The truth layer version-stamps and hashes every source at ingestion, so the prompt your prefill reads reflects the current state of the world, not last month's article. The memory layer makes retrieval deterministic and version-pinned, so the same query returns the same context — the property that keeps answers reproducible even as you scale the serving underneath them. The audit layer issues a retrieval receipt per request — which sources, versions, and chunks produced the context — so when a fast, smooth answer turns out to be wrong, you can replay exactly what the model was shown.

To be clear about where we are: Inherent is early, and this is an architecture argument, not a claim that context infrastructure replaces good serving. You should absolutely split your phases where the load justifies it. The point is narrower and it holds: making the first token arrive faster and the stream run smoother are serving-layer jobs, and doing them perfectly does nothing to make the answer correct.

The bottom line, and where to start

One model runs two workloads. Prefill reads the whole prompt in a compute-bound parallel pass and sets your time to first token; decode writes the answer one memory-bound token at a time and sets your inter-token latency. Most production latency pain is the two fighting over one GPU — fixed by chunked prefill on a single node or by disaggregating the phases onto separate pools at scale. None of it changes whether the context was right.

Small task for today: for the AI feature drawing the most complaints, split "slow" into two numbers — the pause before the first token (TTFT, a prefill problem) and the gap between tokens under load (inter-token latency, a decode problem). Fixing the wrong one is a common, expensive mistake. Then ask a second question: of the last ten costly incidents on that feature, how many were about speed versus a wrong or stale answer? If most were about correctness, you just learned that the phase you were about to optimize isn't where the cost lives. Tune the phase that hurts — then close the context axis: start with the Inherent Public API — get started in the docs. Split your phases and the wrong answers didn't budge? DM Flow on X with what broke — that's the gap we're building against.

Next read: The KV Cache Is Your Concurrency Ceiling — what fills the memory that makes decode the slow phase, and how many users fit on the card once it does.