Inference Engineering: TTFT, TPOT, and Two Clocks

Key takeaway: LLM inference has two latency clocks: TTFT (how long until the first token arrives) and TPOT (how long between subsequent tokens). They are driven by different bottlenecks and respond to different optimizations. Conflating them leads to expensive infra changes that move the metric that was never your actual user complaint. The decision this post helps you make: which clock is costing you, what moves it, and what neither clock can touch.

Your users feel two separate things when they talk to an LLM. First, a pause: the wait before anything appears. Then, a stream: the rate at which text arrives. Those are not the same latency, and the hardware is not the same bottleneck. Treating them as one metric is how teams end up adding expensive GPU capacity and finding that the user complaint didn't budge.

TTFT and TPOT are the names for those two things. Inference engineering (the discipline of shipping LLMs fast and cheaply) starts with knowing which clock is broken.

Previous post: Speculative Decoding: Faster Tokens, Identical Answers.

What this post covers

By the end you should be able to: (a) name the bottleneck behind each metric, (b) match the right optimization to the right complaint, and (c) recognize the one thing neither metric measures.

• What TTFT and TPOT are: first principles, where they come from, how they fit into the LLM serving stack.
• Why they have different bottlenecks: prefill is compute-bound; decode is memory-bound.
• What moves each one: the levers and their real-world tradeoffs.
• A workload-to-optimization map: when to go after TTFT vs. TPOT vs. both.
• The third clock: the latency dimension neither metric touches, and why it produces your most expensive failures.

First principles: what TTFT and TPOT actually measure

LLM generation splits into two phases. Prefill processes the entire input prompt in parallel. It is one big matrix multiply, fast relative to its size, because modern hardware can parallelize across all input tokens at once. Decode generates the output one token at a time, autoregressively. Each token depends on the previous one, so it cannot be parallelized and runs in a tight loop until the model hits an end-of-sequence marker or a length limit.

TTFT and TPOT land on those two phases respectively:

• TTFT (Time to First Token): wall-clock time from when the request is sent to when the first output token is received by the client. It covers: queueing, prefill computation, and network delivery of the first token. The user experiences this as "the wait before it starts."
• TPOT (Time Per Output Token): average time between each subsequent output token once generation has started. The user experiences this as "how fast the text streams."

Neither is "latency" alone. They are two components of it. End-to-end latency is roughly TTFT + (num_output_tokens × TPOT). For a short answer, TTFT dominates. For a long-form generation, TPOT dominates.

Why the bottlenecks are different

This is the most important section, and the one most articles skip.

Prefill is compute-bound. Processing a prompt of N tokens is one large matrix operation. The GPU multiplies the prompt tokens through every layer of the model simultaneously. This saturates the GPU's arithmetic units (FLOPS), not its memory bus. A longer prompt means more compute; a bigger model means more compute per layer. The practical limit is how many FLOPS per second the chip can sustain.

Decode is memory-bandwidth-bound. Generating one token requires loading the model's full weight matrix through the chip to compute a single new vector. The arithmetic is tiny: one token is a dot product against each layer. The bottleneck is how fast the GPU can stream gigabytes of weights from HBM to the compute cores. Every token step reloads the same weights. You are paying memory bandwidth, not FLOPS.

This asymmetry has hard consequences:

• Adding more FLOPs (bigger GPU, faster chip) helps TTFT but gives you almost nothing on TPOT per request.
• Memory-bandwidth optimizations (weight compression, KV cache management, batching) are what move TPOT.
• A perfectly memory-bandwidth-optimized system that is then given a longer prompt sees TTFT rise linearly; TPOT barely changes.

What moves each metric

Moving TTFT

Prefill is the TTFT bottleneck. Every lever above reduces prefill work or spreads it across more compute.

Moving TPOT

Decode is memory-bandwidth-bound. Every lever above reduces the amount of data moved per token step, or amortizes the decode cost across more requests (batching).

Matching the optimization to the workload

There is no universal "make inference faster." There is the right clock for the right complaint.

User says: "It takes forever to start responding."     → TTFT problem → prefill optimizations
User says: "It's so slow to read, the text trickles."  → TPOT problem → decode/batching optimizations
User says: "Long docs take much longer to answer."     → TTFT problem (prompt length scaling)
User says: "When traffic peaks, everything slows."     → batching / continuous batching → TPOT
User says: "Short Q&A feels instant, summaries feel slow." → mixed: TTFT fine, TPOT the pain

A rule of thumb: interactive chat (Q&A, copilots, support) is TTFT-sensitive. Users feel the pause acutely. Long-form generation (summarization, document drafting, code generation) is TPOT-sensitive. Users are already reading and care more about stream cadence than startup latency.

The scorecard that exposes which clock is broken:

The third clock: what neither metric measures

TTFT measures when the tokens start. TPOT measures how fast they arrive. Neither metric asks: are those tokens grounded in the right information?

A system with excellent TTFT and TPOT can stream a confident wrong answer at peak speed, drawn from a stale document, a deleted policy, or an ingestion event that failed silently. The answer arrives fast. It is still wrong.

That matters because the failure modes that produce real business costs all happen in the context layer, not in the decoder. A support agent quoting deprecated terms, an internal tool answering from a document that no longer exists, a legal review grounded in an outdated contract. Inference engineering makes the model respond faster. It has no mechanism to ensure the model is responding from correct, current, replayable context.

The bridge is narrow but precise: TTFT and TPOT are signals of your serving stack's efficiency; neither is a signal of your retrieval layer's correctness. A team that has dialled both metrics down to their hardware limits still has an open question. What was the model actually shown, and can you prove it was right?

Inherent sits on that third axis. Managed ingestion keeps the documents entering retrieval current. Deterministic retrieval means the same query returns the same context, a property fast serving frameworks do not provide. Retrieval receipts let you replay exactly what any past request was grounded in. None of those things affect TTFT or TPOT; they are orthogonal. That is the point: serving speed and context correctness are two separate engineering problems, and closing one does not close the other.

Where to start

Today's task: open your serving logs and look at TTFT and TPOT separately.

• If TTFT is high relative to your prompt lengths, prompt caching or tensor parallelism is likely your first lever.
• If TPOT is high at peak load, continuous batching configuration is where to look.
• If both look fine but users still complain, the complaint is probably not about token speed.

Once you have closed the serving efficiency gap, the open question shifts to the context layer. That is a different debug session, with different tooling, and a different kind of receipt to produce.

Want to see where your retrieval correctness sits? Start with the Inherent Public API. If your serving stack is dialled in and the wrong answers didn't move, DM Flow on X. That is the gap we are building against.

Next read: Disaggregated Inference: When the KV Cache Has to Cross the Network, what happens to TTFT once prefill and decode move onto separate GPU pools.