What Is Inference Engineering? A First-Principles Map

Key takeaway: Inference engineering makes a trained model run fast and cheap in production, and almost every technique exists because of two physical constraints: prefill is compute-bound, decode is memory-bandwidth-bound. The decision this post helps you make: take your actual complaint (slow to start, slow to stream, expensive, falling over under load, or just wrong) and go straight to the lever, and deep dive, that moves it.

You train a model once. You pay for inference on every request, forever. That asymmetry is why inference engineering exists: it decides how fast a user sees the first word and how much each word costs on the GPU bill. The classic failure: a team adds GPUs because the app feels slow, and the complaint does not move, because compute was never the bottleneck the user was hitting. Almost every technique below traces back to two physical constraints on how a transformer runs. See those two, and the field stops being a grab bag of acronyms and becomes a map.

Previous post: Test-Time Compute: The Model Decides Your Bill.

What this post covers

By the end you should be able to: (a) explain what inference actually is and where the term "inference engineering" comes from, (b) name the two physical constraints that explain nearly every optimization technique, and (c) take a specific user complaint and know which lever, and which deep dive, addresses it.

• Inference is the forward pass you pay for on every request: what it is, what an inference engine and serving layer are, and where this discipline sits in the stack.
• Two physical constraints explain almost every technique: prefill is compute-bound, decode is memory-bandwidth-bound, and the KV cache is the variable cost in between.
• Every technique moves one of four levers: latency, throughput, cost, or quality, and each has a name and a deep dive.
• Start from the complaint, not the technique: a table that turns "what you're seeing" into "what to read next."
• Inference engineering makes the answer fast; it cannot make it right: where this discipline ends and the context layer begins.

Inference is the forward pass you pay for on every request

A trained model is a fixed set of weights. Inference is the act of running input tokens through them, a forward pass, to produce output tokens. It happens in two phases: prefill, where the whole prompt is processed at once, and decode, where the model produces one new token at a time, each one depending on every token that came before it.

An inference engine is the runtime that executes this: vLLM, SGLang, and TensorRT-LLM are the common ones. It schedules incoming requests, manages GPU memory, and decides how many requests run at once. Serving is the layer around the engine: routing, autoscaling, health checks, queues. Inference engineering tunes both.

The term is recent for a reason. Gergely Orosz, writing in the Pragmatic Engineer in March 2026, defines inference engineering as optimizing how models run in production after training, "from CUDA to Kubernetes." Three years earlier, he notes, perhaps a few hundred people did this work, almost all inside frontier labs; it spread to product companies once open-weight models became practical to self-host. It excludes training. It also differs from MLOps, which covers pipelines and deployment hygiene; this discipline covers the physics of a GPU at request time.

Where it sits in the stack: model weights, then the inference engine, then serving infrastructure, then the application layer (prompting, retrieval, context). Inference engineering owns the middle two.

Technical explanation: the forward pass splits into a compute-heavy prefill and a memory-heavy decode loop, and the engine's job is to keep the GPU busy in both without running out of memory.

Business explanation: picture a support copilot team at a mid-size company. The model answers tickets all day, and the bill is per token, every day. Two complaints show up in Slack: "it takes forever to even start typing" (a prefill problem) and "the answer trickles out one word at a time" (a decode problem). They sound the same to the support lead. They are fixed by different work, and knowing which one you have is the first decision this field asks of you.

Two physical constraints explain almost every technique

Prefill and decode are bottlenecked by different things, which is why the field has many separate techniques instead of one.

Prefill is compute-bound. Processing a prompt is a stack of large matrix multiplications, one per layer, run over all prompt tokens in parallel. For prompts of any real length the GPU's arithmetic units saturate; the limit is FLOPS.

Decode is memory-bandwidth-bound. Each new token means streaming the model's full weights (or, for MoE, the active experts) from high-bandwidth memory (HBM) through the chip to produce one vector. The arithmetic per step is small; moving the weights is what takes the time. Pope et al. (2022) lay out this split directly.

Between the two sits the KV cache: the attention state (keys and values) kept for every token already processed, so the model does not recompute the whole prompt for each new token. It is the memory cost that grows per request and per generated token, and it is what caps how many concurrent requests fit on one GPU. Kwon et al. (2023), the vLLM paper, showed it is the largest variable memory cost per request and that earlier engines lost much of it to fragmentation.

Nearly everything downstream of this section is one of two moves: relieve the compute bottleneck in prefill, or relieve the memory-bandwidth bottleneck (and KV cache pressure) in decode.

Every technique moves one of four levers, and the map tells you which

Once you know which constraint you are fighting, the technique list sorts into four levers: latency, throughput, cost, and quality.

Latency. End-to-end latency is approximately TTFT plus (output tokens times TPOT): time to first token, then time per output token, repeated for every token generated.

• TTFT and TPOT are the two clocks users feel, driven by prefill and decode respectively: TTFT, TPOT, and Two Clocks, Prefill and Decode: Two Bottlenecks.
• Prompt and prefix caching skip re-running prefill on repeated prompt segments, cutting TTFT directly; tensor parallelism splits prefill across GPUs when the prompt is the problem: Prompt Caching Cuts Cost.

Throughput. How many requests, and how many tokens per second, one GPU actually serves.

• Continuous batching packs decode steps from many requests onto one GPU pass instead of running requests one at a time: Continuous Batching.
• The KV cache is the concurrency ceiling, running out before compute does; paging it (PagedAttention) is how engines stop wasting it: The KV Cache Is Your Concurrency Ceiling.
• Disaggregation splits prefill and decode onto separate GPUs so neither phase steals capacity from the other: Disaggregated Inference.
• The engine you pick sets the latency floor before any of the above applies: The Serving Layer Sets Your RAG Latency Floor.

Cost. What each token actually costs to produce.

• Quantization shrinks the weights moved through memory every decode step, usually the cheapest lever available: Quantization Is the Cheapest Way to Cut Your Inference Bill.
• Routing sends easy requests to cheap models and hard ones to expensive models: Model Routing and Fallbacks.
• Mixture of experts activates only a fraction of parameters per token, cheap to compute but expensive to hold in memory: Mixture of Experts.
• The train-versus-rent decision sets your cost floor before any serving optimization applies: Inference Engineering for CEOs: Train or Rent?

Quality. The lever the other three trade against, and the one people forget is a lever at all.

• Speculative decoding speeds up decode with output-identical results, so it costs no quality (it does cost draft-model memory and some wasted compute): Speculative Decoding: Faster Tokens, Identical Answers.
• Quantization taken too far is the one technique above that can cost you quality.
• Reasoning models spend extra output tokens at decode time specifically to buy higher quality: Test-Time Compute: The Model Decides Your Bill.

Start from the complaint, not the technique

Most teams pick a technique first and hope it matches their problem. Go the other direction: start from what you observe.

Inference engineering makes the answer fast; it cannot make it right

Every lever in this post changes how quickly and how cheaply tokens arrive. None of them check what the model was shown before it started generating. A stale document chunk, a policy deleted last week but still surfaced by retrieval, a search that returns different context on two runs of the same query: those are failures in the context layer, above the serving stack this post describes.

This is where Inherent's own work sits, and it is worth being precise about the boundary. We build managed ingestion, deterministic retrieval, and retrieval receipts, so what a model sees is fixed and auditable before generation starts. That work is orthogonal to TTFT and TPOT: you can have a perfectly tuned inference stack and still get confidently wrong answers, because speed and correctness are decided by different parts of the system.

TLDR

• Inference is the forward pass through fixed trained weights: prefill processes the whole prompt at once, decode generates one token at a time. Inference engineering optimizes that in production; it excludes training.
• Prefill is compute-bound (limited by FLOPS); decode is memory-bandwidth-bound (limited by moving weights through HBM).
• The KV cache is the per-request memory cost that caps how many concurrent requests one GPU can serve.
• Every technique in the field moves one of four levers: latency, throughput, cost, or quality.
• None of those levers check whether the model was shown the right context; that failure mode lives one layer up, in retrieval.

Where to start

Today, pull p50 and p95 TTFT and TPOT, plus GPU utilization, from your serving logs, and classify your top user complaint using the table above. That single exercise tells you whether you have a latency problem, a throughput problem, a cost problem, or something the inference stack was never going to fix.

If your serving stack is tuned and the wrong answers did not move, that is the gap Inherent is built for: DM Flow on X, or read the Public API docs first.

Next read: Inference Engineering: TTFT, TPOT, and Two Clocks