Continuous Batching Is How One GPU Serves a Crowd

Key takeaway: Continuous batching is the scheduling trick that lets a single GPU serve many users at once without wasting hardware. Instead of loading a fixed group of requests, running them to completion, and then loading the next group — which leaves the GPU idle whenever a short request finishes early and waits for the longest one — continuous batching makes a fresh scheduling decision every generation step: the moment any request finishes, a new one drops into the freed slot. The measured result is large — Anyscale reported up to 23x the throughput of a naive static-batching setup at similar latency. The decision this post helps you make is not whether to use it — every modern serving engine does it by default — but understanding what it buys you: it multiplies how many answers your GPU can produce per second and slashes your cost per token, while doing nothing at all about whether any single answer is correct.

If you serve your own models and your GPU bill is climbing faster than your traffic, the problem is usually not the model — it is that your hardware sits idle a large fraction of the time. A large model generating text is bottlenecked by memory, not math, so a single request barely occupies the chip. Continuous batching fixes the underutilization by packing many requests through the GPU together and refilling the batch continuously as requests come and go, rather than in rigid, all-start-all-finish groups (Anyscale).

So the "so what": continuous batching is the highest-leverage throughput win in self-hosted LLM serving — it can cut your cost per token by an order of magnitude with no change to the model or its outputs — but it is a utilization technique, and utilization is not where your worst answers come from. It was introduced as "iteration-level scheduling" in the Orca serving system and is now the default in vLLM, TGI, and TensorRT-LLM (Yu et al., OSDI 2022). This post is the operator's map of that lever: what it is, where the throughput comes from, where it stops helping, and the one thing it leaves untouched.

Previous post: Speculative Decoding: Faster Tokens, Identical Answers — its latency-per-request sibling.

What this post covers

By the end, you should be able to decide how much throughput headroom continuous batching is leaving on the table in your stack — and recognize the trap of serving more answers per second when the real cost is bad context, not idle silicon.

• What continuous batching really is — iteration-level scheduling, in plain terms.
• Why static batching wastes your GPU — the ragged-finish problem.
• Where the throughput comes from — no padding, higher effective batch size, paged memory, and the 23x.
• Where it stops helping — the prefill/decode tradeoff and latency under load.
• A business-life example and a readiness scorecard — one GPU, a room full of chats, and a worksheet.
• Where Inherent fits — why more answers per second is not the same as a correct answer.

Continuous batching is iteration-level scheduling

Start with the mechanism, because everything else follows from it. A GPU serving an LLM runs the model in repeated forward passes, one token per pass for every request in the batch; continuous batching makes a fresh scheduling decision at each of those passes, so a request can join or leave the batch mid-flight instead of being locked in from start to finish. The unit of scheduling is the iteration, not the request — which is why the original work called it iteration-level scheduling (Orca, OSDI 2022).

The technical explanation: text generation is autoregressive, so every request needs many sequential forward passes, and — critically — you don't know in advance how many. One user's request finishes in 20 tokens; another runs to 800. A scheduler that fixes the batch at the start is stuck waiting for that 800-token request while the 20-token slot sits empty. Iteration-level scheduling re-evaluates the batch every pass: finished requests are evicted immediately and queued requests are admitted into the freed slots, so the GPU is always working on a full batch of live requests. NVIDIA ships the identical idea in TensorRT-LLM under the name in-flight batching (NVIDIA).

The business explanation you can picture in a day: it is the difference between a fixed-departure shuttle bus and a ride that never stops moving. Static batching is the shuttle — it waits until it has a full load, drives everyone to the end, returns empty, and only then takes the next group; if one passenger has a long trip, everyone else's seat is wasted for the whole ride. Continuous batching is a moving walkway: the instant someone steps off, the next person steps on, and the belt is always full.

Why static batching wastes your most expensive hardware

Here is the pattern that turns continuous batching from a trick into a predictable win: the cost of a GPU is fixed per hour whether it is busy or idle, and static batching guarantees it is idle a large fraction of the time — because requests in a batch finish at wildly different lengths, but the batch cannot advance until the slowest one is done. You are paying for capacity you cannot use.

• Generation lengths are unpredictable and uneven. A batch is only as fast as its longest request; every request that finishes sooner leaves a slot doing nothing until the batch rolls over.
• Padding makes it worse. To batch requests of different lengths together, static systems pad them to a common length and compute the padding — burning cycles on tokens that don't exist. Continuous batching, with the right kernels, eliminates that padding (Anyscale).
• The waste compounds under real traffic. Requests arrive continuously, not in tidy groups, so a static scheduler is forever either holding new arrivals in a queue or running half-empty batches.

The net effect is a GPU that looks "in use" but is mostly waiting. Continuous batching reclaims that stranded capacity, which is why the throughput gains are so large rather than incremental.

Where the extra throughput comes from

The answer first: continuous batching raises throughput by keeping the effective batch size high at every step — no idle slots, no padding — so each expensive weight-load through the GPU produces useful tokens for the maximum number of requests. More live requests per forward pass means more tokens per second from the same hardware.

Anyscale's widely-cited benchmark put concrete numbers on it: continuous batching (as implemented in vLLM) delivered up to 23x the throughput of a naive Hugging Face static-batching baseline, while also reducing p50 latency (Anyscale). Two design choices unlock the ceiling:

• Iteration-level scheduling removes the idle gaps and padding (the mechanism above).
• Paged memory removes the memory ceiling that would otherwise cap batch size. vLLM's PagedAttention allocates KV-cache memory in small fixed-size pages just-in-time instead of reserving a large contiguous block per request, cutting memory waste to under 4% and freeing room to fit more requests in the batch (vLLM).

The two work together: scheduling keeps the batch full over time, paged memory lets the batch be large at any instant. That combination is why the number is 23x, not 2x.

Where continuous batching stops helping

Continuous batching is close to free, but it is not a magic dial, and one tradeoff surprises teams in production: mixing new requests (which must process their whole prompt at once — the "prefill") into a batch of in-progress requests (generating one token each — the "decode") can stall the decoders, spiking the inter-token latency everyone else feels. More throughput can quietly cost you a smooth stream.

• Prefill/decode interference. A newly admitted request's prefill is a big, compute-heavy chunk. Drop it into a running decode batch and the token-by-token responses already streaming to users hitch. Naive continuous batching trades tail latency for throughput; chunked prefill (splitting the prefill into small pieces interleaved with decode) is the standard fix (Sarathi-Serve, arXiv 2403.02310).
• The memory ceiling is real. Batch size is ultimately bounded by KV-cache memory; long contexts and long outputs eat it fast, and when you run out, requests queue no matter how clever the scheduler.
• It is a throughput technique, not a single-user speedup. For one request on an empty GPU, continuous batching does little — its whole value is packing concurrent load. If your problem is one user waiting on one slow answer, speculative decoding is the lever, not this one.

The practical read: continuous batching is on by default in your serving engine, but its tuning — chunked prefill, max batch size, memory headroom — is where the throughput-vs-latency balance is actually set.

The business-life example: one GPU, a room full of chats

Picture a support assistant at peak hours: 200 customers typing at once, each expecting a live-feeling reply. With static batching, your GPU processes them in rigid groups; short "what are your hours?" answers finish instantly but their slots sit idle until the group's one 900-token troubleshooting reply completes. To keep latency acceptable you over-provision — three GPUs where one would do — and your inference bill triples.

Turn on continuous batching (you likely already have) and the same GPU keeps every slot full: the instant a short answer finishes, the next queued customer drops in. You serve the whole peak on a fraction of the hardware, and p50 latency improves because requests stop waiting in a queue. That is a genuine, large win — and it is the right thing to do.

Now watch what did not change. One of those 200 customers asks about your refund window and gets a confident, fast answer quoting last quarter's policy — because the help-center article was updated but the retrieval index still holds the old chunk. Continuous batching served that stale answer more cheaply and to more people simultaneously. The expensive event — the escalation, the compliance dispute, the churned account — was never about GPU utilization. It was about which context the model was handed.

A continuous-batching readiness scorecard

Use this to find the throughput you are leaving on the table — and to catch the case where you are scaling up the wrong thing. Each row is a decision; a row you cannot clear is a specific, named gap, not a vague "we should optimize serving."

The pattern the scorecard exposes: the top six rows are about serving throughput — do you control the stack, is concurrency the pain, is utilization measured, is padding gone, is memory paged, is prefill tuned. The bottom two rows are about context, and they are the ones a batching project silently skips, because they are not what it is for. You can clear every throughput row perfectly and still serve a system whose expensive failures come entirely from the two rows it doesn't cover.

Where Inherent fits

Only now, with the mechanism clear, does the product framing earn its place — and it lands on the bottom two rows of the scorecard, Context freshness and Replayability, because those are the rows a serving-layer optimization cannot touch and most teams therefore never close. Packing more answers through a GPU is the easy, well-trodden win; keeping the context those answers are built on fresh, deterministic, and auditable is the half that decides whether your cheap, high-throughput answers are also right.

That is precisely the layer Inherent provides: it sits above your vector storage and below your orchestration, making ingestion managed and retrieval governed rather than best-effort — on a different axis from the batch scheduler entirely. The truth layer version-stamps and hashes every source at ingestion, so the context feeding your (now higher-throughput) model reflects the current state of the world, not a stale snapshot. The memory layer makes retrieval deterministic and version-pinned, so the same query returns the same context — the property that keeps answers reproducible even when thousands are served per second. The audit layer issues a retrieval receipt per request — which sources, versions, and chunks produced the context — so when one of those many answers is disputed, you can replay exactly what the model was shown.

To be clear about where we are: Inherent is early, and this is an architecture argument, not a claim that context infrastructure makes continuous batching unnecessary. You should absolutely use it — it is the best throughput lever in self-hosted serving. The point is narrower and it holds: scaling throughput and correcting context are two different jobs on two different layers, and doing the first one perfectly does not do the second one at all.

The bottom line, and where to start

Continuous batching is how one GPU serves a crowd: instead of running fixed groups start-to-finish and stranding capacity whenever a request ends early, it re-decides the batch every step, dropping new requests into freed slots so the hardware never idles — up to 23x the throughput of static batching at similar latency, with no change to the model or its outputs. It is on by default in every serious serving engine; the leverage is in tuning it (paged memory, chunked prefill, batch-size headroom) and in not mistaking throughput for correctness.

Small task for today: for the AI feature carrying the most concurrent load, answer two questions. First — what is your GPU's actual busy fraction at peak, and are you on continuous batching with paged KV-cache? If you don't know the number, you are almost certainly over-provisioned, and that is a large bill sitting unclaimed. Second — of the last ten costly incidents on that feature, how many were about capacity versus wrong or stale answers? If most were about correctness, you just learned that the thing continuous batching scales isn't the thing costing you. Reclaim the throughput where it helps — then close the context axis: start with the Inherent Public API — get started in the docs. Scaled your serving and the wrong answers didn't budge? DM Flow on X with what broke — that's the gap we're building against.

Next read: Batch Ingestion Is Why Your RAG Answers Go Stale — the context axis no serving optimization can touch.