The Serving Layer Sets Your RAG Latency Floor

Key takeaway: vLLM, TensorRT-LLM, and SGLang all solve the same problem — keep the GPU busy and memory-dense across many uneven, concurrent requests. They do it with continuous batching, paged KV-cache memory, and (SGLang specifically) automatic prefix reuse. Picking one sets your throughput floor. It says nothing about whether the context each request was handed is correct — that's a retrieval problem, and no serving engine touches it.

Serving is the layer between a model's weights and your application. If your RAG system feels slow, look here before you blame the model — and once you've picked the right engine, stop looking here for your correctness problems.

So what: the serving framework you choose sets a hard floor on latency and cost-per-token that no prompt or retrieval tuning can get under. It is also a ceiling you can hit and stop worrying about, because it has no opinion on whether the answer is right.

Previous post: Quantization Is the Cheapest Way to Cut Your Inference Bill.

What this post covers

By the end, you'll know which serving framework fits your workload — and why clearing that decision doesn't touch your remaining correctness problems.

• What serving is, and where it comes from.
• The three mechanisms that make serving fast, and which framework leans on which.
• A business example: a support assistant under spiky traffic.
• What the serving layer does not fix.
• A serving-layer decision scorecard.
• Where Inherent fits.

First principles: what serving is, and where it comes from

A trained model is just weights sitting in GPU memory. Serving is the software layer that turns those weights into a running service: it takes a queue of requests and decides, continuously, what the GPU computes right now.

Where it sits in the stack:

• Below orchestration — Ray Serve, KServe, and Triton route traffic across many models and machines; they don't execute a model themselves.
• Above the raw checkpoint and hardware — serving is the thing that actually runs on the GPU.
• Independent of the model itself — the same weights run faster or slower purely based on the engine underneath them.

Where it comes from: early deployments grouped a fixed batch of requests and waited for all of them to finish (static batching) — fine for a demo, but one slow request blocks everyone behind it in production. Orca (OSDI 2022) fixed this with iteration-level scheduling: decide the batch at every generation step, not once per batch. Every engine below descends from that idea.

The three mechanisms that make serving fast

• Continuous batching. Admit and evict individual requests at every step. A finished request's slot is refilled immediately; a long request never blocks a short one. (Orca, OSDI 2022)
• Paged KV-cache memory. Each request needs GPU memory for its attention state as long as it runs. Naive engines reserve one worst-case block per request. vLLM's PagedAttention pages memory the way an OS pages RAM — small blocks, allocated on demand — so far more requests fit on the same GPU. (vLLM, SOSP 2023)
• Automatic prefix reuse. RAG workloads resend the same system prompt and retrieved context on every call, with only the question changing. SGLang's RadixAttention stores the KV cache as a radix tree, so any two requests sharing a prefix automatically share its cached computation. (SGLang, arXiv 2312.07104)

Which framework leans on which:

• vLLM — the default. Broadest model support, easiest to run, PagedAttention built in from the start.
• TensorRT-LLM — NVIDIA's own engine. Deepest hardware integration: kernel fusion, in-flight batching, tight coupling with NVIDIA's own quantization path (TensorRT-LLM docs). Best if you're committed to NVIDIA hardware and want the last mile of throughput.
• SGLang — the newest, and it earns its place on RadixAttention. Best fit when your workload is heavy on shared prefixes — which most RAG systems are.
• Ray Serve / KServe / Triton — not a competing choice. They route across multiple models and replicas, on top of whichever engine you picked (Ray Serve, KServe, Triton). Most teams need one engine, and eventually one of these on top.

A business example: spiky traffic

A support assistant answers from your help center. Traffic is quiet, then spikes hard when an incident hits and every customer opens a ticket at once.

• Static batching: the burst either queues (customers wait) or forces over-provisioned GPUs (you pay for idle capacity the rest of the day).
• Continuous batching + paged memory: the same GPU fleet absorbs the burst — no new hardware, no code change.
• Prefix reuse: every ticket resends the same policy documents and system prompt, so a prefix-aware engine serves the burst faster still.

None of that tells you whether the policy document being retrieved is this quarter's or last quarter's — that's a different question entirely.

What the serving layer does not fix

Two separate failure surfaces exist in a production RAG system: how fast tokens come back, and whether the tokens fed in were the right ones. Serving only touches the first.

• Prefix and KV-cache reuse are a performance bet: they assume the cached computation is still valid to reuse.
• Nothing in the serving layer checks that assumption against the state of your source documents.
• A fast, well-tuned SGLang deployment reusing a stale prefix is still reusing a stale prefix — just quickly.

A serving-layer decision scorecard

The top four rows are about serving. The bottom two are about context — and a serving-optimization project routinely never touches them, because they aren't what a serving framework is for.

Where Inherent fits

The scorecard's bottom two rows — context correctness and replayability — are exactly what a faster GPU scheduler cannot close. That's the layer Inherent provides, orthogonal to whichever engine you run:

• Truth layer — version-stamps every source, so a reused prefix reflects a known state, not a silent stale copy.
• Memory layer — makes retrieval deterministic, so the same query returns the same context regardless of caching.
• Audit layer — issues a retrieval receipt per request, so a fast answer can still be replayed and disputed on the facts.

This is early, and it's an architecture argument, not a claim that context infrastructure replaces picking the right serving engine. Do both — scheduling tokens fast and getting the context right are two different jobs.

The bottom line

Pick vLLM as your default, TensorRT-LLM if you're committed to NVIDIA hardware, SGLang if your workload leans on shared prefixes. Add Ray Serve, KServe, or Triton only once you need multi-model orchestration. A fast response isn't a correct one — your worst failures live in the context layer, which no serving framework touches.

Small task for today: check whether your engine runs continuous batching with paged memory, or still does static batches. Then check the last ten "slow" complaints — how many were latency versus a fast but wrong answer? Fix serving first, then close the context axis: start with the Inherent Public API — get started in the docs. Wrong answers still showing up? DM Flow on X with what broke.

Next read: KV Caching Is Not Deterministic Retrieval — the mechanism behind prefix reuse, and where it stops being a correctness guarantee.