Key takeaway: Disaggregated inference runs prefill on one pool of GPUs and decode on another, joined by a network link that hands off the KV cache. It has gone from a 2024 research paper to how Meta, Kimi (Moonshot AI), and NVIDIA's Dynamo framework actually serve production traffic. Splitting the phases does not remove the bottleneck between them, it relocates it onto the network. The decision this post helps you make: whether you're at the scale where that trade is worth making, and if you are, whether you need exotic hardware to make it work or just better engineering.
Disaggregated inference means running prefill and decode as two separate services on two separate pools of GPUs instead of one process on one chip. The KV cache (the keys and values the model just computed while reading your prompt) has to physically leave the prefill machine and land, intact, on the decode machine before a single output token can be produced. That handoff now stands between your prompt and your answer.
The so what: this is a real production pattern now, not a paper. It trades a memory-bandwidth ceiling for a network-transfer floor, and the second one is worse if you build it badly. This post maps out what disaggregation is, why the KV cache transfer becomes the hard part, how production systems actually move it, and the scale at which it's worth the added distributed-systems cost.
Previous post: Inference Engineering: TTFT, TPOT, and Two Clocks, the two latency clocks disaggregation is trying to move.
What this post covers
By the end, you should be able to tell whether a latency problem needs a second GPU pool or just better scheduling on the one you have.
- What disaggregated inference is, and where it sits after chunked prefill in the serving stack.
- Why the KV cache becomes a network problem: what's actually being moved, and why plain TCP struggles.
- How production systems move it: NIXL/RDMA, Mooncake's Transfer Engine, and Meta's TCP-based path (yes, TCP).
- What changes as traffic scales: a business example from one GPU box to a fleet.
- A decision framework: colocated vs. chunked prefill vs. disaggregated, and when to move on.
Disaggregation is prefill and decode running as two separate services
Mechanically, disaggregated inference removes the assumption that one model has to live on one GPU for the whole request. Prefill and decode become two independently scaled services, each with its own pool of hardware, connected by a KV-cache transfer path instead of shared GPU memory.
The idea started as a research argument. DistServe (OSDI 2024) formalized the split as a goodput-optimization problem and reported serving 7.4× more requests, or holding a 12.6× tighter latency target, than a colocated baseline within the same SLOs (DistServe). Splitwise (ISCA 2024) built the same idea into "prompt machines" and "token machines," each tuned to its own bottleneck (Splitwise).
What changed in 2025–26 is that this moved into production, in three visible steps:
- Mooncake, the serving platform behind Moonshot AI's Kimi assistant, built a KVCache-centric architecture around the same split: 75% more requests under real workloads, up to 525% more throughput in simulated scenarios, versus its prior colocated system, and a FAST 2025 Best Paper award (Mooncake).
- Meta rolled prefill/decode disaggregation into its internal vLLM-based inference stack and is upstreaming pieces to the open-source project, with measured TTFT and inter-token-latency gains over its prior setup (PyTorch & vLLM, Sept 2025).
- NVIDIA Dynamo, an open-source serving framework that sits above vLLM, TensorRT-LLM, and SGLang, ships disaggregation as its headline feature. vLLM itself now has native integration with both Dynamo and the community project llm-d (NVIDIA Dynamo docs).
Where this sits in the serving stack: it's the second of the two prefill/decode fixes, and the one that goes past a single node. Chunked prefill interleaves the two phases on one GPU. It's the single-node mitigation, and where most teams should start. Disaggregation doesn't fit more work onto one chip; it adds a chip, and a network link between them.
The KV cache now has to cross a wire
Once prefill and decode are two separate machines, the KV cache can't stay in local GPU memory. It has to be sent across a network and re-injected into the decode GPU before decode can produce a token. That transfer is now on the critical path to your response. Replace one bottleneck with a slow version of a new one and you've made the system worse, not faster.
What's actually moving, and why it's hard:
- Model weights already sit on both pools; what moves is the request-specific KV tensors, sized by model size and prompt length, so a long prompt on a large model means a bigger transfer on every request.
- A naive network path stages data through the kernel and host memory, copying tensors through the CPU before they land in decode GPU memory, adding latency on the request's critical path.
- NVIDIA's NIXL, used by both Dynamo and vLLM's native disaggregation support, skips that CPU step by running over RDMA-capable networking (UCX). Dynamo's own docs are blunt about the alternative: without RDMA or an equivalent fast fabric, the backend falls back to TCP, and the KV transfer "can dominate TTFT and throughput" (NVIDIA Dynamo docs). Mooncake ships a dedicated Transfer Engine and a separate KV cache store for the same reason, treating the transfer as its own subsystem rather than an afterthought bolted onto the serving loop.
The most useful data point here complicates the "you need RDMA" story. Meta's September 2025 disclosure says its production stack ran over plain TCP and an internal Thrift-based transfer layer, not RDMA, which the team listed as future work. They got it into production anyway, through engineering rather than exotic hardware:
- Multiple network cards routed to the nearest GPUs.
- The KV cache sliced and sent over several parallel TCP streams, because one stream can't saturate the link.
- Sticky routing: a session's requests keep landing on the same prefill host, which raises the prefix-cache hit rate.
- Larger KV cache block sizes (128–256 tokens, instead of vLLM's default 16) to cut the number of small transfer operations.
The result was a 40–50% prefix-cache hit rate while holding 90% HBM utilization, a real, contended production number rather than a clean-room benchmark (PyTorch & vLLM, Sept 2025). RDMA is the faster path, and the one a from-scratch build should reach for, especially cross-rack. But "the network is merely adequate" isn't disqualifying: Meta's numbers show a team can out-engineer a slower transport, at the cost of exactly that engineering effort.
What changes as one box becomes a fleet
Picture a coding assistant that reads large diffs (heavy prefill) and streams a rewritten file back (long decode). At low volume, one GPU handling both phases is fine; a little interference, nobody notices.
Traffic grows, and a team that already shipped chunked prefill is in good shape: big diffs get sliced and interleaved with in-flight streams, and the stutter mostly disappears. Then a launch triples traffic. Several large diffs land in the same second. Chunked prefill is still fighting for the same finite compute and memory bandwidth on one chip, and TTFT starts climbing during every burst.
That's the point disaggregation is for: a prefill pool sized for compute-heavy bursts, a decode pool sized for the long streaming tail, each autoscaling on its own signal instead of both waiting on the other's headroom.
But that comes at a cost. It's a distributed system now: a prefill-to-decode ratio to tune (Meta's own results show a prefill-heavy workload regresses TTFT under high load if that ratio is wrong), a transfer layer to operate and monitor, and a new failure mode where the network, not either GPU, is the thing that's down. Disaggregation doesn't remove complexity, it moves it from "one crowded GPU" to "two pools and the wire between them." It's worth that move only once the crowded-GPU cost is bigger than the distributed-system cost.
When disaggregation is worth it, and when it isn't
Three tiers, ordered by scale, not by sophistication. Use each until it stops working, then move to the next.
| Approach |
What it buys |
What it costs |
Move on when |
| Colocated (one GPU) |
Simplest to run and debug |
A fresh prefill stalls every decode already in flight |
Latency complaints track concurrent long prompts |
| Chunked prefill (single node) |
Interleaves both phases on one GPU; smooths inter-token latency |
Still bounded by one chip's total compute and bandwidth |
TTFT climbs during bursts even with interleaving |
| Disaggregated (two+ pools) |
Independent scaling per phase, each sized for its own bottleneck |
A transfer layer to build and run, a ratio to tune, network as a new failure mode |
The stall you're fixing costs more than the system you'd build |
None of this (the pool split, the transfer engine, the RDMA link) touches whether the content inside that KV cache was correct. A perfectly engineered NIXL transfer moves a stale support article exactly as fast as a current one; disaggregation optimizes placement and timing, not truth. That's a separate axis, and it's the one Inherent works on: deterministic retrieval and a receipt for what any past answer was actually grounded in, independent of how many GPU pools served it.
Small task for today: if prefill and decode are still colocated, check whether your TTFT complaints cluster around bursts of long prompts. If they do, chunked prefill is your next lever, not a second GPU pool. Disaggregation is a scale decision, not a default. If you're past that point and building a transfer path, start from Meta's tradeoff: RDMA buys speed; routing, batching, and block-size tuning buy the same destination, slower to reach but cheaper to start. Want to see where your retrieval correctness sits once serving is dialled in? Start with the Inherent Public API. Building the transfer layer and hit a wall? DM Flow on X with what broke.
