KV Caching Is Not Deterministic Retrieval

Key takeaway: KV caching and prefix caching are exact-prefix optimizations. They reuse the key/value tensors for a run of tokens the engine has already seen, and they reproduce the same computation the uncached path would have produced. That makes them cheap and fast — and completely silent on the question engineers actually care about in RAG: will the same user query return the same answer tomorrow? It won't, and caching can't fix it, because reproducibility is a property of your retrieval layer, not your decode layer. The decision this post helps you make: stop expecting a cache to buy you consistency, and put your determinism effort where the divergence actually happens.

If you serve RAG in production, you have probably watched an identical question produce two different answers a week apart, and you have probably reached for caching to make it stop. That instinct is aimed at the wrong layer. KV/prefix caching — vLLM's automatic prefix caching, SGLang's RadixAttention, Anthropic/OpenAI/Bedrock prompt caching — is a prefill optimization keyed on the exact token sequence. When your retrieved context changes by one chunk, the token sequence changes, the cache misses, and you get a different answer anyway. The cache did its job perfectly. Your system is still nondeterministic, upstream, at retrieval.

So the "so what" is about where you spend engineering budget, not about latency: a cache hit means "I have seen these exact tokens before," not "this query is answered consistently." Conflating the two is the "same query, different context" failure mode. It produces a false sense of reproducibility — "we cache the context, so it's stable" — while the retrieval layer quietly re-embeds, re-ranks, and overwrites underneath you. This post separates the layers so you can put determinism where it belongs and let caching do the one thing it's actually good at.

Previous post: Production RAG Needs Truth and Memory.

What this post covers

By the end, you should be able to look at your own serving stack and say exactly which layer is making your answers non-reproducible — and confirm that adding more caching would not have fixed it.

• What KV and prefix caching actually do — the mechanism, keyed on exact token prefixes, and why it's determinism-neutral.
• The "same query, different context" failure mode — why identical questions diverge, and why the cache is innocent.
• Determinism is three independent layers — retrieval, prompt assembly, decode; caching lives in the third.
• What caching is good for — the real latency/cost win, and the prompt-ordering tradeoff it forces.
• Semantic caching is a different, riskier animal — why "similar query → stored answer" fails on freshness.
• A cache-hit audit — a worksheet to classify what your stack actually guarantees.
• Where Inherent fits — the retrieval-side determinism a cache can't supply.

What KV and prefix caching actually do

Start with the mechanism, because the whole argument rests on one property. During autoregressive decoding, a transformer computes a key and value tensor for every token in the context. The KV cache stores those tensors so generating token N+1 doesn't recompute K/V for tokens 1..N. It trades VRAM for compute inside a single request. Uncontroversial, universal.

Prefix caching extends that reuse across requests. If two requests share a leading run of tokens — a system prompt, a few-shot block, a pinned document — the engine reuses the already-computed KV blocks for that shared prefix instead of recomputing the prefill. The engines differ only in bookkeeping: vLLM hashes KV blocks and maps identical hashes to the same physical block, evicting LRU when memory is tight; SGLang's RadixAttention stores the KV cache in a radix tree and traces each new prompt to its longest already-cached prefix; Anthropic's prompt caching exposes the same idea at the API as cache_control breakpoints, billing cache reads at 0.1x and writes at 1.25x of base input price on a 5-minute ephemeral TTL.

Here is the load-bearing detail: all three key on the exact token prefix. A hit requires that the tokens up to the cache boundary are byte-for-byte identical to something already stored. That gives caching a property people forget to state out loud — it is determinism-neutral. On a hit, you get the same K/V you would have computed anyway; on a miss, you compute from scratch. Caching never changes the answer for a given input, and it never stabilizes an input that wasn't already stable. It is a speed layer bolted under an unchanged compute path.

The "same query, different context" failure mode

Here's the answer first: identical user queries diverge because the retrieved context changes between them, and when the context changes the token prefix changes — so the cache misses and correctly returns a different, uncached answer. The cache is not the bug. It is the witness.

Walk the request path. A user asks the same question twice, a week apart. Between the two calls your retrieval layer has done entirely normal things: the ANN index was rebuilt, a document was re-embedded under a new model version, two chunks were overwritten on update, and the re-ranker broke a top-k tie the other way. The query string is identical. The retrieved context is not. Your prompt template splices that context into the prompt, so the token sequence is different — which means, mechanically, a prefix-cache miss on the context region. The model prefills the new tokens and generates a new answer.

Notice the trap has two false "sames" sitting on top of each other:

1. Same prompt tokens → prefix-cache hit → same prefill compute. This is what caching gives you.
2. Same user query → same retrieved context → same answer. This is what the business asked for.

Caching guarantees (1). Nobody guarantees (2). Teams install prompt caching, watch their prefill cost drop, and unconsciously file "consistency" under the same win — because on the turns where the context happened to be stable, answers were reproducible. That's correlation, not causation. The reproducibility came from the context being stable, which caching then rewarded with a hit. Destabilize the context and you get both a miss and a different answer, and no amount of cache tuning touches it.

Determinism is three independent layers

The fix is to stop treating "deterministic RAG" as one property. It is three, and they compose in order. The exhibit below is the whole model on one page: what each layer must guarantee, what breaks it, and where caching actually lives.

Read it top to bottom. Layer 1, retrieval: same query plus same corpus state must return the same chunks. This is the layer that breaks most often and the one caching cannot reach — a KV cache has no idea your index was rebuilt. Layer 2, prompt assembly: the same chunks must serialize to the same tokens; unstable ordering, non-deterministic templating, or a datetime.now() spliced into the system prompt silently defeats every prefix cache downstream. Layer 3, decode: the same tokens must produce the same output — greedy or temperature=0, a fixed seed, and an honest asterisk that batch scheduling and non-deterministic GPU kernels can still perturb logits at the margin.

KV/prefix caching is an optimization inside layer 3. It presupposes the token sequence; it cannot produce one. If you want "same query → same answer," you buy it in layers 1 and 2, and only then does a cache hit in layer 3 mean something — it becomes the reward for an input you made reproducible, instead of a coincidence.

What caching is actually good for (and the tradeoff it forces)

Kill the misconception and caching becomes genuinely valuable — just for a different metric. Its real win is prefill cost and time-to-first-token on shared, stable prefixes: system prompts, few-shot exemplars, long static context, multi-turn history. RadixAttention reports high hit rates on workloads with heavy prefix overlap; Anthropic prices cache reads at a tenth of input cost. When your prefix is stable, that is free money and free latency.

The design move that follows is concrete: order your prompt stable-prefix-first, volatile-suffix-last. Put the system prompt and rarely-changing context up front where they cache; put the freshly retrieved chunks and the user turn at the end where they're expected to change. In Anthropic's API that's literally where you drop the cache_control breakpoint:

{
  "system": [
    { "type": "text", "text": "<stable system prompt>" },
    { "type": "text", "text": "<pinned policy doc>",
      "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [
    { "role": "user", "content": "<freshly retrieved chunks + user query>" }
  ]
}

And on a self-hosted engine it's one flag — --enable-prefix-caching on vLLM — with the same ordering discipline in how you build the prompt.

But state the tradeoff honestly, because engineers punish hand-waving: the ordering that maximizes cache hits can fight the ordering that maximizes answer quality. Models attend differently to different positions; the "recency" slot at the end of the context is often where you'd want your most relevant retrieved chunk, not your least cacheable one. Maximizing prefix reuse pushes volatile-but-important context to the back. That's a real latency-versus-relevance decision, and it's yours to make per workload — not something the cache resolves for you.

Semantic caching is a different, riskier animal

One clarification prevents a genuinely dangerous conflation. Semantic caching caches answers keyed on the embedding similarity of the query — "this new question is close enough to one I answered, return the stored answer." That is not KV/prefix caching. KV/prefix caching is exact-prefix and lossless; semantic caching is approximate and lossy by design.

For anything freshness-sensitive, semantic caching actively manufactures the bug this whole post is about. Two questions that embed similarly are not guaranteed to have the same correct answer — "what's our refund window?" and "what's our return window?" may be different policies — and a stored answer can be stale the moment the underlying document changes. Exact-prefix caching at least fails safe: change the input, miss the cache, recompute. Semantic caching fails open: change the underlying truth, still hit the cache, serve the old answer. Use it only where a slightly-wrong-or-stale answer is acceptable, and never as a substitute for retrieval-layer determinism.

A cache-hit audit

Run this on one production endpoint before you touch a cache config. For each layer, decide what your stack actually guarantees — not what you hope it does. Anything you can't check "Guaranteed" is a source of divergence that caching will not repair.

The pattern the worksheet exposes: five of the six rows live above the cache. If your "consistency" problem is in retrieval or prompt assembly — and it almost always is — no caching change will move it.

Where Inherent fits

Only now, with the layers separated, does the product framing earn its place. Caching cannot give you deterministic retrieval; something has to, and that something is a managed context layer that owns layers 1 and 2. That's what Inherent is — it sits above your vector storage and below your orchestration, and it maps onto the exact layers a cache can't reach.

The truth layer owns managed ingestion: it version-stamps and hashes each source, so "the corpus state for this query" is a pinned, replayable fact rather than whatever the index happened to hold today. The memory layer makes retrieval deterministic and tenant-safe: the same documents plus the same query return the same chunks, inside an access boundary — which is precisely the layer-1 guarantee that makes a downstream cache hit meaningful instead of accidental. The audit layer issues a receipt: which sources, versions, and chunks produced the context, so a reproduced answer is provably reproduced, not just plausibly so. Caching then does what it's good at — cheap prefill on a prefix you made stable on purpose.

To be clear about where we are: Inherent is early and this is an architecture argument, not a benchmark claim. If your answer non-determinism turns out to live entirely in decode-layer kernel jitter and your retrieval is already pinned, then this framing doesn't apply to you and you should ignore it. But if you've been tuning cache configs to fix a consistency problem, the layers above are almost certainly where it actually lives.

The bottom line, and where to start

A cache hit proves the tokens matched. It says nothing about whether they'll ever match again — and in RAG, whether they match again is decided upstream, at retrieval and prompt assembly, long before the KV cache sees a token. Spend your determinism budget there. Let caching do the one honest job it has: cheap, fast prefill on a prefix you deliberately kept stable.

Small task for today: take one endpoint and run the cache-hit audit above. Log the retrieved chunk IDs for the same query on two different days — if they differ, you've just proven the cache was never your consistency problem. Then wire that endpoint's retrieval behind a layer that pins it: start with the Inherent Public API — get started in the docs. Building deterministic retrieval yourself and hitting the index-rebuild or tie-break walls this post describes? DM Flow on X with where it breaks — that failure mode is exactly what we're building against.

Next read: RAG Architecture Tradeoffs in Plain English.