Prompt Caching Cuts Cost. Semantic Caching Cuts Corners.

Key takeaway: Prompt/prefix caching is a lossless discount on compute you already paid for — reuse the exact token prefix, pay roughly a tenth of the input price on the hit. It pays back on the first reuse within the cache window and only ever costs you a small premium on prefixes you don't reuse in time. Semantic caching is a different trade: it reuses the answer to a similar query, skips the model call entirely, and is lossy by design — so its bigger savings come attached to a stale-answer liability that scales with how much a wrong answer costs you. The decision this post makes for you: bank exact-prefix caching aggressively; treat semantic caching as a scoped optimization with a freshness gate, never a default.

If you are trying to cut an LLM bill, "turn on caching" is not one decision — it is two, with opposite risk profiles, and conflating them is how teams either leave safe savings on the table or ship a correctness bug to production. Exact-prefix caching (KV/prefix caching on your own engine, or provider prompt caching from Anthropic, OpenAI, and Bedrock) discounts the prefill on tokens you have already sent. Semantic caching skips the whole call when a new query embeds close to an old one. Both reduce spend. Only one can be wrong.

So the "so what" is a budgeting decision, not a latency one: spend your caching effort where the savings are free, and put a guardrail where they aren't. The free savings are in exact-prefix caching, and most teams under-collect them because their prompt is ordered wrong. The dangerous savings are in semantic caching, and most teams over-collect them because "the bill dropped" hides the stale answers underneath. This post gives you the cost math for both so you can tell them apart.

Previous post (the determinism angle on the same machinery): KV Caching Is Not Deterministic Retrieval.

What this post covers

By the end, you should be able to look at one production endpoint and say which caching savings you can bank without risk, which ones carry a stale-answer liability, and roughly what each is worth.

• Two caches, two economics — why exact-prefix caching is lossless and semantic caching is lossy, and why that decides everything.
• The cost math of prompt caching — read/write multipliers, the 5-minute window, and why it pays back on the first reuse.
• The cost math of semantic caching — why the savings look bigger and where the expected value goes negative.
• A worked example and the decision boundary — the "refund window vs return window" hit that serves the wrong policy, plus one exhibit for which cache to use where.
• A 10-minute cost audit — a worksheet to price your own endpoint.
• Where Inherent fits — why cheap cache reads presume a fresh retrieval layer.

Two caches, two economics

Start with the base distinction, because the entire cost argument rests on it. The two things people call "caching" reuse different objects.

Exact-prefix caching reuses compute. KV caching stores the key/value tensors for tokens the model has already processed; prefix caching (and provider prompt caching) extends that reuse across requests when a leading run of tokens is byte-for-byte identical — a system prompt, a few-shot block, a pinned document. On a hit you skip the prefill for that prefix and pay a steep discount instead of full price. It is lossless: the answer is exactly what the uncached path would have produced. You can never be wrong because you cached; you can only be cheaper.

Semantic caching reuses answers. It embeds the incoming query, finds a stored query that is similar enough by vector distance, and returns that query's stored answer — skipping the model call entirely. It is lossy by design: "close in embedding space" is not "same correct answer," and the stored answer was true whenever it was written, not necessarily now.

Here is the business translation. Exact-prefix caching is like reusing a document you have already read to the model instead of re-reading it aloud every time — same document, cheaper delivery. Semantic caching is like a support rep answering a new ticket from memory because it "sounds like" one they handled last month — fast, and fine right up until the policy changed. The first can only save you money. The second can save you money and hand a customer a wrong answer. That asymmetry is the whole game.

The cost math of prompt caching (what you actually bank)

Answer first: prompt caching pays back on the first reuse inside the cache window, and the only way to lose is to cache a prefix you don't reuse in time. There is no correctness risk to weigh — just a small, bounded premium.

Use Anthropic's published prices as the worked example, because they are explicit: a cache write costs 1.25× the base input price, a cache read costs 0.1×, and the ephemeral cache lives on a 5-minute TTL (Anthropic prompt caching docs). The mechanics differ by provider — OpenAI caches automatically with no write premium and its own read discount; Bedrock uses explicit breakpoints like Anthropic — but the core shape is identical: a repeated prefix costs a fraction of the first pass. Put two requests that share a cacheable prefix side by side:

• Uncached: 1.0× + 1.0× = 2.0× on that prefix.
• Cached: 1.25× (write) + 0.1× (read) = 1.35×.

You are already ahead at the second request. Every further read inside the 5-minute window costs 0.1× instead of 1.0× — a 90% discount on that slice of tokens. The exhibit below is the curve.

The lever, therefore, is within-window reuse rate, and it is almost entirely a prompt-ordering problem. Prefix caching only hits on the leading identical tokens, so you order the prompt stable-prefix-first, volatile-suffix-last: system prompt and pinned context up front where they cache; freshly retrieved chunks and the user turn at the end where they are expected to change. In Anthropic's API that is literally where you place the cache_control breakpoint; on vLLM it is --enable-prefix-caching plus the same ordering discipline.

State the tradeoff honestly, because this is where cost fights quality: the ordering that maximizes cache hits can fight the ordering that maximizes answer quality. Models attend more strongly to the end of the context, so the "recency" slot you would want to fill with your most relevant retrieved chunk is exactly the slot you must keep volatile and uncached. Maximizing prefix reuse pushes important-but-fresh context toward the back of the cacheable region. That is a real latency-and-cost-versus-relevance decision, and it is yours to make per workload — but notice it is a tuning decision, not a correctness one. Worst case, you cached slightly too much and paid a 25% premium on a prefix that expired unused. Nobody got a wrong answer.

The cost math of semantic caching (and where the number lies)

Semantic caching looks like the better deal on the invoice, and that is the trap. A prefix-cache hit still runs the model — you only saved the prefill. A semantic-cache hit skips the entire generation, so the per-hit saving is much larger. If you optimize on the bill alone, semantic caching wins every time.

The bill is the wrong objective function. The honest one prices the downside:

Expected value per query ≈ (hit rate × saving per hit) − (hit rate × stale rate × cost of a wrong answer).

The first term is what shows up on your dashboard. The second term shows up in a support queue, a compliance review, or a churned account, and it never gets attributed back to the cache. When the cost of a wrong answer is low — a phrasing suggestion, a non-personalized FAQ blurb — the risk term is negligible and semantic caching is genuinely good. When the cost of a wrong answer is high — a pricing quote, a refund policy, a medical or legal statement, anything a customer acts on — even a small stale rate drives the expected value negative, and the cheaper your cache made the bill, the more confidently you served the stale answer.

And notice the failure direction, because it is the opposite of exact-prefix caching. Exact-prefix caching fails safe: change the input by one token, miss the cache, recompute the correct answer — the worst case is you paid full price. Semantic caching fails open: change the underlying truth (someone updates the policy doc), and a new query still matches the old query, so you serve the old answer at a discount, with no signal that anything is wrong. Exact-prefix caching's failure costs money. Semantic caching's failure costs trust, and bills you a discount for the privilege. That is why it belongs behind a freshness gate — a TTL tied to how often the underlying source changes, and a scope limited to queries where stale is tolerable — and never as a blanket default. (The determinism mechanics behind "same query, different answer" are the subject of the previous post.)

A business-life example you can picture

Put it in a support workflow. Two tickets arrive an hour apart: "What's your refund window?" and "What's your return window?" Those two questions embed almost identically — same domain, same shape, one near-synonym apart — so a semantic cache scored on vector similarity will happily treat the second as a hit on the first and return the refund policy to a return question. They may be different policies with different windows. The cache did exactly what it was configured to do; the customer got the wrong one.

Now add time. Finance updates the refund window from 30 days to 14 in the source document at 2pm. Your semantic cache was populated at 1pm and its answer is on a longer TTL. Every "refund" question between 2pm and cache expiry gets the old 30-day answer — quoted confidently, at a tenth of the cost, with a provenance trail that says nothing changed. The saving is real and on the dashboard. So is the liability, and it is not.

An exact-prefix cache cannot produce either failure. Different query tokens → cache miss → the model actually reads the current retrieved context and answers. The most it ever costs you is the compute you were trying to save.

The caching decision boundary

So stop asking "should we cache?" and ask "which cache, where?" The two axes that decide it are how much a wrong answer costs and how stale-tolerant the query is. The exhibit maps them.

Read it as an operating rule. Exact-prefix caching is on by default, everywhere — it spans the whole grid because it has no correctness downside; your only job is to order the prompt so it actually hits. Semantic caching is opt-in, for the bottom-right corner only — low cost of a wrong answer, high tolerance for staleness — and even there it needs a TTL tied to how often the source truth changes. Anywhere in the top half of the grid, where a wrong answer is expensive, semantic caching is a liability wearing a discount, and the right move is to not cache the answer at all — cache the prefix, and let the model read fresh context.

A 10-minute cost audit

Run this on one endpoint before you touch a cache config. The point is to separate the savings you can bank from the ones that carry a bill you can't see.

The pattern the worksheet exposes: the top two rows are pure upside you are probably under-collecting, and the bottom four are risk you may be over-collecting without knowing it. Cost optimization is moving spend from the second group to the first.

Where Inherent fits

Here is the relevance bridge, and it is the honest reason the two caches have different risk. A cache read is only as good as the freshness of what it reuses. Exact-prefix caching stays safe precisely because it reuses compute over a prefix your system reassembles each call — if the retrieved context changed, the tokens change and the model reads the new truth. Semantic caching is dangerous precisely because it reuses an answer and has no idea the underlying source moved. Both problems are really one problem: is the retrieval layer underneath your cache fresh and deterministic?

That is the layer Inherent owns. The truth layer version-stamps and hashes each source at ingestion, so "the current policy" is a pinned, replayable fact rather than whatever a stale cache last saw. The memory layer makes retrieval deterministic and tenant-safe — the same query over the same corpus state returns the same chunks — which is what lets you cache the prefill aggressively without caching a wrong answer. The audit layer issues a receipt of which sources and versions produced a context, so a cached answer is traceable rather than invisible. Get that layer right and you can bank every safe caching discount, because the thing underneath the cache is no longer moving without your knowledge.

To be clear about where we are: Inherent is early, and this is an architecture argument, not a benchmark claim. If your only caching is exact-prefix and your retrieval is already pinned and versioned, this framing just confirms you're doing it right. But if you reached for a semantic cache to cut cost and it's sitting over an unversioned corpus, the bill went down and your exposure went up at the same time.

The bottom line, and where to start

Two caches, two economics. Exact-prefix caching is free money on a prefix you deliberately keep stable and correctly ordered — bank it everywhere, its worst case is a 25% premium on tokens you wasted. Semantic caching is a real saving with a real liability — confine it to queries where a stale answer is cheap, gate it on a freshness TTL, and keep it out of anything a customer acts on. If you can't tell which of your savings is which, you are not optimizing cost; you are trading a visible bill for an invisible one.

Small task for today: take one endpoint and price it with the audit above. Compute the within-window reuse rate on your cached prefix (that's your bankable win), then list every semantic-cached query and its cost-of-wrong-answer (that's your hidden exposure). If the second list has anything a customer acts on, you found the bug before it found you. Then put your retrieval behind a layer that keeps the underlying truth versioned so you can cache without fear — start with the Inherent Public API, get started in the docs. Cutting LLM cost and hitting the fail-open wall this post describes? DM Flow on X with where the bill and the truth diverged.

Next read: Managed Ingestion: A Field Guide.