Batch Ingestion Is Why Your RAG Answers Go Stale

Key takeaway: Most RAG stacks ingest on a schedule — a nightly or hourly job re-reads sources, re-chunks, re-embeds, and rebuilds the index. That batch cadence guarantees a window in which your index disagrees with the source of truth, and inside that window the model retrieves confidently wrong context: the old price, the closed ticket, the deprecated policy. The fix is not a faster cron job; it is a different shape of pipeline — capture source changes as they happen (change data capture), re-embed only what changed, and upsert those chunks into a versioned index. The decision this post helps you make: treat freshness as a retrieval-correctness requirement with an explicit staleness budget, move to streaming, incremental ingestion for the sources that change under live traffic — and pin the index version on every write so "fresh" never costs you "reproducible."

If you run RAG in production, you have almost certainly shipped a batch ingestion pipeline: a job that walks your documents on a timer, chunks them, calls the embedding model, and writes vectors to the index. It is the default because it is simple and every tutorial teaches it. It also has a property nobody puts in the design doc: between two runs, your index is a snapshot of the past, and every answer served in that gap is grounded in stale context. A support agent updates a refund policy at 10am; your reindex runs at 2am; for sixteen hours the assistant cites the old policy with total confidence and zero errors in the logs.

So the "so what": freshness is not an ops nicety, it is a correctness property of retrieval, and batch ingestion trades it away by construction. The tools to do better already exist and are boring in the best way — change data capture streams row-level changes off database transaction logs (Debezium on Kafka Connect), and modern ingestion pipelines already dedup and upsert by content hash so you only re-embed what actually changed (LlamaIndex document management). This post is the freshness blueprint: why batch guarantees staleness, what streaming ingestion actually is, the specific ways it breaks, the architecture that keeps it both fresh and reproducible, and a scorecard to grade your own pipeline.

Previous post: Speculative Decoding: Faster Tokens, Identical Answers.

What this post covers

By the end, you should be able to look at each source feeding your index and say: how far behind the truth can this get, is that gap acceptable for the questions it answers, and if not, what is the smallest change that closes it — without turning your retrieval into something you can no longer reproduce.

• Why batch re-indexing guarantees a staleness window — the base failure mode, made visible.
• What streaming ingestion actually is — change data capture plus incremental, hash-based upserts, not a faster full rebuild.
• The three ways streaming ingestion silently breaks — dangling chunks, re-embed churn, and freshness that quietly destroys reproducibility.
• The architecture that stays fresh and reproducible — capture changes, upsert by hash, pin the version.
• A streaming-ingestion readiness scorecard — a worksheet to grade your pipeline and find the open gap.
• Where Inherent fits — why deterministic, version-pinned retrieval is what makes a live index trustworthy.

Batch re-indexing means your index is always a little bit wrong

Start with the mechanism, because the failure is structural, not a bug you can patch. A batch pipeline defines freshness by its interval: if it runs every N hours, then in the worst case your index is N hours behind the source, and every retrieval in that window returns context that no longer matches reality. The model does not know it is stale. Retrieval succeeds, the chunks are relevant to the query, the answer is fluent — and wrong, because the ground truth moved after the last run.

The base concept to hold onto: there are two clocks in a RAG system — when the source changed, and when the index learned about it — and batch ingestion maximizes the distance between them. A nightly job means a full day of drift. An hourly job means up to an hour. Shrinking the interval shrinks the gap but never closes it, and it does so by paying to re-read and re-embed your entire corpus every cycle, most of which did not change. You are burning embedding spend to reduce a window you cannot eliminate.

The exhibit makes the gap literal: the source changes at an arbitrary moment, but the index only catches up at the next scheduled run, and everything served in between is grounded in the old state.

Streaming ingestion is change capture plus incremental upserts, not a faster rebuild

The answer first: streaming ingestion inverts the pipeline — instead of periodically pulling the whole corpus, it reacts to individual source changes and updates only the affected chunks. It has two halves, and teams that get this wrong usually implement only one.

Half one: capture the change. You need to know a source changed the moment it changes, without polling everything. For databases, this is change data capture: read the transaction log and emit a row-level event for every insert, update, and delete. Debezium does exactly this as a set of Kafka Connect source connectors — it captures changes from the database's own log and streams them as events, with support for an initial snapshot plus incremental snapshots at runtime and include/exclude filters so you only watch the tables that matter (Debezium features). For documents and object stores, the equivalent is a webhook or an event notification on write. The point is the same: the pipeline is driven by change events, not a clock.

Half two: update only what changed. A change event should not trigger a full rebuild — it should re-chunk and re-embed the one document that moved, then upsert those vectors into the index. The mechanism that makes this safe is content hashing: store a hash per document (or per chunk), and on each event compare it. If the hash is unchanged, skip the embedding call entirely; if it changed, re-embed and upsert; if the document was deleted, delete its chunks. This is not exotic — LlamaIndex's ingestion pipeline ships it as document management, with upserts, duplicates-only, and upserts-and-delete strategies keyed on document id and hash, and it explicitly only re-runs the expensive transformations when the hash changed (LlamaIndex).

The base-versus-advanced distinction that matters: batch asks "what does the whole corpus look like now?" and pays for the whole corpus; streaming asks "what changed since I last looked?" and pays only for the delta. The second question is both cheaper and fresher — but only if you handle the delta correctly, which is where it breaks.

The three ways streaming ingestion silently breaks

Here is the pattern that reorganizes how you defend a streaming pipeline: going incremental removes the staleness window but introduces three new failure modes, and all three are silent — the index stays up, it just goes subtly wrong.

Dangling and duplicated chunks. A document that used to produce five chunks now produces three. If your upsert writes the three new chunks but never deletes the two old ones, retrieval can still surface the orphans — stale text that no longer exists in the source, now permanently retrievable. The inverse bug is duplication: re-ingesting without id-stable upserts leaves two copies of the same chunk, skewing similarity scores. Incremental updates are only correct if deletes and replacements are handled as carefully as inserts — which is exactly why the "upserts-and-delete" strategy exists.

Re-embed churn and cost. If your change detection is coarse — "the file's modified timestamp moved" rather than "the content hash changed" — you re-embed documents that did not meaningfully change, and a single upstream batch job touching timestamps can stampede your embedding bill. Streaming that re-embeds on the wrong signal can cost more than batch, not less. Hash-based change detection is what keeps the delta actually small.

Freshness that destroys reproducibility. This is the subtle one, and it is the reason a naive stream is dangerous. If the index mutates continuously, then the context for a given query depends on what second you asked it. Two identical requests moments apart can retrieve different chunks because an upsert landed in between. Now your evals are non-reproducible, your A/B test of a retrieval change is confounded by background mutation, and a production incident cannot be replayed because the index no longer holds the state it held at incident time. You traded a staleness window for a reproducibility hole.

The tension is real and worth stating plainly: batch is reproducible but stale; naive streaming is fresh but irreproducible. The architecture below is how you get both.

The architecture: capture changes, upsert by hash, pin the version

The answer first: you keep freshness and keep reproducibility by making the index versioned — writes stream in continuously, but every read pins to a specific corpus version, so retrieval is always both current-enough and replayable. Three controls, in order.

Capture changes at the source, not on a timer. Drive the pipeline from CDC events (Debezium/Kafka Connect for databases) or write-webhooks for document stores. Filter to the tables and sources that actually feed retrieval so you are not streaming noise. This closes the staleness window: the index learns about a change seconds after it happens, not at the next scheduled run.

Upsert by content hash, and delete on removal. For each change event, hash the new content; skip if unchanged, re-chunk-and-re-embed if changed, delete chunks if the source was removed. This keeps the delta minimal (no wasted embeddings) and keeps the index consistent (no dangling orphans, no duplicates). Hashing is the cheap primitive that makes incremental ingestion both correct and affordable.

Pin the corpus version on every write and every read. Stamp each ingest with a monotonic version (or snapshot id) and let a query request retrieval "as of" a version. Live traffic reads the latest; an eval, an A/B test, or an incident replay pins to a fixed version and gets identical chunks every time. This is the move that resolves the freshness-versus-reproducibility tension — writes are streaming, but reads are deterministic against a named version.

The business-life version: think of a support team's knowledge base. An agent updates the refund policy at 10am. With batch ingestion, the assistant keeps quoting the old policy until the 2am rebuild — sixteen hours of confidently wrong answers. With streaming ingestion, the CDC event fires at 10:00:03, the one changed article is re-embedded and upserted, and the assistant is correct by 10:01. And because the index is version-stamped, when a customer disputes an answer from last Tuesday, you can pin retrieval to Tuesday's corpus version and see exactly which policy text the model was shown — fresh for the live agent, reproducible for the audit.

Why this ordering holds: capturing changes closes the staleness window, hashing keeps the delta correct and cheap, and versioning restores the reproducibility that continuous mutation would otherwise destroy. Drop any one and the other two lose their value — streaming without versioning is fresh but unrepeatable, versioning without hashing is repeatable but expensive, and hashing without capture is efficient but still on a timer.

A streaming-ingestion readiness scorecard

Grade your pipeline against this. Each row is a control; a row you cannot clear is a named, specific gap — not a vague "we should make ingestion faster."

The pattern the scorecard exposes: the top rows are about freshness — react to change, re-embed only the delta — but the bottom rows are about reproducibility. You can build the freshest streaming pipeline on earth and still be unable to explain an answer if the index mutates underneath your evals. Freshness is necessary; version-pinned determinism is what makes the freshness safe to trust.

Where Inherent fits

Only now, with the freshness model built, does the product framing earn its place — and it lands on the two bottom rows of the scorecard, Version pinning and Replayability, because those are the ones every streaming pipeline gets wrong first. Closing the staleness window is the easy, well-trodden half; keeping a continuously-mutating index reproducible is the half that turns your evals into noise and your incident reviews into shrugs.

That is precisely the layer Inherent provides: it sits above your vector storage and below your orchestration, and it makes ingestion managed and retrieval governed rather than best-effort. The truth layer version-stamps and hashes every source at ingestion, so incremental upserts land against a known corpus version instead of "whatever the index looked like at 3pm" — which is what makes hash-based, delete-aware updates correct rather than approximate. The memory layer makes retrieval deterministic and version-pinned: live traffic reads the latest state while an eval, A/B test, or replay pins to a fixed version and gets identical chunks — the exact resolution of the freshness-versus-reproducibility tension the scorecard exposes. The audit layer issues a retrieval receipt per request — which sources, versions, and chunks produced the context — so when an answer is disputed, you can replay the precise state the model saw, not reconstruct it from a mutating index.

To be clear about where we are: Inherent is early, and this is an architecture argument, not a magic ingestion box. You still choose your CDC connector and your embedding model, and you should. What managed context supplies is the missing half of a streaming pipeline: the versioning and provenance that turn "the index is fresh but I can't reproduce anything" into "fresh for live traffic, pinned and replayable for everything that needs to be trusted."

The bottom line, and where to start

Batch ingestion is simple and it is quietly wrong: it defines freshness by a clock and grounds every answer in the gap between runs. Streaming ingestion closes that gap — capture changes at the source, re-embed only the delta by hash, delete what was removed — but it introduces a reproducibility hole that a naive implementation leaves wide open. The fix is to make the index versioned: writes stream in, reads pin to a version, and freshness stops costing you replayability.

Small task for today: pick the one source that changes most often under live traffic and answer three questions about it. How long, worst case, between a change to that source and your index reflecting it? If a customer disputes an answer from last week, can you reproduce the exact context the model saw? And when a document is deleted at the source, are its chunks actually removed from the index? If any answer is "we don't know," you just found the staleness — or the orphan — that will surface as a confidently wrong answer no one can explain. Start there. Then close the version-pinning and provenance half a streaming pipeline depends on: start with the Inherent Public API — get started in the docs. Building your own streaming ingestion and hitting the "fresh but not reproducible" wall? DM Flow on X with where it breaks — that's the gap we're building against.

Next read: Managed Ingestion for RAG: A Field Guide.