LLM Observability: A Trace You Can't Replay Is a Screenshot

Key takeaway: LLM observability has mostly converged on the right first move — trace every stage of the pipeline as a span tree, with token counts, latencies, and costs attached. Do that; it's table stakes, and the OTel GenAI conventions mean you don't even have to invent the schema. But a trace answers "what did the system do?", and the question that actually closes an incident is "can I make it do that again?" For RAG, the answer is usually no — because the trace records which chunks were retrieved, while the corpus that produced them has already moved on: re-embedded, re-indexed, overwritten. The decision this post helps you make: instrument in two layers — spans for visibility, retrieval receipts for replay — and stop treating a screenshot of the incident as if it were a repro.

Here is the incident this post is about. A user reports that your assistant gave a customer the wrong cancellation policy on Tuesday. It's Thursday. You open the trace — and it's a good trace: the span tree shows the query rewrite, the vector search, the rerank, the generation; you can see the six chunk IDs that went into the prompt and the couple hundred milliseconds the retriever took. Then you re-run the same query to reproduce the bug, and you get different chunks — the nightly ingestion job re-embedded that document Wednesday. The trace shows you exactly what happened, and you still cannot make it happen again.

So the "so what" is a debugging-economics decision, not a dashboard decision: span coverage tells you where the failure was; only replay tells you why, and replay is a property of your retrieval layer, not your tracing SDK. Most teams buy the first and assume they got the second. This post covers both layers — what to trace, and what to pin.

Previous post (the offline half of this loop): RAG Evaluation: Prove a Retrieval Change Actually Helped.

What this post covers

By the end, you should be able to pull one production trace and say whether it is a repro or a screenshot — and know exactly which attributes and receipts to add if it's the latter.

• What a RAG trace actually is — the span tree, the OTel GenAI conventions, and why you should not invent your own schema.
• The retrieval span is where most traces go blind — the chunk-level attributes that make a trace debuggable.
• Three signals, three budgets — metrics, traces, and online evals, and the sampling economics that keep them affordable.
• Replay: the difference between a trace and a repro — why a span tree over a non-deterministic retrieval layer can't reproduce an incident.
• A 20-minute instrumentation audit — a worksheet to grade one endpoint.
• Where Inherent fits — retrieval receipts and pinned corpus state, the layer a tracing SDK can't supply.

What a RAG trace actually is

Answer first: a RAG trace is one request rendered as a tree of spans, where each pipeline stage — query rewrite, embedding, vector search, rerank, prompt assembly, model call, post-processing — is a span with a start time, a duration, a status, and stage-specific attributes. If you have only ever logged the final answer, this is the single biggest visibility upgrade available to you, and it is boring, standardized work.

flowchart LR
  R[Request span] --> QR[Query rewrite]
  R --> EMB[Embed query]
  R --> VS[Vector search]
  R --> RR[Rerank]
  R --> PA[Prompt assembly]
  R --> GEN[Model call]
  VSA["chunk IDs, scores,<br/>index + doc versions"] -.-> VS
  GENA["model, tokens,<br/>latency, cost"] -.-> GEN

Don't invent the schema. The OpenTelemetry GenAI semantic conventions define standard span names and attributes for model calls — operation, model name and version, token usage in and out, tool calls — so a span from your agent looks the same as a span from a raw API call, and any compliant backend can read it. Two honest caveats engineers should know before betting on it. First, the conventions are still marked Development status as of mid-2026 — attribute names have churned before and may churn again, so pin your instrumentation library version and expect a migration. Second, adoption is real but uneven: Datadog, MLflow, and most of the 2026 tool field (Langfuse, Phoenix, LangSmith all speak OTel to varying depth) can ingest it, but each still has proprietary extensions where the conventions run out. Emit OTel as your wire format anyway — it's the difference between switching backends with a config change and switching with a rewrite.

On picking a backend: this is deliberately not a comparison post — the 2026 SERP has plenty, and the differences are real but second-order (Langfuse self-hosts well and is OTel-native; Phoenix is strongest on RAG evals; LangSmith is deepest if you live in LangChain/LangGraph). The schema you emit matters more than the backend you pour it into, because the schema is the part you can't cheaply change later.

The retrieval span is where most traces go blind

Here is the failure pattern: teams instrument the model call thoroughly — model, tokens, latency, cost, sometimes the full prompt — because that's what the SDKs give you for free, and leave the retrieval span as a black box with a duration. That's backwards for RAG debugging. When an answer is wrong, the first question is which chunks the model was given and whether the right ones were in the set — the two-layer split from the evaluation post: retrieval failure and generation failure need different fixes, and a trace that can't distinguish them can't route the incident.

The retrieval span needs, at minimum: the chunk IDs returned, their similarity scores, the document ID and version each chunk came from, the index/collection version searched, the embedding model and version used for the query, and the retriever strategy (hybrid weights, filters, top-k, reranker version). None of this is exotic — it's a dozen span attributes:

with tracer.start_as_current_span("rag.retrieve") as span:
    results = retriever.search(query_embedding, top_k=8)
    span.set_attributes({
        "rag.retriever.strategy": "hybrid_rrf",
        "rag.retriever.top_k": 8,
        "rag.embedding.model": "text-embedding-4",
        "rag.index.version": index.version,          # which index build
        "rag.chunks.ids": [c.id for c in results],
        "rag.chunks.scores": [c.score for c in results],
        "rag.chunks.doc_versions": [c.doc_version for c in results],
    })

The one attribute in that block that most teams cannot fill in is doc_version — not because the span API is hard, but because their ingestion pipeline doesn't version documents at all. Hold that thought; it is the entire second half of this post.

Two practical warnings before this hits production. Payload cost: logging full chunk text on every span multiplies your telemetry volume; log IDs and scores on every trace, and sample full payloads (or fetch text lazily by ID at debug time — which, again, requires that the ID still resolves to the same text later). Privacy: prompts and chunks are user data and corpus data; if your traces leave your boundary, you've built a second, less-governed copy of your knowledge base inside your observability vendor. Decide deliberately what gets redacted, sampled, or self-hosted.

Three signals, three budgets

Observability isn't one signal, and pricing all three like they're one is how bills explode. The stable pattern across the 2026 tooling field separates them by unit cost:

1. Metrics — request rate, latency percentiles by span type, token spend, error rate, retrieval score distributions. Cheap enough to compute on everything, and score distributions are your early-warning system: a drifting mean similarity score often precedes user-visible quality drops.
2. Traces — the span trees above. Cheap enough to capture broadly (head-sample if volume forces it, but keep 100% of errors and outliers), and they're the raw material every other signal reads from.
3. Online evals — quality judgments on live traffic. This is where the money goes, so the common production pattern is tiered: run cheap heuristic checks (citation present, refusal detection, empty-retrieval, format validity) on essentially all traces, run LLM-as-a-judge scoring on a 10–20% sample, and use periodic human annotation to calibrate the judge and refresh your frozen eval set. The judge-bias controls from the evaluation post apply unchanged online.

The connective tissue between these tiers is the trace ID. A heuristic flag on a trace should link to the span tree; a bad judge score should link to the exact chunks; and — the point of the next section — the chunks should link back to a corpus state you can still reconstruct.

Replay: the difference between a trace and a repro

State the claim plainly: a trace is evidence; a repro is a trace plus the ability to re-run the request against the same state that produced it. In conventional backend systems the gap is small — replay the request, and deterministic code plus a database with point-in-time recovery gets you close. In RAG the gap is where debugging goes to die, because the "state" is your corpus as seen through an embedding model, an index build, and a ranking function — and every one of those mutates continuously and silently. Re-embedding jobs, index rebuilds, document updates, reranker deploys: each one means the same query now retrieves different chunks, and your Tuesday incident is unreproducible by Thursday.

This is the same three-layer determinism argument from the KV caching post, surfacing in a new place: decode-layer nondeterminism you can squeeze with temperature=0 and a seed, but retrieval-layer nondeterminism is invisible to every tool that only watches requests. Your tracing SDK faithfully records a different chunk set on the replay and has no opinion about why.

What replay actually requires is a retrieval receipt: alongside the span, persist which source documents, at which content versions, through which embedding model and index build, produced the context. Then keep old versions addressable, so "re-run this trace as of Tuesday" is a query your retrieval layer can answer. With the receipt, three moves open up that a screenshot never allows: you can reproduce the incident exactly; you can bisect it (same query against Tuesday's corpus vs. today's — if answers differ, the corpus changed, not the model); and you can regression-test the fix by replaying last week's failed traces against the candidate change before it ships — which is also how production failures become eval-set entries for the offline gate.

The honest tradeoff: receipts are not free. Versioning sources and keeping old versions retrievable costs storage and ingestion discipline, and it constrains how casually you can rebuild an index. Whether that's worth it depends on what a wrong answer costs you — the same expected-value logic as the semantic caching decision. If your RAG app is a low-stakes internal search box, level 2 may genuinely be enough. If an unreproducible wrong answer to a customer is an escalation, level 3 is the difference between a fix and a shrug.

A 20-minute instrumentation audit

Pull one production trace from your busiest RAG endpoint and grade it against this worksheet. Each row is a question the trace either can or cannot answer — no partial credit.

Rows 1, 2, 4, and 7 are solved by adopting a tracing SDK and an afternoon of span attributes — do them this week. Rows 3, 5, and 6 cannot be solved by any tracing SDK, because they are properties of your retrieval and ingestion layer. That asymmetry is the point of the post.

Where Inherent fits

The receipt rows are where Inherent sits. It's a managed context layer above your vector storage, and its three layers map onto exactly the rows the tracing SDK can't reach. The truth layer version-stamps and hashes every source at ingestion, so doc_version is a fact your spans can carry rather than a field you wish existed. The memory layer makes retrieval deterministic — same query, same corpus state, same chunks — which is what turns "re-run the trace" into a meaningful experiment instead of a new roll of the dice. And the audit layer is the retrieval receipt: every retrieval records which sources, versions, and chunks produced the context, addressable after the corpus has moved on.

To be clear about where we are: Inherent is early, and this is an architecture argument, not a benchmark claim. You can build receipts yourself — version your ingestion, snapshot index manifests, log the mapping — and if your corpus changes rarely, that build is modest. The claim is narrower: replay is a retrieval-layer property, someone has to own it, and a tracing dashboard — however good — is not that someone.

The bottom line, and where to start

Trace everything; that argument is over, and the OTel GenAI conventions mean the schema is a download, not a design project. But grade your observability by the incident it can close, not the dashboard it can draw. A span tree over a mutating, unversioned corpus documents your failures beautifully and reproduces none of them — a screenshot, not a repro.

Small task for today: pull one wrong-answer trace from the last month and try to reconstruct the exact chunk set the model saw, at the versions it saw them. If you can — you're at level 3, and your remaining work is sampling and eval coverage. If you can't, you've found the gap this post is about, and no tracing vendor will close it. Wire that endpoint's retrieval through a layer that issues receipts: start with the Inherent Public API — get started in the docs. Building receipts and versioned ingestion yourself? DM Flow on X with where it breaks — the unreproducible-incident failure mode is exactly what we're building against.

Next read: KV Caching Is Not Deterministic Retrieval — the same determinism argument, one layer down.