Sketchnote: a RAG chunk loses its document provenance; contextual retrieval reattaches it at index time

Retrieval & Vector Search • Sensei Learning System

Why a Chunk Forgets the Document It Came From

A chunk keeps its words. It loses everything the document knew about it.

Core Concept

Standard RAG embeds each chunk on its own. The chunk keeps its words and loses everything the document knew about it — the company, the year, the product. Its embedding is therefore a vector for a generic sentence, and it will match a query about any company’s revenue roughly as well as the right one.

“Revenue grew 3% over the previous quarter.” — which company? which quarter? which product? The vector does not know either.

Contextual retrieval fixes this at index time, not query time: before embedding, prepend a short generated description of where the chunk sits, then embed the concatenation. The vector now carries the disambiguating context the raw chunk never had.

Key Components

The Provenance Gap

A chunk embedded alone is a sentence-shaped vector with no address. Identical sentences across documents produce near-identical vectors, so retrieval can’t tell them apart.

The Contextual Prefix

A short generated line: “from ACME’s Q2 2024 filing, in the section on segment revenue.” Prepend it to the chunk, then embed the concatenation.

Write-Side, Not Read-Side

One extra LLM call per chunk at ingest, cached forever. Prompt caching is what makes the cost affordable at all — the fix is paid once, not on every query.

Measured Gain

Contextual Embeddings plus Contextual BM25: failed retrievals down 49%. Add a reranking step: 67%. Never quote 49% as embeddings alone.

How It Works

Split and embed (the flaw). Document → chunks → each chunk embedded alone. Company, year, product all stay in the document, not in the vector.
Watch it fail. Query “ACME’s Q2 revenue” → the ranking can’t separate the right filing from another company’s identical sentence.
Generate provenance at ingest. For each chunk, one LLM call writes a line describing where it sits in the whole document.
Embed the concatenation. Context + chunk become one unit; index both the contextual embedding and the contextual BM25 text.
Query as usual. Nothing changes at read time — but every vector now carries the document’s context the raw chunk never had.
Verify both ways. Measure retrieval failure rate on held-out queries with and without contexts; sample generated contexts for factual accuracy before trusting the index.

49%

fewer failed retrievals — Contextual Embeddings + Contextual BM25, quoted as the pair

67%

with a reranking step added — quote only when reranking is named

Numbers as reported by Anthropic for Contextual Retrieval.

Real-World Applications

Financial filings — sentences like “revenue grew 3%” recur across every company; provenance is the only thing that separates them.
Enterprise knowledge bases — near-duplicate boilerplate across teams; the team and product name live one level up from the chunk.
Support documentation — “reset the password” means different things per product; the product identity is exactly the lost context.

When to skip it — the falsification side. Not free, and not always a win: you pay an LLM call per chunk at ingest, and you must re-run the whole index when the context prompt changes — on a corpus that churns hourly, the ingest cost can exceed the query saving outright. A self-describing corpus (API reference pages that name their own endpoint in every paragraph) gains almost nothing. And a wrong generated context is worse than none: it embeds a confident false provenance that retrieval metrics will not flag.

Checkpoint — answer before you move on

A chunk reads “revenue grew 3% over the previous quarter.” What does its embedding actually represent — and why does that make it a false-match magnet in RAG? Think: the vector is for a generic sentence, not for a fact about a specific company.
Contextual retrieval is a write-side fix, not a read-side one. What exactly do you pay per chunk at ingest, and what mechanism makes that cost affordable? One LLM call, cached — and prompt caching is the enabler.
Name two situations where contextual retrieval is NOT a win, and explain why a wrong generated context is worse than no context at all. Churning corpus, self-describing corpus — and confident false provenance passes silently through retrieval metrics.