How Chunking Works: Why Your Best Document Never Gets Retrieved

Chunking gets treated as a formatting decision. Cut documents into pieces because whole documents are too big to send to the model; pick a size, add some overlap, move on. It looks like the least interesting part of building a retrieval system.

A small experiment makes it obvious that this framing is wrong, and it is easy enough to reproduce.

The experiment

Construct a document with eight sections, where section 2 answers a specific question perfectly. Embed the whole document as a single vector. Measure its cosine similarity to that question.

Then repeat with only seven sections. Then six. Down to one.

Using a simple synthetic model where each section is a distinct topic direction, the curve looks like this:

- 1 section: 0.989

- 2 sections: 0.699

- 3 sections: 0.571

- 4 sections: 0.477

- 5 sections: 0.430

- 6 sections: 0.394

- 7 sections: 0.366

- 8 sections: 0.343

Steady decay. Nothing truncated. Nothing errored. Every one of those vectors is perfectly valid.

Now add a second document to the comparison. A thin, low-quality page that mentions the topic in passing and answers nothing at all. In the same model, it scores 0.796.

So the document containing the correct answer scores 0.343, and the document that answers nothing scores 0.796. Top-1 retrieval returns the useless one.

The exact figures depend on the model, but the shape does not. It is a property of averaging, not of any particular embedding.

Why it happens

An embedding model outputs a fixed number of values no matter how much text goes in. Four words in, 1024 numbers out. Four thousand words in, still 1024 numbers out.

Under the hood, the encoder produces a vector per token and then pools them, usually by averaging, into that single fixed-size output. So a document covering eight topics is averaged into a point sitting between all eight. And the average of eight directions points at none of them.

The vector does not mean "security policy." It means something closer to "corporate document.

The part that matters

Look at where the failure occurs.

The usual objection to oversized chunks is that they waste context and dilute the prompt. That is real, but it happens later. In this experiment the failure happened before the prompt existed. The correct document was never retrieved. It never reached the model. The context window concern never got a chance to matter.

Which means chunking is not preprocessing.

Chunking decides what is findable.

A chunk is the unit that gets scored, ranked and truncated away. Get it wrong and the right answer becomes structurally unreachable, and no reranker rescues it, because reranking only reorders things that were already found.

The failure that is easy to miss

Here is a chunk of the kind of character-count splitter produces:

> "It must be renewed within 30 days of expiry. Failure to do so incurs a penalty of 15% of the outstanding balance, and access is suspended until the matter is resolved."

A user asks: How long do I have to renew my contractor license?

Most people spot that "It" has no referent. The model cannot tell which license this is about. That is a real problem.

The bigger problem is that this chunk contains neither the word "contractor" nor the word "license." Not once. Its embedding is about renewal deadlines and penalties in the abstract. Keyword search finds no overlap either.

The chunk is not ambiguous. It is invisible. Nothing you build will retrieve it for that query.

What actually fixes it

Overlap. Sliding the window so consecutive chunks share some text. Helps when a fact is severed exactly at a boundary. Does nothing here, because "It" refers to a heading pages back, not fifty tokens back.

Parent-document retrieval. Embed small chunks for precision, hand the model the whole enclosing section when one hits. Genuinely fixes the ambiguity. Does not fix retrieval, because you are still searching over the small chunk, which still has no hook for the query.

Contextual retrieval. At index time, send each chunk together with the full document to a cheap model and ask it to write one or two sentences situating that chunk. Prepend that before embedding, and before keyword indexing.

The chunk becomes:

> "This chunk is from Section 4 of the Contractor Licensing Policy, covering renewal deadlines and late penalties. It must be renewed within 30 days of expiry..."

Now the vector carries "contractor licensing." The keyword index carries those literal terms. And any model reading it knows what "It" is. One intervention, both failures.

Anthropic published measurements on this approach: contextual embeddings cut failed retrievals by 35%, adding contextual keyword search took it to 49%, and adding a reranker reached 67%.

But the cost, surely

One LLM call per chunk sounds absurd until you notice when it happens.

Index time is offline. Nobody is waiting. If it takes six hours, nobody notices. Query time is the opposite: someone is watching a spinner and the budget is a couple of hundred milliseconds.

Which gives a principle that applies far beyond this one technique:

Push every possible unit of work from query time to index time.

Contextual retrieval is that principle. So is precomputing summaries, extracting metadata with a model, resolving entities. All expensive, all fine, because you pay once instead of on every request.

Before any of that, the free wins

Most pipelines leave easy quality on the floor by splitting on character counts.

Split on the document's own structure instead. Headings, sections, list boundaries. A heading exists because the author decided the topic changed there, which is free labelling somebody already did.

Never split a table from its headers. This one is worth being paranoid about. A severed row turns "4,200" into a number with no column attached, and the model will confidently report it as revenue when it was headcount. Silent, plausible, completely wrong.

If one thing survives from this: chunking is not preprocessing. It is the decision about what the system is capable of finding at all.

Visual walkthrough

Explore the animated walkthrough of chunking and contextual retrieval:

https://sagartrivedi.dev/learn-rag/#s4

Chapter 4 demonstrates embedding dilution. Chapter 5 covers the invisible-chunk failure and the contextual retrieval fix.

What is your chunking strategy, and has it actually been measured, or does it just feel reasonable? "Feels reasonable" is the default state for a surprising number of production systems.