Retrieval looks like magic. Up close it's arithmetic.

A number

Here is the whole trick, and it is not much of a trick. A model reads a piece of text and outputs a list of numbers. That is it. Everything else in this guide is built on top of that one move.

What is any single number actually measuring?

Nothing you can name. There is no dimension for "legal-ness" or "urgency". The meaning is spread across all 1024 values at once, which is why you cannot read an embedding and why nobody tries. What matters is not any individual number but where the whole list points.

A vector

Put those 1024 numbers together and you have coordinates. Two numbers locate a point on a page. Three locate a point in a room. A thousand locate a point in a space you cannot picture, which turns out not to matter at all, because every operation you need works exactly the same way in two dimensions as in a thousand.

Why the space has that shape

It was trained that way, deliberately and unmysteriously. You feed the model millions of pairs of things people treat as related, a question and its answer, a title and its article, and you pull each pair together while shoving unrelated text apart. Do that enough times and the geometry falls out.

Which tells you exactly where it will let you down. The model only knows the notion of similarity that was in its training pairs. Nobody trained it on pairs where your internal project codename means the same thing as your product. In your corpus, it does.

Four things a vector cannot represent

The distance between two vectors

Two points. One question: how close are they? The answer is the single most important calculation in retrieval, and it is three lines of arithmetic you already know.

The arithmetic, in full

# a and b are just lists of numbers
a = [3, 1, 0, 2]
b = [2, 0, 1, 3]

# 1. dot product: multiply position by position, add it up
a·b = (3×2) + (1×0) + (0×1) + (2×3) = 12

# 2. magnitude: the length of each arrow (Pythagoras, extended)
|a| = √(9+1+0+4) = √14 = 3.7417
|b| = √(4+0+1+9) = √14 = 3.7417

# 3. cosine: divide the lengths out, leaving pure direction
cos(a,b) = 12 / (3.7417 × 3.7417) = 0.8571

A chunk

You do not embed documents. You embed chunks. Almost everyone assumes this is a workaround for context limits, and almost everyone is wrong. The real reason goes back to the property from scale 01: the output is always the same size.

Why it happens, mechanically

An encoder produces one vector per token, then pools them, usually by averaging, into the single vector you store. So a chunk covering eight topics is literally the average of eight directions, and the average of eight directions points at none of them. Nothing is truncated. Nothing errors. The vector is perfectly valid. It just means "corporate document".

Which gives you the size trade-off, in both directions

The usual band is 200 to 800 tokens, but treat any number you read as a starting point rather than an answer. It depends on your content, and it is measurable, so measure it.

A document

Now zoom out to the thing being cut up. A document has structure, and the moment you slice it on character counts you throw that structure away. Here is the failure that results, and it is more interesting than it first looks.

The ladder of fixes, weakest to strongest

Before you reach for any of that, take the free wins

• Split on the document's own structure. A heading exists because the author decided the topic changed there, which is free labelling.
• Never split a table from its headers. A severed row turns "4,200" into a number with no column, and the model will cheerfully tell someone it is revenue when it is headcount.
• Never split code blocks, and repeat table headers on every row-chunk.
• Recursive splitting order: paragraphs, then sentences, then words. Reach for semantic chunking last, it is expensive and the results are mixed.

A corpus

Something new becomes possible at this scale, and it is the reason a scoring function from the 1970s is still in every serious retrieval system built today. Once you can see the whole collection, you can measure how rare a word is. An embedding model never can.

Rarity is the whole signal

# inverse document frequency, over a 2,000 document corpus
IDF(term) = ln( (N − df + 0.5) / (df + 0.5) + 1 )

So combine them. But you cannot just add the scores.

Cosine is bounded in [−1, 1] and means the same thing on every query, forever. BM25 is unbounded and its scale is set by whichever words you happened to type. The same scoring function over the same corpus gave a top score of 2.692 for one query and 9.550 for another, purely because the second contained a rarer word.

So a BM25 score of 6.1 is meaningless on its own, normalising is fragile because the distribution shifts per query, and the standard answer throws the scores away entirely and keeps only the ranks.

# Reciprocal Rank Fusion. Σ means "add up, over each list it appears in".
score(d) = Σ 1 / (k + rankᵢ(d))          # k ≈ 60

# vector list : docX(1)  docY(2)  docZ(3)
# bm25   list : docW(1)  docY(2)  docV(3)

docY = 1/(60+2) + 1/(60+2) = 0.0323   # 2nd in both
docX = 1/(60+1)            = 0.0164   # 1st in one, absent from the other

# docY wins despite never being anyone's top pick.

An index

Comparing your query against ten million vectors one at a time is a lot of arithmetic. So at scale you stop doing it exactly. You should know precisely what you are giving up, because nothing in the system will tell you.

A query

Zoom out from the index to the thing arriving at it. The raw string a user typed is usually a bad search key, and what you do between receiving it and answering is a funnel, not a lookup.

Why two different models, and not just the good one

Fixing the query before you search with it

A prompt

Five chunks go in a box with some instructions and a question. This step gets treated as string concatenation, and four real decisions hide inside it.

A pipeline

Far enough out to see both halves at once. There are two pipelines, not one, and they obey completely different physics. Almost every good decision in RAG comes from knowing which half you are standing in.

Your index is a copy, and copies drift

The fix is a posture, not a patch: ingestion is a data pipeline, not a script. Content hashing for change detection, deterministic chunk IDs from (doc_id, version, index), delete-then-insert semantics, and a reconciliation job that diffs source against index. Then pick a freshness SLA deliberately, because "eventually" is how ghosts happen.

Where to run it

Graduate when you actually hit the wall: tens of millions of vectors, sustained high QPS, multi-region. Then choose on filtered-search behaviour, native hybrid support, real delete semantics and multi-tenancy model, not on benchmark latency, which is almost never your binding constraint.

A product

Out one more level and the question stops being how to build RAG and starts being whether to. The most senior thing you can know about an architecture is when it is the wrong one.

Route by the shape of the question

A field

Zoom out past your own system and look at what has actually been built. Thousands of RAG deployments later, they are mostly five shapes. Knowing which one you are building tells you which failures to expect before you meet them.

What the successful ones have in common

• Narrow scope, deliberately. The deployments that work answer one kind of question for one kind of user. The ones that stall tried to be an assistant for everything.
• A human already did this job. There is a support team, a paralegal, an on-call engineer. That means there is a definition of a good answer, and someone who can tell you when it is wrong.
• Being wrong is recoverable. A cited draft a human reviews, not an autonomous action. The successful pattern is overwhelmingly assistive, and the ones reaching for autonomy are still mostly demos.
• Someone owns search quality. Not as a launch task. As a job, with a metric, indefinitely.

An organization

The last zoom out. At this scale RAG stops being a technique and becomes a mirror. Most projects do not fail on retrieval quality. They fail here.

Teams that ship this discover their documentation contradicts itself, the same policy exists in four systems with three different answers, and nobody can say authoritatively who is allowed to see what. None of that was created by the retrieval system. It was already true. The system just started reading it out loud.

Measurement, or you are guessing

Eleven failure modes produce one symptom: the answer was wrong. Without per-stage instrumentation you cannot tell them apart, and most teams respond to all eleven by swapping the LLM. The single highest-value artifact in a RAG project is the one most commonly skipped.

The corpus is a write path into your prompt

Mitigations exist and none are complete. Structurally delimit retrieved content and declare it as data rather than instructions. Apply least privilege to any tool in the loop. Require human confirmation before consequential actions. Never auto-render generated links. Treat it like XSS before frameworks handled escaping: layer defences and assume some will fail.

Two governance facts that surprise people

Where this is all going

The linear pipeline you just read is increasingly a special case. The newer shape hands a model a search tool and lets it decide when to search, what for, and whether to search again after reading. Multi-hop stops being a special technique and becomes default behaviour. Long context, meanwhile, genuinely ate the low end: nobody builds an ingestion pipeline for a single 120-page contract now.

The five invariants

These are properties of the problem, not of any solution, which is why they have survived every architectural change so far. Any future approach still has to answer all five. If one appears to vanish, it has moved somewhere you are not looking.

Quick reference

The parts worth keeping open in another tab.

The eleven failure modes · one symptom, eleven causes

Pre-flight · run it, do not recall it

Numbers worth remembering

Fifteen principles, compressed

• Retrieval implies selection. If it applies to every query, there is nothing to retrieve.
• Behaviour goes in training or the prompt. Knowledge goes in retrieval.
• Push work from query time to index time. Index time is cheap and unobserved.
• Capacity failures are loud. Quality failures are silent. The silent ones hurt.
• Chunking exists because embeddings have a fixed information budget, not because of context windows.
• BM25 knows your corpus. Embeddings know language. That is why you need both.
• Fuse on ranks. Raw scores are on incompatible, query-dependent scales.
• Stage-one recall is a ceiling. Everything downstream is capped by it.
• Over-fetch early, cut ruthlessly late.
• A correct answer can hide a broken guarantee.
• Match the fix to the mechanism. Ask what a technique actually changes.
• RAG answers questions whose answer lives in a few passages. Not properties of the whole corpus.
• You cannot fix what you cannot localize.
• Corpus size does not determine architecture. Per-query working set does.
• Everything retrieved is untrusted input, including your own corpus.