Hybrid Retrieval: What Embeddings Cannot See

Picture the query. Something like ERR_4021, pasted straight in.

Semantic search returns three documents. "Troubleshooting common errors." "What to do when something fails." "Error handling best practices." All thematically reasonable. None of them mentions ERR_4021.

The document explaining that error exists. It is indexed. It is two paragraphs long and it is exactly what was needed. Vector search cannot find it.

The reason is structural, and once it is clear, hybrid retrieval stops looking optional.

Embeddings do not know what is rare

An embedding model reads a chunk and produces a vector. Here is the part that gets glossed over: it reads that chunk in isolation. It has never seen your corpus. It does not know what else you have. It cannot know that ERR_4021 appears in 2 of your 2,000 documents and is therefore almost a unique fingerprint.

So it does what it does with any unusual token. Smears it into the general region of "something error-code shaped" and moves on.

BM25, the keyword scoring function that has been in search engines since the 1970s, is built on exactly the quantity embeddings cannot see. Inverse document frequency measures one thing: how rare is this term across the collection.

Over a 2,000-document corpus, using the standard formula:

- "the" appears in 1,990 documents. IDF: 0.0053

- "policy" appears in 400. IDF: 1.6087

- "refund" appears in 120. IDF: 2.8098

- "err4021" appears in 2. IDF: 6.6851

That last term contributes roughly 1,270 times what "the" contributes. Rarity is the entire signal.

Which gives the cleanest statement of why both exist:

BM25 knows about your corpus. Embeddings know about language.

BM25 literally cannot compute a score without knowing what else is in the collection. An embedding is computed by a model that has never seen your collection and never will.

Which is why you want both

This is not old versus new. They are good at genuinely different problems, and the failures do not overlap.

Embeddings win on paraphrase. A user types "I can't log in" and the document is titled "Credential recovery procedures." Zero words in common. Semantic search handles it effortlessly and BM25 returns nothing.

BM25 wins on exact tokens. Error codes, SKUs, function names, stack traces, rare surnames, internal project codenames. Anything where the literal string is the point.

Because they fail in uncorrelated ways, combining them is not a marginal gain. It is the single highest return change available to most retrieval systems.

Where combining gets tricky

Running both and adding the scores does not work, and the reason is worth understanding.

Cosine similarity is bounded between minus one and one, and it means the same thing on every query, forever. A cosine of 0.82 is a specific geometric fact.

BM25 has no upper bound, and its scale is set by whichever words the user typed. On the same corpus with the same scoring function, two different queries produce top scores of 2.692 and 9.550. Three and a half times apart, and not because the second match was better. Purely because the second query contained a rarer word.

So a BM25 score of 6.1 means nothing on its own. It cannot be added to a cosine score, and normalising is fragile because the distribution shifts on every query.

The standard answer throws the scores away

Reciprocal Rank Fusion uses only the ranks.

score(document) = sum over each list of  1 / (k + rank in that list)

with k usually around 60. That is the whole algorithm.

Worked through: vector search returns docX first, docY second, docZ third. Keyword search returns docW first, docY second, docV third.

- docY: 1/(60+2) + 1/(60+2) = 0.0323

- docX: 1/(60+1) = 0.0164

docY wins, despite never being anyone's top pick, because both retrievers independently liked it.

That is the design intent. RRF rewards agreement between retrievers over enthusiasm from one of them.

Ranks work as the common currency for a simple reason. Rank 1 means the same thing in every list, on every query, from every retriever. It is the only comparable quantity available for free.

The constant k controls the balance. Large k compresses the gaps between top positions so consensus dominates. Small k makes first place almost unbeatable. At k=60 the consensus document wins by 2.0x. At k=0.1, by only 1.05x.

One honest caveat: RRF is not strictly better than score-based fusion. It discards magnitude, so a runaway first place earns no more credit than a marginal one. It is chosen for robustness and zero tuning, not peak quality. Very little in this field is a free lunch, and anything presented as one is usually the wrong answer.

The practical read

A system running pure vector search has a blind spot shaped exactly like its users' most specific queries. Error codes. Order numbers. Product SKUs. The exact strings people paste in when they are frustrated and want a precise answer.

Those are also the queries where a wrong answer costs the most trust.

Adding a keyword index alongside a vector index is roughly a day of work. Fusing with RRF is an afternoon. It gets skipped mainly because keyword search feels like a step backwards.

It is not a step backwards. It is half of the problem embeddings structurally cannot see.

Visual walkthrough

Explore the hybrid retrieval mechanics and arithmetic:

https://sagartrivedi.dev/learn-rag/#s6

Chapter 6 covers BM25, IDF, and rank fusion. Chapter 8 shows where retrieval and reranking fit into the full query funnel.

Hybrid or pure vector? And if pure vector, what happens when someone pastes an ID into it?