When Not to Use RAG: You Are Sizing It Off the Wrong Number

Here is a scenario that sounds like an obvious retrieval problem. Eight thousand contracts, around eighty pages each. Roughly half a billion tokens. Users asking questions about them all day.

Nothing that size fits in a context window, so the reflex is to build a RAG pipeline.

Then you notice one line in the requirements: users only ever ask about one contract at a time.

The number that actually decides it

Think about what that means mechanically.

If every question is about a specific contract, you never need semantic search across eight thousand documents. You need to identify which contract, which is a metadata lookup. A database query on customer name or contract ID. Once that returns, you are holding one document of about forty thousand tokens, which fits comfortably in a modern context window.

The corpus is eight thousand documents. The working set per query is one document.

Corpus size does not determine architecture. Per-query working set does.

This may be the most commonly misapplied number in the field. A 500GB corpus where every query is scoped to a single document is not a retrieval problem; it is a filtering problem followed by a small-context problem. Meanwhile, a 50MB corpus where the answer is scattered across documents nobody can identify in advance genuinely is a retrieval problem, at a fraction of the size.

Total corpus size is intimidating and largely irrelevant. It gets used anyway because it is the number that is easiest to look up.

What scoping first buys back

There is a second payoff that takes longer to notice.

When the model holds an entire contract, it can answer questions that top-k retrieval structurally cannot.

"Does this contract exclude planned maintenance from the SLA?"

That is an absence question. You cannot retrieve the absence of something. Chunk-based retrieval returns passages that exist, so a contract with no maintenance clause returns nothing relevant, and the model has no way to distinguish "there is no such clause" from "I did not find it."

Give it the whole document and the question becomes trivial.

The same applies to comparison across sections, to summarising obligations, and to anything needing the shape of the whole rather than a passage from it.

So filtering to a bounded document does not just simplify the stack. It buys back a class of question that would otherwise be lost.

Routing by the shape of the question

A more useful question than "should we use RAG" is "what shape is this question?"

The answer is a live authoritative value. Account balance, current inventory, order status, seat count. This is a database query or an API call. It was never a retrieval problem. A surprising number of production RAG systems are built over data that should have been a SELECT statement, and they fail in ways nobody can debug because the data was stale the moment it was indexed.

The requirement is about behaviour. Output format, tone, always ending with a structured block. That is not knowledge, it is disposition. It belongs in the system prompt, in structured output constraints, or in fine-tuning. The tell is that it applies to every single query, and retrieval implies selection. If there is nothing to select between, retrieval is the wrong mechanism by definition.

Every query is scoped to one bounded document. Filter, then send the whole thing.

The question contains "how many", "all of", "every", or "which ones lack". This is the dangerous one. Top-k retrieval has no notion of completeness. Ask how many enterprise customers churned in Q2 and it returns a specific, confident, correctly cited number computed from whatever five chunks ranked highest. Route these to analytics.

A lookup depends on the result of a previous lookup. "Who manages the person who wrote the auth RFC?" The second query cannot be constructed until the first returns. That needs iterative retrieval, not a cleverer single search.

The answer lives in a few passages, scattered, and nobody can say which in advance. That is real RAG. "Which of our past incident postmortems describe this failure mode?" There is nothing to filter on. Build the pipeline.

The cost of building it anyway

A retrieval pipeline is a chunking strategy, a reindexing job, an evaluation harness, and an embedding migration path. Every one of those is real engineering that needs maintaining forever. An index that drifts from its source. Deletes that have to propagate. An embedding model that cannot be upgraded without re-embedding everything. A golden set somebody has to keep current.

That is a permanent tax, and it should be paid only when the problem demands it. Teams routinely pay it for a corpus that fits in a context window and changes twice a year.

The honest default at moderate query volume: filter, then stuff. Build the boring thing. The signal that it is no longer enough will be specific rather than vague.

The question to ask first

Before tools, before embedding models, before vector databases:

How much of this corpus does a single query actually need to see?

If the answer is "one document," the problem is much easier than it looks.

If the answer is "no idea, it could be anywhere," now retrieval is worth discussing.

Visual walkthrough

Explore the complete decision framework:

https://sagartrivedi.dev/learn-rag/#s11

Chapter 11 explains why the per-query working set matters more than total corpus size. Chapter 12 maps the five common product shapes and their predictable failure modes.

What is the most over-engineered data pipeline you have encountered? Bonus points if the corpus never changed.