RAG is not a chatbot: the five ways retrieval systems actually fail
Most retrieval-augmented generation demos work and most production systems do not. The gap is almost never the model.
A retrieval-augmented generation demo takes an afternoon. A production one takes a quarter. The difference is not prompt quality — it is that a demo is measured by whether it answered, and a system is measured by whether it was right.
Here is where they break, roughly in order of how often we see it.
1. Chunking destroys the answer
The most common failure happens before a model is involved. A fixed 500-token window cuts a table in half, separates a clause from its definition, or strips the heading that gave a paragraph its meaning.
Chunk on the document's own structure — sections, clauses, headings — and keep the parent heading in the chunk text. If your source is a contract or a statute, the section boundary is the chunk boundary.
2. Retrieval optimised for recall, answered with precision
Pulling the top twenty chunks feels safe. It is not. The model now has nineteen near-misses competing with the one correct passage, and it will happily blend them.
Retrieve broadly, then rerank, then pass three to five chunks. A cross-encoder reranker is usually the highest-return component you can add to an existing pipeline.
3. Semantic search alone misses exact terms
Embeddings are excellent at meaning and poor at identifiers. Query for a specific section number, part code or surname, and a pure vector search will return things that are about the same topic while missing the exact match.
Run hybrid retrieval: BM25 for lexical precision, embeddings for meaning, then fuse the results. This one change fixes a surprising share of "it cannot find the thing I literally named" complaints.
4. No refusal path
If retrieval returns nothing relevant, a well-behaved system says so. Most systems instead pass weak context to a model that has been trained to be helpful, and helpfulness fills the gap with something plausible.
Set a relevance floor on the reranker score. Below it, return "not found in the sources" and show what was searched. In legal, medical or financial contexts this is the difference between a tool and a liability.
5. No evaluation set, so no idea if changes help
Teams tune prompts by vibes for months. Build a set of 100 to 200 real questions with known correct sources, and measure two things separately:
- Retrieval: was the right passage in the context at all?
- Generation: given the right passage, was the answer faithful to it?
Splitting the metric tells you which half to fix. Almost always, it is retrieval.
A rough order of work
- Fix chunking to follow document structure.
- Add hybrid retrieval.
- Add a reranker and cut the context down.
- Add a refusal threshold.
- Build the evaluation set — then go back to step one and actually measure it.
Notice how far down the list "change the model" appears. It is not on it.
The uncomfortable part
Retrieval quality is data engineering wearing an AI costume. Parsing PDFs properly, keeping metadata, handling tables, tracking document versions — that is the work, and it is why teams with strong data engineers ship better AI products than teams with strong prompt writers.