RAG Failures: 6 Reasons Your AI Agent's Memory Lags, and How to Diagnose Retrieval

RAG (Retrieval-Augmented Generation) pulls in an external knowledge base to support LLM answers, promising to free the model from hallucinations. In practice, 73% of RAG system breakdowns stem from bad search, and 40% of pipelines can't retrieve the right data at all. Engineers often try to treat the symptoms — swapping embedding models, bumping up top-k, upgrading the LLM. But if the wrong context made it into the prompt in the first place, the retrieval layer is broken, and the hallucination is just a downstream effect of that error.

The RAGGED study found that LLaMA and Claude peak at just 5 chunks, after which quality drops sharply. Claude performed best with a single, highly precise fragment. Padding the context to the max turns out to be harmful.

Six Points of Failure

1. Chunking destroys context — fixed 512-token splitting separates an entity from the sentence that qualifies it, a chunk without its heading loses meaning, and a 200-row table turns into noise. Solution: splitting along document structure, carrying headings into every chunk, and contextual embeddings. According to Anthropic, these cut errors by 35%, by 49% with contextual BM25, and by 67% with reranking.

2. Semantic gap — embeddings fail to capture intent. A user asks "How do I cancel my subscription?" while the document is titled "Account Termination Policy." Solution: hybrid search (embeddings + BM25 via Reciprocal Rank Fusion with k=60), query reformulation, and HyDE (searching against a hypothetical answer).

3. Retrieval-generation mismatch — increasing top-k improves Hit@k, but answer quality then drops. Liu et al.'s "Lost in the Middle" study showed the model makes better use of information at the beginning and end of the context. The optimal top-k value has to be tuned experimentally; RAGGED found the optimum ranges from k ≤ 1 to k ≥ 10. The recall threshold is 85% — below that, retrieval needs fixing.

4. Multi-hop reasoning — RAG can't reason across a chain of facts. The question "Who is the CEO of the company that was acquired by a competitor's supplier?" requires four search steps. Solution: iterative (agentic) RAG or graph-based memory (Neo4j, Cognee, Microsoft GraphRAG). The cost is added latency and expense.

5. No sense of time — the index doesn't know a document is outdated. Solution: metadata filtering, versioning, automatic reindexing, and quality drift monitoring.

6. Entity resolution — "Apple" (the company or the fruit?), "Ivanov A.V." and "A. Ivanov" look like different entities. Solution: extract entities during indexing, build a linked index, and search through a graph.

When You Don't Need RAG

A query like "show me all customers with churn risk above 0.7" is SQL, not search. RAG can't do aggregations, sorting, or numeric filtering. You need Text-to-SQL or a semantic layer over the data store instead. Generic detectors miss 43% of computational errors.

Diagnosis: Three Metrics

Context Relevance catches chunking and retrieval errors (the wrong fragments came back). Context Recall catches omissions (the chunk with the answer never came back). Groundedness catches hallucinations despite correct context. The key question: is retrieval broken or is generation? These are different engineering problems with different fixes.