Naive RAG Beat Agentic Memory Systems in Six of Seven Tests

A team of researchers tested whether LLM memory needs all the complexity built into it. As a rival to sophisticated agentic systems, they chose Raw-RAG: the simplest possible pipeline, in which dense retrieval and reranking run over unprocessed conversation history and feed the retrieved fragments to a reader model. On four long-term memory benchmarks (LoCoMo, LongMemEval, MEME and MemoryAgentBench), this baseline ranked first among the compared systems in six of seven benchmark–model pairs. It beat the strongest agentic system by 0.13 to 5.53 points.

What Exactly Was Compared

Agentic memory systems have piled on mechanisms in recent years. They rewrite entries, build structure into storage, train retrieval strategies and perform other complex operations. The rationale is clear: a dialogue hundreds of turns long does not fit in context, so it has to be compressed, organized and searched carefully.

Raw-RAG takes the opposite approach. Nobody summarizes or restructures the history. It is stored as is, retrieval finds relevant chunks, a reranker orders them, and the reader LLM answers the question. The authors deliberately made this design as straightforward as possible to get an honest reference point.

In places the margin is small; the lower bound is just 0.13 points. But the point is the direction, not the size: a system with no mutable memory, no structure and no trained retrieval came out ahead almost everywhere.

Why Raw History Works

The authors did not stop at the results table. They analyzed where the baseline's competitiveness comes from. Their evidence analysis showed that Raw-RAG wins on coverage. It more often retrieves all the evidence needed to answer, and that outweighs the reader's lower efficiency in using it.

One detail about the evidence itself matters. Useful information includes more than final values such as a date or a name. It also includes the relationships between facts and the dependency paths from which the answer is derived. When memory is converted into compressed entries or a structure, these links are easily lost.

The most telling observation: even full access to the transformed stores leaves an accuracy gap compared with raw evidence. In other words, the problem is not only that agentic retrieval sometimes misses. The transformation itself discards some information that cannot be recovered later.

Key Data

• Benchmarks: LoCoMo, LongMemEval, MEME, MemoryAgentBench.
• Raw-RAG pipeline: dense retrieval, reranking, reader model.
• Result: first place in six of seven benchmark–model setups.
• Margin over the strongest agentic memory: 0.13–5.53 points.
• Mechanism: higher coverage of the needed evidence, with less efficient use of it.
• The gap relative to raw data persists even with full access to the transformed memory.

What to Change in Practice

The first implication concerns evaluation methodology. It now makes sense to compare any new memory system against Raw-RAG, and the authors explicitly call it a mandatory baseline. If a complex architecture does not beat retrieval over raw history, its cost and extra points of failure are unjustified.

The second implication concerns engineering. Before building a layer that rewrites and reorganizes an agent's memory, it makes sense to store the original turns and measure what share of the needed evidence retrieval finds. If coverage is high but answers are still weak, the problem lies with the reader. If coverage is low, improve retrieval rather than rebuilding the storage.

Third: compression is not free. If a system still needs summarization or structuring, keep the original text alongside as a fallback source, so that relationships and dependency chains can be recovered.

The conclusions have a limitation, which the authors state carefully: they apply to the tested long-dialogue understanding tasks. The authors do not claim that sophisticated memory is useless everywhere. But for this class of tasks, the default assumption changes: it is the added complexity that must first be shown to beat the simple baseline, not the other way around.