Stale Facts in Agent Memory Drop Qwen2.5 Accuracy from 100% to 44.6%

More retrieved context does not mean a better answer. Researchers from the group of Oleeviya Babu Poikarayil (seven authors) demonstrated this on the FactConsolidation task from the MemoryAgentBench benchmark. Qwen2.5-7B-Instruct with BM25 retrieval, working over a 6K-token history, answers correctly 45% of the time when given 64 tokens of retrieved evidence. At 512 tokens, accuracy falls to 32%. This happens even though an audit with gold annotations found the most recent statement in every retrieved set, without exception.

Where the chain breaks: retrieval or reading

Many agent memory benchmarks report a single overall score for the combined "retrieve and answer" pipeline. When the score drops, it is unclear whether retrieval failed to surface the update or the model received it and still did not use it. The authors separated the two steps. On the single-hop variant of FactConsolidation, they checked whether the current version of a fact was among the retrieved fragments. It always was. That means the problem lies in how the model reads the retrieved results.

Most errors came on questions where the outdated version of a statement was retrieved alongside the new one. A larger budget raises the chance that an old record about the same entity slips into the context. The model sees two conflicting statements and often picks the wrong one.

Controlled experiment: stale fact versus neutral one

To rule out the effect of length, the authors fixed the prompt and the evidence size at four statements. In one configuration, one of the statements was irrelevant. In the other, it was replaced with an outdated competing version of the needed fact. The context volume stayed the same; only the nature of a single statement changed.

• Qwen2.5-7B-Instruct: accuracy falls from 100% to 44.6%.
• Qwen3-14B: a drop from 100% to 87.8%.
• All 50 harmful paired flips (the answer was correct and became incorrect) were classified as stale, meaning the answer was taken from the outdated statement.

The authors' conclusion: an outdated statement hurts more than neutral evidence of the same size. The extra tokens are not to blame on their own. The culprit is conflicting content that looks relevant.

Replication on a long history

On a 32K-token history, the picture was the same. BM25 with a 64-token budget scores 69%, while a 512-token budget scores only 58%. Again, the narrow retrieval beats the wide one. The paper was accepted to the NeurIPS 2026 Workshop on Personalized, Aligned, Long-Term Memory for AI Systems (PALM) and runs 23 pages including appendices.

What to change in agent systems

These are conclusions drawn from the paper's results, not recipes the authors themselves tested. But they suggest themselves.

• Metrics. Measure retrieval recall (is the current fact present) and reading quality (was it used) separately. A single end-to-end score masks the cause of failure.
• Context budget. Tune it for the specific task. On both tests, 64 tokens produced a better result than 512, and was also cheaper in tokens per call. For agents with long sessions, that saving accumulates.
• Memory writes. If a new statement replaces an old one, mark the old one as superseded or delete it at write time, rather than hoping the model sorts it out at read time. Timestamps and version labels in the retrieved results should help too.
• Post-processing of retrieved results. Before passing results to the model, it makes sense to group statements about the same entity and keep only the most recent one. This matters especially for small models: the gap between Qwen2.5-7B and Qwen3-14B shows that a stronger model is more robust, but not immune.

For those building personal assistants and coding agents with long-term memory, users constantly change preferences, addresses, and settings, so outdated versions inevitably pile up. Widening the retrieval window "just in case" turns out to be riskier than it seems. The first thing to consider is what goes into storage at all and how it will be cleaned up.