Why RAG and Long Prompts Get Dumber: One Conflicting Fact Drops Small-LLM Accuracy from 97% to 11%

The common assumption is that neural networks get lost in long documents simply because there is too much text. Researchers at the University of California, San Diego tested this on a fact-retrieval task and got a different answer. Imagine a RAG system whose prompt contains two versions of the same config: an old one and a new one, with different values for the same parameter.

Entries like these, not the thousands of tokens themselves, break the answer. While a context of up to 32 thousand tokens is filled with neutral text, Qwen2.5-1.

5B answers correctly in 97–100% of cases. When the same space is occupied by entries in the format of the target fact, accuracy drops to 11–21%. The practical takeaway: before sending content to the prompt, clean out conflicting duplicates rather than only shortening the document.

How the experiment works

The task is deliberately simple. A single line such as "the secret code of the river is 4821" is hidden in a long text, and at the end the model is asked what the river's code is. The line sits at a random depth, and the context length varies from 512 to 32 thousand tokens. The test was run on five open models ranging from 0.5 to 3 billion parameters, from four families, including Qwen2.5, Llama-3.2, SmolLM2 and OLMo-2.

The main design rule: the length and position of the target line are always the same; only what surrounds it changes. In one variant the filler is random numbers, random words or a single repeated sentence. In another, it is entries in the template of the target line, with keys drawn from a set of 51 nouns. A third variant hides the same neutral text from the model's attention but keeps the token positions.

What the measurements showed

On neutral text, accuracy stays at 97–100% for four of the five models: up to 4 thousand tokens for all four, and up to 32 thousand for Qwen2.5-1.5B. The exception is OLMo-2-1B: at 2048 tokens of neutral text it loses 0.57 nats (91% accuracy). A nat here is a unit of loss on the probability of the correct answer: the higher it is, the worse the result.

Entries in the format of the target line at lengths of 8 to 32 thousand tokens push accuracy down to 11–21%. The most harmful case is an entry with the same key and a different value. A single such entry raises the model's loss 9–110 times, and each additional one adds more. Comparison on Qwen2.5-1.5B at 32 thousand tokens: neutral text costs 0.004 nats, hidden neutral text costs 1.20 nats, and competing entries cost 4.76 nats. Median losses with competitors on this model are 2.82, 3.94, 4.03 and 4.76 nats at 8, 16, 24 and 32 thousand tokens.

Inside the model, the authors tracked one quantity: the share of attention that stays on the target line. They computed it over the eight attention heads that look at this line most strongly, from the last token of the query. Across 126,473 runs, this share explains 35 to 58% of the variance in loss, while the logarithm of context length explains only 2–8%. The gap holds at lengths from 8 to 32 thousand tokens. The idea in short: similar entries take attention away from the right one, and this is what coincides with the errors.

Why "length hurts by itself" did not hold up

Earlier studies hid extra tokens from attention while preserving positions and saw accuracy drop, from which they concluded that length is harmful. The authors reproduced this on 1000 MMLU questions with Llama-3.2-3B and Qwen2.5-7B-Instruct: at 30 thousand hidden tokens, the losses were 11–18 points. But in the retrieval task, the same masking costs 46 points of accuracy at 32 thousand tokens, while the same text left visible costs nothing. So this protocol does not measure the pure cost of length.

What to do about it

In practice, what matters is not the number of tokens but the number of statements that dispute the right answer. Outdated contract revisions, drafts and duplicate logs with different values are the most dangerous neighbors for the target passage. The authors also published the code, the attention-measurement tool and all 126 thousand measurements on GitHub and Hugging Face, so on open models you can check the share of attention on the target passage yourself.

There are limitations. The models are small, 0.5 to 3 billion parameters, and the task is simple value lookup by key, not real corporate documents. The link between attention and errors comes from statistics across runs. So the conclusion applies specifically to fact retrieval in long contexts, not to all tasks with long inputs.