It's Not Length, It's Conflicting Entries: LLM Accuracy Drops From 97–100% to 11–21%

The usual assumption is that neural networks get lost in long documents simply because there is too much text. Researchers at the University of California, San Diego tested this on a fact-retrieval task and got a different answer. Picture a RAG system whose prompt contains two versions of the same config: an old one and a new one, with different values for the same parameter.

Entries like these, not the thousands of tokens themselves, break the answer. As long as the context of up to 32 thousand tokens is filled with neutral text, Qwen2.5-1.

5B answers correctly 97–100% of the time. When the same slots are taken up by entries in the format of the target fact, accuracy drops to 11–21%. The practical takeaway: before sending documents to the prompt, clean out conflicting duplicates rather than just shortening the document.

How the experiment works

The task is deliberately simple. A single line such as "the secret code of the river is 4821" is hidden in a long text, and at the end the model is asked what the river's code is. The line sits at a random depth, and the context length varies from 512 to 32 thousand tokens. The test was run on five open models ranging from 0.5 to 3 billion parameters, from four families, including Qwen2.5, Llama-3.2, SmolLM2 and OLMo-2.

The main design rule: the length and the position of the target line are always the same, and only what surrounds it changes. In one variant the surroundings are random numbers, random words or a single repeated sentence. In another they are entries that follow the template of the target line, with keys drawn from a set of 51 nouns. There is also a third variant: the same neutral text is hidden from the model's attention, but the token positions are preserved.

What the measurements showed

On neutral text, accuracy holds at 97–100% for four of the five models: up to 4 thousand tokens for all four, and up to 32 thousand for Qwen2.5-1.5B. The exception is OLMo-2-1B, which loses 0.57 nat at 2048 tokens of neutral text (91% accuracy). A nat here is a unit of loss in the probability of the correct answer: the higher it is, the worse the result.

Entries in the format of the target line push accuracy down to 11–21% at lengths from 8 to 32 thousand tokens. The most damaging case is an entry with the same key and a different value. A single such entry increases the model's loss 9–110 times, and each additional one adds more. Comparison on Qwen2.5-1.5B at 32 thousand tokens: neutral text costs 0.004 nat, hidden neutral text costs 1.20 nat, and competing entries cost 4.76 nat. Median losses with competitors on this model are 2.82, 3.94, 4.03 and 4.76 nat at 8, 16, 24 and 32 thousand tokens.

Inside the model, the authors tracked one quantity: the share of attention that stays on the target line. They measured it from the last query token, across the eight attention heads that look at this line most strongly. Across 126 473 runs, this share explains 35 to 58% of the variance in loss, while the logarithm of context length explains only 2–8%. The gap persists at lengths from 8 to 32 thousand tokens. The idea in short: similar entries draw attention away from the target, and this is what coincides with the errors.

Why "length hurts on its own" did not hold up

Earlier studies hid extra tokens from attention while keeping the positions, saw accuracy fall, and concluded that length is harmful. The authors reproduced this on 1000 MMLU questions with Llama-3.2-3B and Qwen2.5-7B-Instruct: with 30 thousand hidden tokens, losses were 11–18 points. But in the retrieval task, the same masking costs 46 points of accuracy at 32 thousand tokens, while the same text left visible costs nothing. That means this protocol does not measure the pure cost of length.

What to do about it

In practice, what matters is not the number of tokens but the number of statements that contradict the correct answer. Outdated versions of contracts, drafts and duplicate logs with different values are the most dangerous neighbors for the target passage. The authors have also published the code, the attention-measurement tool and all 126 thousand measurements on GitHub and Hugging Face, so on open models you can check the share of attention on the target passage yourself.

There are limitations. The models are small, from 0.5 to 3 billion parameters, and the task is a simple key-to-value lookup, not real enterprise documents. The link between attention and errors comes from statistics across runs. The conclusion therefore applies specifically to fact retrieval in long contexts, not to all tasks with long inputs.