Competing Claims Cut Models' Retrieval Accuracy from 0.97–1.00 to 0.11–0.21 at 32K Tokens

The conventional wisdom is that language models degrade on long inputs because of length itself: the more tokens, the worse attention works. A team from the Department of Computer Science at the University of California, San Diego tested this on a retrieval task and reached the opposite conclusion. In retrieval, length costs almost nothing. Competition is what costs a lot: claims that resemble the target fact, and especially claims that contradict it. The experiments ran on five open models ranging from 0.5 to 3 billion parameters, drawn from four families including Qwen2.5, SmolLM2, OLMo-2 and Llama-3.2.

How the experiment works

The task is key-value retrieval. The context contains a sentence such as "the secret code of the river is 4821", the prompt ends with a question about the river's code, and the researchers measure the log-probability of the correct answer. The key is an ordinary noun; the value is a four-digit number.

Context length varies from 512 to 32K tokens, and the position of the evidence is chosen at random. For each run, however, the researchers build four contexts with the same evidence, the same query and the same total length. Only the surroundings differ: neutral numbers, random words or a repeated phrase; claims in the evidence template with keys drawn from 51 nouns; and the same tokens hidden from attention while positions are preserved.

The result is stark. Neutral filler costs nothing: for Qwen2.5-1.5B, accuracy stays within 0.96–1.00 up to 32K tokens, and the median loss does not exceed 0.006 nat (a nat is a unit of logarithmic loss). If claims in the evidence format take the same positions, accuracy drops to 0.11–0.21 in the 8K–32K range. For Qwen2.5-0.5B, the loss reaches 1.79–4.07 nat. At 32K, the median cost for Qwen2.5-1.5B is 0.004 nat for neutral text, 1.20 nat for masking and 4.76 nat for competing claims. The only notable exception among the neutral cases is OLMo-2-1B: at 2048 tokens it loses 0.57 nat, with accuracy of 0.91.

Most of the damage comes from claims that assign a different value to the queried key. A single such claim multiplies the loss by 9–110 times, and each additional one adds more. Their number grows with length: 0.8 on average at 512 tokens, 3.2 at 2048 and 50.5 at 32 768. From 4096 tokens onward, every context contains a conflicting claim.

Attention as a predictor of failure

During the same forward pass, the authors measured the share of attention that the eight layer–head pairs most attentive to the evidence give to its tokens. Across 126 473 runs, this quantity explains 35–58% of the variance in loss, whereas the logarithm of length explains only 2–8%. The gap persists in long contexts of 8K to 32K tokens. The intuition is simple: with softmax attention, the weight of the evidence is divided by the sum of the scores of all keys. A thousand dissimilar tokens barely increase the denominator, while a handful of close copies of the evidence can cut its share in half.

What's wrong with masking

Earlier work claimed that length hurts on its own: every distractor token was hidden from attention while positions were preserved, and a drop was still observed. The authors reproduced this setup on 1000 MMLU questions: Llama-3.2-3B and Qwen2.5-7B lose 11–18 points at 30K masked tokens. In retrieval, however, such masking costs 46 points of accuracy at 32K, while identical text left visible costs nothing. In other words, this particular experiment fails the visible-filler control and does not prove a pure length effect.

Practical takeaway

If degradation in retrieval is a matter of competition, the fix lies in selecting what goes into the context, not only in architecture. For RAG systems, this means prioritizing deduplication and the resolution of conflicting claims before the content reaches the model. The share of attention on the evidence can serve as a diagnostic for any open model. The authors do not claim to have discovered the phenomenon itself; instead, they connect measured behavior, context content and attention on the evidence within a single experiment. The code, the attention probe and all 126 thousand measurements are published on GitHub and Hugging Face.