Needle in a Haystack
Needle in a Haystack is a long-context evaluation test that measures whether a language model can accurately retrieve a specific planted fact (the "needle") from a large irrelevant text corpus (the "haystack") at varying positions and context lengths, revealing practical limits of long-context recall.
Needle in a Haystack is an evaluation methodology for long-context language models, popularized by independent researcher Greg Kamradt in November 2023. The test inserts a short, distinctive fact—the needle—at a precisely controlled position within a large block of unrelated filler text, such as concatenated news articles or Wikipedia passages. The model is then prompted to retrieve that specific fact. By repeating the procedure across a grid of total context lengths and needle depths (expressed as a percentage of total document length from 0% to 100%), the evaluator obtains a two-dimensional matrix of retrieval accuracy scores.
Results are typically rendered as a color-coded heatmap: context length on one axis, needle depth on the other, with color indicating whether the model correctly recalled the planted fact. This visualization exposes systematic failure modes that aggregate accuracy figures obscure. Many early long-context models showed sharp degradation when the needle was placed in the middle third of a long document—a pattern consistent with the independently documented "lost in the middle" effect, where attention mechanisms assign lower weight to tokens far from both ends of the context window. Models advertised as supporting 100K-token contexts frequently scored near chance at 50–70% depth for inputs exceeding roughly 32K tokens.
The test gained rapid adoption through 2024 because it provided an intuitive demonstration that advertised context window sizes and practically usable context lengths were often different quantities. Anthropic, Google, and OpenAI incorporated Needle in a Haystack results into technical reports for Claude 2.1, Gemini 1.5 Pro, and GPT-4 Turbo respectively, using full-window near-perfect recall as a differentiating claim. Gemini 1.5 Pro's reported near-perfect recall up to 1 million tokens in early 2024 attracted particular attention.
By 2025–2026, leading models largely achieve near-perfect single-needle retrieval across their full context windows, rendering the basic test insufficient as a standalone measure of long-context competence. Researchers have extended the methodology to multi-needle retrieval (several scattered facts, all must be recalled), needle-and-distractor setups (similar but incorrect facts also embedded), and cross-lingual haystacks. The original single-needle test persists as a baseline sanity check and a minimum bar for any system claiming long-context capability.