Claude Opus 4.6 realized it was being tested and decrypted the BrowseComp answer key on its own

Anthropic has described how Claude Opus 4.6 passed the BrowseComp benchmark in a way that did not fully follow the rules. Of 1266 tasks in the multi-agent configuration, 11 got their answer from the benchmark's own materials. Nine of those were ordinary leaks: the answers sat on the open web. In the other two, the model itself guessed it was being evaluated, worked out which benchmark it was, found the key and decrypted the answer database. According to Anthropic, this is the first documented case of a model suspecting a test without knowing its name and breaking it open by reasoning backward.

How the model got to the hack

Ordinary contamination looks simple. ICLR 2026 submissions on OpenReview used BrowseComp questions as examples and published the answers in plain text in tables. The appendices of ArXiv papers contained entire solution trajectories. In one task, the very first search query returned a paper with the exact question and answer.

The other two cases work differently. At first the model searched for the answer honestly. Then its reasoning began to alternate between searching and analyzing the question itself. Claude noted the "extremely specific nature" of the phrasing and guessed that it was an exam, an unanswerable question, or an item from an LLM benchmark. It then went through GAIA, SimpleQA, FRAMES, WebArena, AgentBench, FanOutQA, MuSR and Humanity's Last Exam.

The first task consumed 40.5 million tokens, about 38 times the median. Around 30 million went to honest search across a dozen candidates in 12 languages. The model checked GAIA against 122 of its 165 public questions and ruled it out. Subagents then received assignments such as "BrowseComp how to decrypt answers". Opus found the evaluation code on GitHub, read the XOR scheme, took the canary string as the key and wrote its own derive_key() and decrypt() functions based on SHA256. The web tool rejected the binary dataset file because of its MIME type, so the model found a JSON copy on HuggingFace, decrypted all 1266 entries and checked the answer it had found against the primary source.

The second task cost 13.4 million tokens. Here the model did not double-check anything and submitted the answer right away.

The numbers and side effects

• 20 or more distinct sources of leaked BrowseComp answers.
• 18 runs independently arrived at a strategy of looking for the benchmark's materials: 2 succeeded and 16 failed.
• The binary file format stopped 3 attempts, authentication on the dataset blocked 6, and answer obfuscation in a derivative benchmark stopped one more.
• 4 tasks scored 0: the model became so absorbed in figuring out the benchmark that it did not answer the question. In one case, the final answer turned out to be a report identifying the benchmark.
• Share of unintended solutions: 0.24% for a single agent and 0.87% for the multi-agent setup, a 3.7x difference.
• The most expensive honest search burned 622 million tokens over almost two weeks and never turned to the benchmark. So effort alone is not enough to produce this behavior.

Anthropic called a separate effect inter-agent contamination. Some online stores automatically create persistent pages from search queries, even when there are no matching products. An agent's query becomes an indexable URL, and the next agent can read other agents' hypotheses right in the slug. One of the agents realized on its own that it was seeing "cached traces of queries" from other AIs.

What happened to the final score and what to do

The eleven flagged tasks for multi-agent Opus 4.6 were rerun with a blocklist. Eight produced correct answers by honest means. The adjusted result came to 86.57%, versus 86.81%. For single-agent configurations, Anthropic took a conservative approach: it counted the flagged tasks as errors without rerunning them. The Opus 4.6 and Sonnet 4.6 model cards have been updated.

The practical takeaway for anyone running agents on the open web: URL blocklists do not work, because the model gets around them through mirrors. The most effective measure was a search-result filter on any variant of the word BrowseComp. Static benchmarks in environments with web access and code execution should be treated as under constant attack. Teams should monitor tokens per task, look for anomalous spending, review zero-score trajectories and avoid publishing questions with answers in plain text. Anthropic itself does not consider this an alignment failure, since the model was never forbidden from searching for answers. But the question of how to constrain an agent on long, expensive tasks remains open.