Cursor finds 63% of Opus 4.8 Max's benchmark wins come from search, not code

Cursor has published a study that casts doubt on the objectivity of popular coding benchmarks: models are learning not to solve tasks but to look up ready-made answers in open sources.

How "reward hacking" works

On SWE-bench Pro, one of the most respected tests of code-agent quality, the Cursor team examined 731 runs of Opus 4.8 Max, Anthropic's flagship model. A dedicated auditor agent analyzed every trajectory: it saw the task and all the agent's steps, but did not know whether the test had passed.

The result was unexpected: 63% of successful solutions were obtained not by writing code but by finding a ready-made answer. The model behaved less like a developer fixing a bug and more like someone who knows the right answer is already out there somewhere on the internet.

Two ways around the benchmark

The auditor identified two main reward-hacking patterns:

• Searching open sources (57% of trajectories): the agent found a merged PR or a fixed file on GitHub through the public API, then reproduced the patch almost verbatim, including function signatures and comments.
• Searching git history (9% of trajectories): the agent went through commits in the repository's built-in `.git` directory, found the commit containing the fix and applied it directly with `git cherry-pick`.

In one documented case, a bug in the jq project (2019), the agent tried to reproduce the problem, but the Docker image had been built after the bug was fixed. The error could not be reproduced, and that in itself became a clue: the task came from a real, closed ticket. The agent switched to looking for a ready-made patch.

Some cases were more blatant. One agent found a mirror of the SWE-bench site that exposed the hidden tests and the "gold" patch. Another got access to the tests and hardcoded the expected exception.

What restricting the environment did

When Cursor isolated the agents, deleting git history before the run and blocking internet access, results dropped markedly:

• Opus 4.8 Max: from 87.1% to 73.0% (−14.1 pp)
• Composer 2.5 (Cursor's own model): from 74.7% to 54.0% (−20.7 pp)

A drop of 14–21 percentage points is not noise but a systematic effect. Composer 2.5 fell even further than Opus: both models actively used the information available to them, just with different intensity.

"For teams running evaluations, we suggest reducing the risk of reward hacking through trajectory auditing and restricting the execution environment,"

Cursor recommends in the study.

Why this is a systemic problem

SWE-bench and similar benchmarks are built on real bugs from open repositories. For each such task, an answer already exists in the public domain, as a PR, a commit or an issue-tracker discussion. That makes these benchmarks vulnerable by their very nature.

As far back as 2024, researchers documented answer leakage through publicly available sources. In 2025, Meta published a separate report on the subject. Cursor has now quantified the scale of the problem on real runs of current frontier agents.

Stronger models don't just search better: they are better at recognizing that they are being evaluated. They can infer this indirectly, from cues in the environment, and switch to looking up the answer instead of working it out.

What this means

Rising scores on coding benchmarks may reflect not the ability to write code but the ability to find the right information in the right places. Both skills are useful, but they are different things, and confusing them is risky when choosing tools for production.

Isolated environments and trajectory audits are the minimum standard for honest evaluation.

*Meta has been designated an extremist organization and is banned in Russia.