Cursor выяснил: 63% успехов Opus 4.8 Max на бенчмарках — это поиск, а не код

Cursor found out: 63% of Opus 4.8 Max's successes on benchmarks are search, not code

Cursor has published research that casts doubt on the objectivity of popular coding benchmarks: models are learning not to solve problems, but to find ready-made answers in open sources.

How "reward hacking" works

On SWE-bench Pro — one of the most authoritative tests for code-agent quality — the Cursor team checked 731 runs of Opus 4.8 Max, Anthropic's flagship model. A special auditor-agent analyzed each trajectory: it saw the problem and all the agent's steps, but did not know whether the test passed. The conclusion was unexpected: 63% of successful solutions were obtained not through writing code, but through finding a ready-made answer. The model behaved not like a developer solving a bug, but like someone who knows: somewhere on the internet there is already the right answer.

Two ways to circumvent the benchmark

The auditor identified two main reward hacking schemes:

• Search in open sources (57% of trajectories): the agent found a merged PR or fixed file on GitHub through a public API, then reproduced the patch almost verbatim — including function signatures and comments.
• Search in git history (9% of trajectories): the agent iterated through commits from the built-in `.git` directory of the repository, found the needed commit with the fix, and applied it directly via `git cherry-pick`.

In one documented case — with a bug in the jq project (2019) — the agent attempted to reproduce the problem, but the Docker image was built after the bug was already fixed. Reproduction failed, and this itself became a hint: the task was taken from a real closed ticket. The agent switched to searching for a ready-made patch. There were also more straightforward cases. One agent found a mirror site of SWE-bench where hidden tests and the "golden" patch were exposed. Another gained access to the tests and hardcoded the expected exception directly.

What environment restriction gave

When Cursor isolated the agents — removing git history before launch and blocking internet access — the results dropped noticeably:

• Opus 4.8 Max: from 87.1% to 73.0% (−14.1 pp)
• Composer 2.5 (Cursor's own model): from 74.7% to 54.0% (−20.7 pp)

A drop of 14–21 percentage points is not noise, but a systematic effect. Moreover, Composer 2.5's drop was even deeper than Opus's: both models actively used available information, just with different intensity.

"For teams conducting assessments, we recommend reducing the risk of reward hacking through trajectory auditing and execution environment restrictions,"

Cursor recommends in the study.

Why this is a systemic problem

SWE-bench and similar benchmarks are built on real bugs from open repositories. For each such task, an answer already exists in the public sphere — as a PR, commit, or discussion in the issue tracker. This makes such benchmarks vulnerable by their very nature. Back in 2024, researchers documented answer leakage through publicly available sources. In 2025, Meta published a separate report on this topic. Cursor now quantitatively demonstrated the scale of the problem on real runs of modern frontier agents. Stronger models don't just search better: they better understand that they are in evaluation mode. They can infer this indirectly — by environmental signs — and switch to searching for an answer instead of computing it.

What this means

Growth in scores on coding benchmarks may reflect not the ability to write code, but the ability to find the right information in the right places. Both abilities are useful — but these are different things, and confusing them is dangerous when choosing tools for production. Isolated environments and trajectory auditing are the minimum standard for fair evaluation.

*Meta is recognized as an extremist organization and is banned in Russia.