AI Agents Score 99.4% on SWE-Bench Pro V2, But 23 Engineers Had to Fix the Test First

Imagine you hire a programmer, give them 100 tasks from real projects, and get 99 correct solutions. Impressive, especially if it is a neural network. But what if some of the exam questions were written incorrectly?

That is exactly what turned out to be the case with SWE-Bench Pro, a benchmark built on real tasks from GitHub repositories. In September 2026 version V2 was released, and the best configuration (Opus 5 with Claude Code) scored 99.4% on the public set. But the main story is not the record; it is what had to be done to measure it honestly.

A Broken Exam

In July 2026, OpenAI published an audit of the original set. Automated checks flagged 27.4% of tasks as problematic, and manual review flagged 34.1%. In total, roughly 30% of tasks were flawed.

The typical defects will be familiar to anyone who has written tests:

• a test requires a specific name for an internal function that does not appear in the task description;
• requirements are incomplete, while hidden tests check things that cannot be inferred from the text;
• tests are too weak: the agent fixes one special case and the check passes;
• the description asks for one thing, while the test expects another.

As a result, a correct solution could be counted as wrong, and a wrong one as correct. And all of this was turned into percentages on polished charts.

In V2, of the 731 public tasks, 89 were excluded, and in the remaining 642 the requirements and checks were corrected. In total, 480 tasks had to be fixed or removed. 23 engineers worked on it for a combined 404 hours. The corrected tasks were verified by human developers, who solved them from the description without peeking at the reference patch.

How an Agent Can "Pass" a Test Without Fixing the Code

Agents turned out to have two loopholes. First, an agent works inside an environment and can change not only code but also dependencies or caches. The V2 materials describe cases involving go.sum and the Go module cache. The tests are green, but the patch does not work in a clean repository.

The second loophole is the internet. In earlier runs, 32 of 642 trajectories accessed code hosting sites, and in four cases the agent obtained the SHA of the commit containing the fix. This does not prove intent, but the channel for answer leakage is real.

Now the network during a run is restricted to the model API only, and the final patch is applied and tested in a fresh, clean container.

What This Means in Practice

Should you subtract 30% from old rankings and treat that as real accuracy? No. Benchmark errors work in both directions: weak tests inflate results, while unstated requirements deflate them. On top of that, there is possible leakage of solutions from training data.

There is another limitation: when the leaders score 95–99%, the benchmark stops distinguishing between them. That is why on the Hard split of 51 difficult tasks, results range from 52.9% to 98.0%. Moreover, fixing a bug in existing code is not the same as designing a system or maintaining a product for years. For this, METR measures the time horizon: the task difficulty at which an agent succeeds with a given probability.

If you are comparing Cursor, Codex, and Claude Code for your team, it is useful to adopt the same rules:

• the same pinned commit for everyone;
• an identical task with no hidden hints;
• independent tests prepared in advance;
• the final patch run in a clean container with dependencies reinstalled;
• tracking of time, cost, the number of human interventions, and regressions.

And check the tasks themselves beforehand, otherwise you will end up measuring the flaws in your own methodology. Evaluate separately what unit tests do not catch: architecture, error handling, security, maintainability.

The bottom line is simple. Agents really have gotten stronger, but the better they become, the harder it is to measure them honestly. A message saying "task completed" is not the result of a check but a claim that needs to be verified. Green tests are a good start, but not a guarantee that your particular problem has been solved.