SWE-Duel: Coding Agents Plant Bugs for Each Other, and GPT-5.5 Fixes 98.3%

Static coding benchmarks go stale fast: their tasks end up in training sets, and strong agents hit the ceiling. The authors of the SWE-Duel preprint (arXiv cs.CL) propose a different approach: an evaluation with no pre-built content. Two agents take turns creating tasks for each other and checking each other's work. The 1500-turn tournament featured 11 model-plus-harness pairings across 15 repositories in five languages. GPT-5.5 with Codex CLI took first place, fixing 98.3% of the defects that reach it for review.

How a duel turn works

In each turn, the Red agent works with a real repository across three sessions, each isolated by context. First, it invents and implements a new feature along with tests for it. Then it embeds a subtle bug in that feature and writes sealed tests that catch the bug. The result is a pull request.

The PR enters the task set only if two conditions are met. First, it must pass the feature tests and the repository's existing tests, while failing exactly the bug tests. Second, a fresh session of the same agent must be able to fix the bug blind. The second condition filters out unsolvable tasks: if the author cannot fix its own planted bug, the task does not count.

Then Blue steps in. It is not told that the PR contains a bug; it simply reviews the code. A win counts only if its fix passes all three test suites. After that, the agents swap roles. As a result, difficulty is not fixed at release but grows with the strength of the participants.

Who wins, and how

The leaders win in different ways. The authors single out two profiles: the strongest solver and the strongest task author.

• GPT-5.5 + Codex CLI: first place, best solver, fixes 98.3% of verifiable defects.
• Claude Opus 4.8 + mini-swe-agent: second place, best task author, Red win rate of 27.3%.
• Tournament scale: 11 participants, 15 repositories, five languages, 1500 turns.
• Within a single model family, tasks from the newest version beat a fixed reference solver 42.9% of the time, versus 4.0% for tasks from older versions.

The last figure is arguably the most interesting. It tentatively suggests that task difficulty grows with a model's ability to devise tasks. For a benchmark that is supposed to "grow on its own," that is the key property. If it holds up across more model families, the saturation ceiling stops being a problem: stronger authors will automatically set harder exams.

It is also worth remembering that the participants here are model-plus-harness pairs. The ranking reflects the quality of the whole combination, not just the model weights. The authors themselves call the result on rising difficulty preliminary, which is a fair caveat: the data comes from a single tournament.

What you can put to use

The idea works even without an arena of 11 participants. A team choosing a coding agent can run it through the duel scheme on its own repository. One agent writes a feature with tests and hides a bug, while a second agent or a fresh session tries to find and fix it. A public leaderboard won't show you this: on your own tasks, it may already be "baked into" the models' training.

The practical takeaways are as follows. Separate your metrics: the ability to fix someone else's code and the ability to devise hard tasks measure different skills, and the leaders in each do not coincide. The task admission filter (passes all tests except the bug's, and is solvable blind) is worth adopting as a standalone quality-control template for any synthetic evaluation. And when comparing products, look at the specific model-plus-harness combination, not just the model name.