Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Five AI Models Tried to Fix the Same Bug: Two Succeeded, Three Broke More Than They Fixed

A Habr author gave five AI models the same site-publishing bug. Claude Code and Codex fixed it; the other three broke even more. But you can't pick the best model from a single bug: even on SWE-bench Verified, with 500 tasks, the top ten differ on only 164 of them. A fix that quietly breaks neighboring code is more dangerous than an unfixed bug, because it gets found later. So accept a fix based on the diff and your own test run, not on the AI's report.

AI-processed from Habr AI; edited by Hamidun News
Five AI Models Tried to Fix the Same Bug: Two Succeeded, Three Broke More Than They Fixed
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Key metric #1On Verified, the two leaders each solve 396 of 500 tasks
Key metric #2The same model swings by up to 29.8 percentage points across different harnesses
Key metric #3Strengthened tests cut the share of solved tasks for the top-10 agents by 4.2–9.0%

One bug, five AI models, two outcomes. A Habr author gave the same site-publishing bug fix to five AI models. Claude Code and Codex fixed the bug. The other three left things worse than before. It is tempting to crown the first two as the best, but the author himself resists that temptation. The experiment does show clearly, though, which kind of error is more dangerous and how to catch it before a fix lands in your project.

Why One Bug Can't Tell You Which AI Model Is Best

A single experiment shows how a particular model works, not how it ranks. The large benchmarks confirm this. SWE-bench Verified is a set of 500 tasks with automatic checking, drawn from real GitHub issues and filtered by humans together with OpenAI. The two top results each solve 396 of the 500 tasks. The top ten differ in results on only 164 tasks (Liu et al., arXiv). A paired test, which checks whether the gap between neighbors on the leaderboard is just chance, did not separate a single one of the 29 adjacent pairs in the top thirty.

There is a second factor: the harness. This is the program wrapped around the model, through which it reads project files and runs commands. According to the same authors, a single model's results vary by up to 29.8 percentage points in the share of solved tasks depending on the harness. Across the entire top thirty, the spread is just 8.8 points. In other words, the harness shifts results roughly three times more than the gap between first and thirtieth place. If 500 tasks can't separate neighbors, one bug certainly can't: on a different task, the picture could look different.

Which Is Worse: An Unfixed Bug or a Silent Regression

An unfixed bug is visible right away. What's dangerous is a fix after which something that used to work stops working. This kind of breakage is called a regression, and it is usually found later.

There are two causes. The first is excess changes. The agent touches more than it was asked to and doesn't check the rest. Anthropic says it plainly in the Claude Code documentation: if the agent has nothing to run to tell whether the work is done, it stops when the work looks done. The role of testing then falls to the human. The SlopCodeBench benchmark (covered by TechRadar) found that agent-written code is 2.2 times more verbose and more convoluted than code in 48 human-maintained Python projects. Verbosity grew over the course of the task in 89.8% of runs, and convolutedness grew in 80%. The more lines a fix touches, the more room there is for a regression.

The second cause is weak tests. Code can pass every automated test and still miss important cases. In the Probe to Generate study, researchers strengthened the tests and re-evaluated the ten best code-repair agents. The share of solved tasks fell by 4.2–9.0%. Some previously accepted fixes had slipped through gaps in the tests and did not do everything the task required. The conclusion is simple: an AI report saying "fixed" proves nothing. What you need to check is the changes and the test results.

How to Constrain the Agent and Verify Its Fix

A narrow task description reduces risk before work even begins. What to specify:

  • the file and function where the bug lives;
  • how to reproduce the bug and what the expected result is;
  • an explicit ban on touching the rest of the code;
  • a request to show a plan first and write code only after you reply;
  • a test that fails before the fix and passes after it.

It's convenient to keep standing prohibitions in an agent rules file, so you don't repeat them in every task. This matters all the more because in Claude Code, new sessions in the terminal and VS Code start in auto mode by default, where commands run without confirmation of each one. The less often you confirm steps, the more closely you need to examine the result.

Step-by-step fix verification:

  1. Open the full diff and look for files and functions you did not ask the agent to touch. Three lines in one function and two hundred lines across six files carry different risk, even though both changes are called a fix.
  2. Compare the changes against the task. A renamed variable, a rewritten neighboring function, or a deleted input check is a reason to send the fix back.
  3. Make sure there is a test for the bug itself. If not, ask the agent to write one.
  4. Run the full test suite yourself and check whether the agent modified the tests themselves. If the project has no tests, you first need a test that reproduces the bug.
  5. Repeat the run after every revision: a new fix can affect something that passed last time.
  6. Bring in a second pair of eyes. The Claude Code documentation advises assigning a subagent, meaning a separate instance of the model, to review the diff against the spec and edge cases, with no comments on style. It also describes a stop hook: a script that sends the agent back to work until the check passes.

How to Choose a Model for Your Own Project

You can't name a winner from a single bug. The result comes from a model-and-harness pair, and you configure much of the environment yourself: the confirmation mode, the rules file, the checks. It's more practical to hand fixes to the model whose work is easiest to verify on your tasks: one that changes few lines, writes a test for the bug, and stays out of files it wasn't asked to touch. You can see this after just a few tasks in your own repository, and a leaderboard won't help here.

The story has an honest limitation: it is a single experiment by a single author, and the author himself notes that on a different bug, things might have gone differently. But the practical takeaway is no weaker for it. The more dangerous model is the one that confidently reports a fix while it has disturbed neighboring code. The decision on a fix should be made by a human, based on the changed lines and on tests they ran themselves.

Story

Key Questions Answered

Why One Bug Can't Tell You Which AI Model Is Best?

A single experiment shows how a particular model works, not how it ranks. The large benchmarks confirm this. SWE-bench Verified is a set of 500 tasks with automatic checking, drawn from real GitHub issues and filtered by humans together with OpenAI. The two top results each solve 396 of the 500 tasks. The top ten differ in results on only 164 tasks (Liu et al., arXiv).

Which Is Worse: An Unfixed Bug or a Silent Regression?

An unfixed bug is visible right away. What's dangerous is a fix after which something that used to work stops working. This kind of breakage is called a regression, and it is usually found later. There are two causes. The first is excess changes. The agent touches more than it was asked to and doesn't check the rest. Anthropic says it plainly in the Claude Code documentation: if the agent has nothing to run to tell whether the work is done, it stops when the work looks done.

How to Constrain the Agent and Verify Its Fix?

A narrow task description reduces risk before work even begins. What to specify: the file and function where the bug lives; how to reproduce the bug and what the expected result is; an explicit ban on touching the rest of the code; a request to show a plan first and write code only after you reply; a test that fails before the fix and passes after it. It's convenient to keep standing prohibitions in an agent rules file, so you don't repeat them in every task.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…