METR Gets Time-Savings Estimates of 4–18%, but Admits the Measurement Is Unreliable
When you buy AI agents for your team, it is easy to confuse fast code typing with fast feature delivery. METR repeated its experiment with developers and obtained estimates of a 4–18% reduction in time, but acknowledged they are unreliable. Some people refused to work without AI, and participants held back tasks they did not want to do by hand. So it is too early to carry these percentages into a release plan: first count your own team's review, fixes, and costs.
AI-processed from Habr AI; edited by Hamidun News
An agent wrote in a minute code that used to take half a day. Great — provided the team did not then spend a day on review and fixes. This is exactly the gap between the speed of generation and the speed of finished work that METR tried to measure. The repeat experiment produced estimates of a 4–18% reduction in time, but the researchers deemed the result unreliable. Developers increasingly declined to participate if it required working without AI.
For a tech lead, the practical problem is this: what savings should be built into timelines and budgets? Imagine an API migration. The agent updated dozens of calls, a human checked compatibility, fixed the tests, and waited for review. All of that work has to be counted. And comparing it with doing the job manually gets harder when participants exclude in advance the tasks that are especially unpleasant to do on their own.
Why the Comparison Became Skewed
The follow-up study involved 57 developers. Under the experiment's terms, they described tasks in advance, after which a random draw determined whether AI was allowed. But 30–50% of surveyed participants said they held back some tasks: they did not want to be barred from using the model's help. At the same time, it became harder to recruit people willing to work without it.
This creates a skew even before measurement. Suppose a developer willingly puts a small bug in a familiar module into the draw, but leaves a bulk update of repetitive files to the agent outside the experiment. The researchers will measure the first job well, but will learn almost nothing about the second. The expected benefit of AI affects what ends up under the stopwatch at all.
For returning participants, the estimated time reduction was 18%; for new ones, 4%. Meanwhile, the confidence intervals allow for a slowdown as well. The authors consider it possible that the benefit is understated because of selection, but these data do not establish the exact size of the effect. The details are published in the METR report of February 24, 2026. The November retelling should be read with this date in mind: we are looking at the result of an earlier experiment.
Refusing to Work by Hand Does Not Prove a Speedup
In the source material, reluctance to go without AI is presented almost as a vote for its effectiveness. That is an understandable reaction, but it is not enough for calculating productivity. A person may value a tool for reduced fatigue, the ability to tackle unpleasant tasks, or convenient code search. All of this is useful, but by itself it does not show how much sooner a release will ship.
Preferences have standalone product value. If an assistant removes drudgery and makes the working day more bearable, that is a reasonable argument for buying it. A manager just needs to discuss convenience, quality, and timelines separately. Otherwise the promise of savings quietly turns into a demand to close more tickets, even though the team bought itself the chance to be less tired.
Nor can the result of METR's first study be transferred to any modern team. In that study, 16 experienced participants completed 246 tasks in familiar projects, and using AI increased time by 19%. This is an important warning about the gap between perception and measurement, but the specific tools, tasks, and participants' experience limit how far the result applies. It does not follow that every agent today slows down every developer.
Parallel agents introduce a separate accounting problem. While one changes code, a person works through another ticket, and a second agent runs tests. Forty minutes have passed: how much belongs to the first task? Without a rule chosen in advance, teams easily end up with attractive but incomparable reports. A person's active time and calendar time to completion answer different questions.
What to Measure Inside Your Team
You can start with a small set of repeating jobs: bug fixes, adding tests, dependency migrations. For each, define readiness in advance: code accepted, checks passed, review comments resolved. Then track human time, waiting, tool costs, and subsequent fixes. The proposed approach is a practical takeaway for teams, not a proven result of the METR experiment.
It is also useful to look separately at where the gain arises. An agent may cut the time to write tests but increase review; speed up an unfamiliar task but get in the way of the author of their own module. If you fold everything into one average figure, a manager loses exactly the information needed to choose tools and allocate work.
For a product manager, the bottom-line question is quite down-to-earth: how much does an accepted change cost, and how quickly does it deliver value to the user? Token spend describes tool consumption. To justify a budget, it has to be tied to results. Until such a link exists, percentages from someone else's study should be used as a reason to run your own measurement, not as a promise to the board of directors.
- Nov 12, 2026Claude in Chrome requires approval for every click on category 3 sites
- Nov 12, 2026Norwest raises record $4 billion fund, betting on agents, voice and physical AI
- Nov 12, 2026AWS shows how 100% retrieval precision can hide an AI agent's failure
- Nov 12, 2026Garage Eight dropped peer approvals for 69 MRs in an internal service using AI review
Key Questions Answered
Why the Comparison Became Skewed?
The follow-up study involved 57 developers. Under the experiment's terms, they described tasks in advance, after which a random draw determined whether AI was allowed. But 30–50% of surveyed participants said they held back some tasks: they did not want to be barred from using the model's help. At the same time, it became harder to recruit people willing to work without it. This creates a skew even before measurement.
What are the key details on Refusing to Work by Hand Does Not Prove a Speedup?
In the source material, reluctance to go without AI is presented almost as a vote for its effectiveness. That is an understandable reaction, but it is not enough for calculating productivity. A person may value a tool for reduced fatigue, the ability to tackle unpleasant tasks, or convenient code search. All of this is useful, but by itself it does not show how much sooner a release will ship. Preferences have standalone product value.
What to Measure Inside Your Team?
You can start with a small set of repeating jobs: bug fixes, adding tests, dependency migrations. For each, define readiness in advance: code accepted, checks passed, review comments resolved. Then track human time, waiting, tool costs, and subsequent fixes. The proposed approach is a practical takeaway for teams, not a proven result of the METR experiment. It is also useful to look separately at where the gain arises.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.