arXiv Paper Recalculates METR: AI Slowed Developers by 19–24%

How much time does an AI assistant really save? In an 18-page paper on arXiv (cs.AI, October 23, 2026), Solomon Shalom Lijo argues that a stopwatch reading is not an adequate answer. He recalculated the data from the METR study, estimated the ceiling on GPT-5's speedup in the GDPval benchmark, and ran his own experiment with AI reviewers. On the same 246 tasks, developers turned out to be 19% or 24% slower, depending on what exactly you count as the effect. And when a reviewer was told that the draft was written by a nearly error-free model, they checked less and missed a wrong figure in 22.5% of briefs.

Stopwatch vs. ledger

The author distinguishes two interpretations. In the "stopwatch" view, time saved equals what the clock recorded. In the "ledger" view, it is a counterfactual quantity: how long the work would have taken without AI, read alongside the errors the work let through. Lijo defends the second view.

To do this, he introduces a pair of model-independent quantities: time saved and excess missed defects. From them he builds a single number for decisions, the break-even defect price threshold. If an error in your process costs more than the threshold, the speedup is not worth it.

The baseline verification model yields three conclusions. As long as the model is not good enough to skip checking it altogether, a calibrated worker's share of missed defects does not depend on model quality. Verification time grows with the logarithm of the model's error rate. And overtrust buys minutes on the clock at the cost of even more minutes in the overall tally.

The METR recalculation and the GDPval ceiling

Lijo reproduced the METR developer study from public data. The result depends on which quantity you choose to estimate:

• a 19% or 24% slowdown on the same 246 tasks, depending on the estimand;
• a 34% slowdown if you do not correct for random imbalance between groups;
• AI helped on somewhere between 3% and 82% of tasks. An interval this wide shows that a randomized trial answers the question about the average effect and says almost nothing about the effect on an individual task.

In the GDPval process, where every output is checked, review time alone caps GPT-5's speedup at 3.6x. Consultants outside the frontier worked faster but made more errors. Here the author sets explicit bounds instead of importing point assumptions.

Reviewers who believed the author

Lijo tested two predictions of his model in a small pre-registered experiment. AI reviewers checked AI-written drafts of uniform briefs built from SEC documents filed after the models' training data was collected, so model memory could not supply the answer.

When the reviewer was told the author's real error rate, they checked more as that rate rose. But it worked on an all-or-nothing basis: the switch happened near the point where costs converged in their prompt. There was no gradual adjustment of effort.

When they were told the author made errors ten times less often than it actually did, they checked 7 fewer of the 9 figures. A wrong figure got through in 22.5% of briefs, and without this deception it got through in none. The work became faster and worse, entirely because of reduced effort.

What to change in adoption metrics

If you are evaluating an AI assistant for your team, a single speed metric is not enough. You need at least three numbers:

• time saved relative to a counterfactual process without AI;
• the number of excess missed defects per unit of work;
• the price of a defect in your process, compared against the break-even threshold.

Separately, watch what you tell reviewers about the model's reliability. An inflated quality estimate reduces the amount of checking faster than the model actually improves. In tokens and engineer-hours, this looks like savings until the defect reaches the customer. The author has posted the code and data on Zenodo, so you can repeat the calculations on your own logs.