METR: Anthropic's Models Are Improving Twice as Fast as a Year Ago, Largely Thanks to AI

Imagine a team of developers in which every engineer suddenly started working several times faster thanks to their own product. According to a new METR report, that is exactly what is happening at Anthropic right now. The company uses AI to build more powerful AI, and gets a noticeable effect.

What METR Found

METR researchers worked inside the company for 11 weeks and studied how AI affects its own development. The main conclusion: Anthropic's models are improving about twice as fast as a year ago. By benchmarks, over the past year the company made roughly two years of historical progress.

An important detail: METR is confident this is not simply a matter of buying more compute. Scale used to be the main engine of progress. Now, in the researchers' assessment, most of the acceleration as of October is linked to AI itself, and its contribution is growing.

This is supported by Slack surveys and interviews. Employees who work with the strongest internal model rate their productivity on average at roughly 12 times higher than without AI. That is about double what they said a few months ago, when they were using Mythos 5. These are subjective estimates, but the trend is noticeable.

For now, the acceleration mostly comes from AI helping human researchers rather than working fully on its own. However, METR warns there are signs that the role of autonomous contributions will soon grow. The researchers found no clear signs of diminishing returns or bottlenecks that would prevent further acceleration.

Models We Don't See

A significant part of the gain comes from internal models unavailable to the public. Over the past six months, Anthropic finished training three new models that have not yet been released. METR calls them Model 2, Model 3 and Model 4 (these are placeholder code names; inside the company they are different). Training finished in July, in August–September, and in September–October, respectively.

For practitioners this means something simple: what developers use through the API lags behind what is used inside the lab. When planning a product a year ahead, you should assume model capabilities will grow faster than in last year's forecasts.

Why Researchers Are Worried

Anthropic was allowed to edit the final report, and it did so extensively. According to METR, most of the most informative evidence from the full report of roughly 100 pages was struck from the public version. Some data requests were declined, and the company canceled a check of how much compute goes to safety rather than to boosting capabilities.

Anthropic co-founder Sam McCandlish acknowledged that this makes verification harder, but explained the decision by commercial sensitivity and the risk of handing competitors a "roadmap" to acceleration. The company promises to publish its own report in the coming weeks.

Critics point to a contradiction: Anthropic talks about safety louder than others, yet is itself accelerating. Daniel Kokotajlo accused it of "recklessly gambling with our lives." Marius Hobbhahn of Apollo Research believes the pace is even faster than the AI 2027 scenario and advises "preparing for madness." Legislator Alex Bores called on Congress to ban recursive self-improvement.

What to Do About It

First, don't treat current model capabilities as a ceiling: plan products with headroom for rapid quality growth and shorter cycles between releases. Second, watch regulation: pressure on labs is increasing, and transparency requirements may affect everyone building on frontier models. Third, treat self-reported productivity figures with caution: the 12x increase is based on surveys, and a significant part of the evidence is hidden. The public has to rely on METR's assessment rather than verify the data itself.