Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)
ReleaseInsight · On RadarAI Alignment Forum→ original

Redwood Research: AI speeds up our work 2–3x, not 5–10x as at Anthropic

METR estimates that AI speeds up Anthropic's progress by 1.7x, while estimates for research itself range from 5–12x. A researcher at Redwood Research expected a similar boost for outside teams, given that they have money for compute. In practice, Opus 5.5 and Claude Code deliver just 2–3x. He points to model weaknesses, differences in tasks, and limited human attention as possible explanations.

AI-processed from AI Alignment Forum; edited by Hamidun News
Redwood Research: AI speeds up our work 2–3x, not 5–10x as at Anthropic
Source: AI Alignment Forum. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAI Alignment Forum ↗
Editor-in-Chief
Key metric #1Agents sometimes make “insane” choices that invalidate results
Key metric #2The bottleneck is managerial judgment, not agent speed
Key metric #3If 5x is real, outside teams should soon flood us with publications

Everyone has grown accustomed to an appealing claim: AI speeds up AI development. But put that claim to work on an actual Monday morning, and the gains turn out to vary considerably between teams. A researcher at Redwood Research posed an uncomfortable question: if things are going so well at Anthropic, why aren't we seeing the same results with the same models?

Where did the 5–10x figure come from?

METR recently estimated that Anthropic's use of AI in its own work delivers an overall 1.7x speedup in progress on model capabilities. That is the aggregate result, implying that research itself has accelerated much more sharply. Ajeya Cotra suggests around 10x; Anthropic employees have cited roughly 12x. Company researchers reported a 5x speedup with Mythos 5. Outside teams have access to Opus 5.5, which the author says is slightly better.

The author draws a straightforward conclusion. If AI also accelerates safety research, Anthropic's internal specialists should be working roughly 10 times faster, and outside researchers should already be working 5 times faster. Outside teams also have an advantage: they can quickly increase their compute spending. Redwood is well funded and would happily spend 5 times as much if it had 5 times as many experiments to run. Faster research work should therefore translate into an equivalent speedup in projects.

In practice, the author puts his own gains closer to 2–3x. That is already substantial, but it is not 5x.

Three possible explanations

We have implemented AI poorly. Perhaps. But the team actively uses agents, primarily Claude Code and, more recently, Claude Tag. It also has its own framework for autonomous research, which has produced published results. The weaknesses are familiar: Claude sometimes makes “completely insane” decisions that invalidate the entire result, and it struggles to write the final report. If Anthropic had a framework that radically improved the models' usefulness, it would probably have turned it into a product. There could, however, be specialized internal tools that are not released externally. The author thinks fundamental model limitations are a more likely explanation than the framework.

The tasks are different. Capabilities research may be easier to accelerate. For example, if the main obstacle in experiments with a new architecture is implementing it efficiently, that plays directly to a model's strengths. One Anthropic employee told the author that capabilities work had accelerated considerably more than their near-term alignment work.

Judgment has become the bottleneck. Much of Redwood's research is done by interns and junior staff who work closely with a manager to set priorities. The people doing the work may indeed have become faster, but now their manager lacks the time to understand what they are doing. Anthropic may have more independent senior researchers, or its tasks may require fewer nuanced decisions.

There is also an example of a genuine leap. Ryan's investigation into Hugging Face was sped up “wildly” through intensive work with agents: the project involved reading an enormous amount of material, and Ryan himself is skilled at building agent workflows.

What this means for practitioners

First, speedup figures cannot simply be transferred between teams. The same model delivers 10x where the task is readily verifiable and experienced people are available, and 2–3x where nuanced decisions and manual oversight are required. If you are introducing agents, examine the bottlenecks: checking results, writing reports, and senior staff time.

Second, the main risk is an agent that confidently breaks the methodology. Fast experiments are useless if some of the conclusions are invalid. You need automated checks and a person who reviews both the final output and the process that produced it.

Third, the hypothesis has a testable implication. If empirical safety work really is accelerating by 5–10x, the flow of valuable publications from outside researchers should soon increase noticeably. The author explicitly writes that if gains fail to appear on the tasks most amenable to acceleration, he will conclude that researchers are doing something wrong. He also cites an Arcadia Impact survey in which outside specialists reported a speedup of roughly 1.6x.

The limitations are clear: this is one researcher's reasoning, not a measurement. Self-reported speed estimates are notoriously unreliable, and the author acknowledges that Anthropic's own figures initially struck him as poorly substantiated. Still, the question is useful: is the claimed boost a property of the model, or of the team, tasks, and processes surrounding it?

Story

Key Questions Answered

Where did the 5–10x figure come from?

METR recently estimated that Anthropic's use of AI in its own work delivers an overall 1.7x speedup in progress on model capabilities. That is the aggregate result, implying that research itself has accelerated much more sharply. Ajeya Cotra suggests around 10x; Anthropic employees have cited roughly 12x. Company researchers reported a 5x speedup with Mythos 5. Outside teams have access to Opus 5.5, which the author says is slightly better. The author draws a straightforward conclusion.

What are the key details on Three possible explanations?

We have implemented AI poorly. Perhaps. But the team actively uses agents, primarily Claude Code and, more recently, Claude Tag. It also has its own framework for autonomous research, which has produced published results. The weaknesses are familiar: Claude sometimes makes “completely insane” decisions that invalidate the entire result, and it struggles to write the final report. If Anthropic had a framework that radically improved the models' usefulness, it would probably have turned it into a product.

What this means for practitioners?

First, speedup figures cannot simply be transferred between teams. The same model delivers 10x where the task is readily verifiable and experienced people are available, and 2–3x where nuanced decisions and manual oversight are required. If you are introducing agents, examine the bottlenecks: checking results, writing reports, and senior staff time. Second, the main risk is an agent that confidently breaks the methodology. Fast experiments are useless if some of the conclusions are invalid.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…