Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarMIT Technology Review→ original

METR Updates Its Famous AI Chart: Task Length Doubles Every 4 Months, but Judgment Lags

METR has updated the chart that measures how long a task AI can complete on its own, with length defined by how long a human takes to do it. Previously, the horizon of frontier models doubled every four months, and in coding that trend holds. But METR staff did not find a single AI-designed test for AI useful. So strength in coding cannot be assumed to carry over to tasks that require taste and judgment.

AI-processed from MIT Technology Review; edited by Hamidun News
METR Updates Its Famous AI Chart: Task Length Doubles Every 4 Months, but Judgment Lags
Source: MIT Technology Review. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceMIT Technology Review ↗
Editor-in-Chief
Key metric #1Every 4 months, the length of tasks available to models doubled
Key metric #2None of the tests designed by AI proved useful to METR researchers
Key metric #3Opus 5.5 reached "almost good enough"; a year ago all models were "clearly worse"

If you have seen a chart on social media with a rising curve labeled "AI does ever longer tasks," it was most likely METR's. It has been retold dozens of times and misunderstood almost as often. Now the nonprofit lab that evaluates models has released a major update. For anyone building products on agents, the main news is that the coding curve has not broken, while on tasks that require judgment, models still show notable failures.

What the chart actually shows

The idea is simple. METR takes a large set of tasks, each of which real people have already solved. Task difficulty is measured by how long it took a human to finish it. METR then checks what "human length" of task a model can complete successfully. It is not "an hour of AI work" but "a task that takes a specialist an hour."

The first version of the chart showed that every four months, the length of tasks that frontier models can handle doubles. But the task set was skewed. All the tasks were ones where the result can be checked automatically: the test either passes or fails. Those are exactly the tasks neural networks are trained on best. A model tries to solve a task many times, and reinforcement learning strengthens what led to success and weakens what led to failure. That is why AI is so strong at code and math, where a machine can verify whether a solution is correct.

What the new version adds

The updated set includes tasks that take a human far longer, plus more open-ended tasks with no clear success criterion. For example, models were asked to come up with new tests for evaluating AI that METR could use in its own research, and the organization's staff then assessed the results. Although the best models today are stronger than almost any human at software development, they lost here: not a single AI-created test was judged useful for the work.

In another task, models evaluated philosophical arguments to test the quality of their judgment. Here too they fell behind humans. The researchers themselves acknowledge that there is debate inside METR about how closely this task relates to automating AI research.

Still, the trend is encouraging. The newest models, such as Opus 5.5, more often get a "slightly above the bar" rating on the test-design tasks. That is not yet enough for METR's research, but it is noticeable progress. A year ago, every model landed in the "clearly below the bar" category.

Why it matters and where the weak spots are

There is a lot of tension around this topic. Earlier this month, METR released a different, heavily redacted report saying that AI may be roughly doubling the pace of development at Anthropic. That raised fears that the company is close to recursive self-improvement, in which AI designs its own successors. The new report neither confirms nor refutes those fears, but it shows that models are still poor at exactly what such automation requires: choosing directions and evaluating ideas.

The experts quoted by MIT Technology Review are cautious. Sayash Kapoor, an incoming professor at UC Berkeley, says it is time for AI evaluations to become more inventive, and he considers "research taste" (the ability to pick promising questions and experiments) the main bottleneck. He is not surprised by the failure on open-ended tasks: his August preprint shows that models still cannot independently reproduce human research in computer science. On Wednesday, OpenAI wrote in a blog post that its models do worse on real tasks from employees than on the simpler, verifiable tasks from METR's original set. Other specialists, by contrast, are confident that models trained on verifiable tasks will eventually master open-ended ones too.

The new chart has honest limitations. The completion time for the longest tasks is mostly estimated rather than measured: paying people for hundreds of hours of work, or finding the few qualified experts, is too expensive. In the philosophical-arguments task, one person evaluated most of the arguments, and agents are scored on how closely their opinion matches that person's.

In addition, the set contains few open-ended tasks, so it is hard to make a forecast from them. METR's Matt Putz says outright that it will not be possible to build trends for such tasks as clear as those for easily formalized ones. Najoung Kim of Boston University notes that the observed growth says something about the current moment, but how to extrapolate it is still unclear.

What this means for practitioners

If you build agents for coding, testing, or other tasks with automatic verification, the METR curve remains a reasonable benchmark: the horizon is growing fast. If you instead hand an agent the job of evaluating ideas, setting priorities, or designing metrics, keep a human in the review loop. Results on coding tests cannot be automatically carried over to such tasks. And when reading any chart about "task length," remember that it shows how much time a human spends on the task, not how long the model works.

Story

Key Questions Answered

What the chart actually shows?

The idea is simple. METR takes a large set of tasks, each of which real people have already solved. Task difficulty is measured by how long it took a human to finish it. METR then checks what "human length" of task a model can complete successfully. It is not "an hour of AI work" but "a task that takes a specialist an hour." The first version of the chart showed that every four months, the length of tasks that frontier models can handle doubles. But the task set was skewed.

What the new version adds?

The updated set includes tasks that take a human far longer, plus more open-ended tasks with no clear success criterion. For example, models were asked to come up with new tests for evaluating AI that METR could use in its own research, and the organization's staff then assessed the results. Although the best models today are stronger than almost any human at software development, they lost here: not a single AI-created test was judged useful for the work.

Why it matters and where the weak spots are?

There is a lot of tension around this topic. Earlier this month, METR released a different, heavily redacted report saying that AI may be roughly doubling the pace of development at Anthropic. That raised fears that the company is close to recursive self-improvement, in which AI designs its own successors. The new report neither confirms nor refutes those fears, but it shows that models are still poor at exactly what such automation requires: choosing directions and evaluating ideas.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…