Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarMETR→ original

METR: The Length of Tasks an AI Agent Completes with 50% Reliability Doubles Every 3.6 Months

How much hard work can an AI agent get done on its own? METR measures this in hours of human work. If a task takes an engineer half a day and an agent solves it in half of its attempts, that is the agent's "horizon." For models up to Opus 5.5 and GPT-6 Astra, the horizon doubles roughly every 3.6 months. That makes it reasonable to hand tasks with clear verification to agents with more confidence, while decisions that cannot be checked quickly should stay with people.

AI-processed from METR; edited by Hamidun News
METR: The Length of Tasks an AI Agent Completes with 50% Reliability Doubles Every 3.6 Months
Source: METR. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceMETR ↗
Editor-in-Chief
Key metric #1A task that is twice as expensive to develop roughly halves the agent's horizon.
Key metric #2Judge models overrate work produced by an equally strong model.
Key metric #3Agent swarms beat the best single attempt only on tasks with fast verification.

Imagine you hand a task to a junior engineer and need to know which part of the work they can handle alone and where you will have to step in. Researchers face the same problem with AI agents, except measuring it is getting harder. METR, the organization that evaluates model capabilities, released a large November 2026 report on public models, including Opus 5.5 and GPT-6 Astra. The main point: capabilities that matter for accelerating AI development itself are growing fast, and measuring them is becoming harder.

What the "task horizon" means, in a simple example

The metric is called the time horizon. Take a task and estimate how long it takes an experienced human to do it. Say it's half a day. If an agent succeeds on such tasks in 50% of attempts, its 50-percent horizon is half a day. The larger this number, the longer and more independent pieces of work you can trust it with.

METR previously used this metric to measure the agentic abilities of AI in software engineering. Now the old TH1.1 task set is no longer enough: models have nearly mastered it. So the researchers combined it with a new, harder set, TH2. On the combined set, the horizon doubles roughly every 3.6 months. Because TH2 is harder, the absolute horizon of any model on the combined set is lower than it is on TH1.1 alone.

There is a caveat. On the verifiable part of TH2 without TH1.1, the doubling is faster. The recent acceleration may be hidden within the overall trend. But METR notes that other explanations are possible. OpenAI has recently cited a similar pace for its internal research tasks.

Where agents are strong, and where they judge worse than humans

Where a task has a clear metric and fast feedback, such as code optimization, models are usually no worse than experienced humans. When METR recruited its own human contractors for comparison, it generally could not find people who beat the frontier models. Where the organization had data from ten or more specialists, the models landed in a very high percentile, though not at the very top.

The picture changes when a choice must be made without a clear signal. For example, tests say little about whether a solution will work under other conditions, or checking takes a very long time. In such tasks, agents' judgment is worse than that of experts. In one example, judge models overrated the quality of output produced by an equally or more capable model. In practice, this means automated grading by a "judge model" tends to praise its peers.

METR also introduced a "lifecycle cost" measure: how much it costs to devise a task, run it, and verify it. Realistic tasks are usually more expensive. According to METR, doubling this cost at the same task length is roughly equivalent to halving the agent's horizon. At the same time, progress rates on cheap and expensive tasks are similar, and even on the most expensive tasks, results are improving over time.

What the checks showed and what to do in practice

For the overall picture, METR built a summary index, MCI. It is based on the IRT statistical technique, which is close to the one behind the Epoch Capabilities Index. On the public frontier, no strong acceleration or slowdown is visible in it. For individual developers there are tentative signs of a change in the curve's slope, but the result is sensitive to methodological choices, and METR itself is not confident in its reliability.

Among its robustness checks, the researchers looked for reward hacking and model failures. They also tested agent swarms. Compared with selecting the best of several independent attempts, swarms help on verifiable tasks with a clear metric and fast feedback. On other tasks, the effect is unclear or negative.

METR's overall position is cautious. Public data is not sufficient to rule out a sharp acceleration of AI development in the next couple of years. By this, the organization means a situation in which capabilities grow so fast that people can no longer meaningfully understand new systems, evaluate them, and respond to catastrophic risks.

What this means for teams:

  • Tasks with clear verification, such as code optimization against a measurable metric, can be handed to agents with more confidence.
  • Decisions without fast feedback should stay under expert control.
  • Don't rely on a judge model alone, especially when it evaluates the work of a model at the same level.
  • Enable agent swarms where results can be checked quickly.

There are limits too. This is a report from the evaluating organization itself, not an independent audit. METR describes some of its findings, including the trends for individual developers, as uncertain. In addition, the source text available to us is truncated: details of the methodology and appendices remain in the full version of the report.

Story

Key Questions Answered

What the "task horizon" means, in a simple example?

The metric is called the time horizon. Take a task and estimate how long it takes an experienced human to do it. Say it's half a day. If an agent succeeds on such tasks in 50% of attempts, its 50-percent horizon is half a day. The larger this number, the longer and more independent pieces of work you can trust it with. METR previously used this metric to measure the agentic abilities of AI in software engineering. Now the old TH1.1 task set is no longer enough: models have nearly mastered it.

Where agents are strong, and where they judge worse than humans?

Where a task has a clear metric and fast feedback, such as code optimization, models are usually no worse than experienced humans. When METR recruited its own human contractors for comparison, it generally could not find people who beat the frontier models. Where the organization had data from ten or more specialists, the models landed in a very high percentile, though not at the very top. The picture changes when a choice must be made without a clear signal.

What the checks showed and what to do in practice?

For the overall picture, METR built a summary index, MCI. It is based on the IRT statistical technique, which is close to the one behind the Epoch Capabilities Index. On the public frontier, no strong acceleration or slowdown is visible in it. For individual developers there are tentative signs of a change in the curve's slope, but the result is sensitive to methodological choices, and METR itself is not confident in its reliability.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…