METR: The Length of Tasks an AI Agent Completes with 50% Reliability Doubles Every 3.6 Months
How much hard work can an AI agent get done on its own? METR measures this in hours of human work. If a task takes an engineer half a day and an agent solves it in half of its attempts, that is the agent's "horizon." For models up to Opus 5.5 and GPT-6 Astra, the horizon doubles roughly every 3.6 months. That makes it reasonable to hand tasks with clear verification to agents with more confidence, while decisions that cannot be checked quickly should stay with people.
AI-processed from METR; edited by Hamidun News
Imagine you hand a task to a junior engineer and need to know which part of the work they can handle alone and where you will have to step in. Researchers face the same problem with AI agents, except measuring it is getting harder. METR, the organization that evaluates model capabilities, released a large November 2026 report on public models, including Opus 5.5 and GPT-6 Astra. The main point: capabilities that matter for accelerating AI development itself are growing fast, and measuring them is becoming harder.
What the "task horizon" means, in a simple example
The metric is called the time horizon. Take a task and estimate how long it takes an experienced human to do it. Say it's half a day. If an agent succeeds on such tasks in 50% of attempts, its 50-percent horizon is half a day. The larger this number, the longer and more independent pieces of work you can trust it with.
METR previously used this metric to measure the agentic abilities of AI in software engineering. Now the old TH1.1 task set is no longer enough: models have nearly mastered it. So the researchers combined it with a new, harder set, TH2. On the combined set, the horizon doubles roughly every 3.6 months. Because TH2 is harder, the absolute horizon of any model on the combined set is lower than it is on TH1.1 alone.
There is a caveat. On the verifiable part of TH2 without TH1.1, the doubling is faster. The recent acceleration may be hidden within the overall trend. But METR notes that other explanations are possible. OpenAI has recently cited a similar pace for its internal research tasks.
Where agents are strong, and where they judge worse than humans
Where a task has a clear metric and fast feedback, such as code optimization, models are usually no worse than experienced humans. When METR recruited its own human contractors for comparison, it generally could not find people who beat the frontier models. Where the organization had data from ten or more specialists, the models landed in a very high percentile, though not at the very top.
The picture changes when a choice must be made without a clear signal. For example, tests say little about whether a solution will work under other conditions, or checking takes a very long time. In such tasks, agents' judgment is worse than that of experts. In one example, judge models overrated the quality of output produced by an equally or more capable model. In practice, this means automated grading by a "judge model" tends to praise its peers.
METR also introduced a "lifecycle cost" measure: how much it costs to devise a task, run it, and verify it. Realistic tasks are usually more expensive. According to METR, doubling this cost at the same task length is roughly equivalent to halving the agent's horizon. At the same time, progress rates on cheap and expensive tasks are similar, and even on the most expensive tasks, results are improving over time.
What the checks showed and what to do in practice
For the overall picture, METR built a summary index, MCI. It is based on the IRT statistical technique, which is close to the one behind the Epoch Capabilities Index. On the public frontier, no strong acceleration or slowdown is visible in it. For individual developers there are tentative signs of a change in the curve's slope, but the result is sensitive to methodological choices, and METR itself is not confident in its reliability.
Among its robustness checks, the researchers looked for reward hacking and model failures. They also tested agent swarms. Compared with selecting the best of several independent attempts, swarms help on verifiable tasks with a clear metric and fast feedback. On other tasks, the effect is unclear or negative.
METR's overall position is cautious. Public data is not sufficient to rule out a sharp acceleration of AI development in the next couple of years. By this, the organization means a situation in which capabilities grow so fast that people can no longer meaningfully understand new systems, evaluate them, and respond to catastrophic risks.
What this means for teams:
- Tasks with clear verification, such as code optimization against a measurable metric, can be handed to agents with more confidence.
- Decisions without fast feedback should stay under expert control.
- Don't rely on a judge model alone, especially when it evaluates the work of a model at the same level.
- Enable agent swarms where results can be checked quickly.
There are limits too. This is a report from the evaluating organization itself, not an independent audit. METR describes some of its findings, including the trends for individual developers, as uncertain. In addition, the source text available to us is truncated: details of the methodology and appendices remain in the full version of the report.
- Nov 21, 2026Apple Is Building an AI Agent and Making It Harder for Outsiders to Get Access: Permission Now Takes 14 Steps, Not 4
- Nov 21, 2026Developer Skips Pinecone: Searching 72,450 Code Chunks in 50 ms
- Nov 21, 2026Gemini 4 Neon shrank the TS Rust binary from 47.44 to 7.97 MB in under 10 hours
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
Key Questions Answered
What the "task horizon" means, in a simple example?
The metric is called the time horizon. Take a task and estimate how long it takes an experienced human to do it. Say it's half a day. If an agent succeeds on such tasks in 50% of attempts, its 50-percent horizon is half a day. The larger this number, the longer and more independent pieces of work you can trust it with. METR previously used this metric to measure the agentic abilities of AI in software engineering. Now the old TH1.1 task set is no longer enough: models have nearly mastered it.
Where agents are strong, and where they judge worse than humans?
Where a task has a clear metric and fast feedback, such as code optimization, models are usually no worse than experienced humans. When METR recruited its own human contractors for comparison, it generally could not find people who beat the frontier models. Where the organization had data from ten or more specialists, the models landed in a very high percentile, though not at the very top. The picture changes when a choice must be made without a clear signal.
What the checks showed and what to do in practice?
For the overall picture, METR built a summary index, MCI. It is based on the IRT statistical technique, which is close to the one behind the Epoch Capabilities Index. On the public frontier, no strong acceleration or slowdown is visible in it. For individual developers there are tentative signs of a change in the curve's slope, but the result is sensitive to methodological choices, and METR itself is not confident in its reliability.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.