Δτ-Bench: The Best AI Agent Solves Only 42.3% of Tasks in a Changing World

Researchers from University College London and their co-authors have introduced Δτ-Bench, an executable benchmark in which an AI agent's toolset does not stand still. 1,588 verified multi-step tasks from 42 domains run through six successive ecosystem states. The best evaluated configuration, whether a model or an agent harness, solves only 42.3% of tasks. Along with the benchmark, the authors propose Steward, an ecosystem manager that keeps success rates higher across state changes and spends fewer resources on skill maintenance.

A test where the textbook is rewritten mid-exam

Most agent benchmarks work like an exam with an unchanging textbook: the set of APIs is fixed, the documentation is clean, and there is one correct route. Real systems are different. Services evolve independently of one another. A familiar tool disappears or gets replaced, the functionality of different services overlaps, and the documentation is noisy.

Δτ-Bench builds this dynamic into the test itself. The ecosystem passes through six states, and the transitions between them are built with capability-preserving transformations, meaning transformations that preserve the fundamental ability to complete the task. The task stays solvable; only the path to it changes. This design tests two skills that were previously measured separately:

• proactively finding more efficient call chains instead of sticking with the first one that works;
• quickly revising accumulated procedural knowledge when old recipes break.

Why skills go stale faster than agents learn

The most worrying part of the results concerns skill-learning methods, where an agent accumulates reusable procedures. They deliver a noticeable gain at the start, but as the ecosystem evolves, that gain does not turn into stable adaptation. Accumulated experience begins to work against the agent: it confidently follows a route that is already closed, or misses a new tool that does the same job in fewer steps.

The overall ceiling is also low. The strongest evaluated configuration reaches 42.3% of tasks completed. For comparison, static benchmarks usually evaluate agents in a stable environment. Here there is no stability, and a significant share of tasks remains unsolved.

Steward: a gardener for tools and knowledge

Based on these observations, the authors built Steward. The idea is that tools and procedural knowledge should be managed together, not as separate subsystems. The manager works through two mechanisms:

• incremental skill maintenance: only what has become outdated gets fixed, without rebuilding the whole library;
• dynamic tool delivery during task execution: the agent receives current options when it needs them.

According to the experiments, Steward sustains a higher success rate through all of the ecosystem's changes while spending less on skill maintenance. It recovers broken workflows better and switches to more efficient alternatives more often.

Key parameters of Δτ-Bench

• Format: an executable benchmark for agents with multi-step dialogues.
• Size: 1,588 validated tasks.
• Coverage: 42 domains.
• Dynamics: six successive ecosystem states created through capability-preserving transformations.
• Interference: overlapping functionality, noisy documentation, and tools that disappear or get replaced.
• Best result: 42.3% of tasks completed.
• Proposed method: Steward, with incremental skill maintenance and dynamic tool delivery.
• Authors: Shuofei Qiao, Xuanguang Pan, Yijun Chen, Wendi Feng, Bin Wu, Runnan Fang, Yuqi Zhu, Shumin Deng, Emine Yilmaz.
• Materials: the arXiv paper 2610.34238 and the zjunlp/delta-tau-bench repository on GitHub.

What to check in your own agent stack

If you build agents on MCP servers, internal APIs, or a set of plugins, Δτ-Bench suggests specific questions to ask about your own system. What happens to the skill library when one of the services changes its schema or disappears? Is there a mechanism that notices when a shorter path appears, not just when an old one breaks? How much does it cost, in tokens and human hours, to maintain the accumulated recipes?

The practical takeaway is simple. A single run against a frozen toolset is not enough to evaluate an agent. You need regular runs against a "drifting" version of the environment, and the agent's memory needs a policy for expiring and surgically updating entries. The benchmark code is open, so you can build such a stress test on your own domains right now.