Δτ-Bench: Best AI Agent Solves Only 42.3% of Tasks in a Shifting World

Researchers from University College London and their co-authors have introduced Δτ-Bench — an executable benchmark in which the toolset available to an AI agent doesn't stay still. 1,588 validated multi-step tasks across 42 domains are run through six sequential ecosystem states. The best evaluated option, whether a model or an agentic scaffold, solves only 42.3% of tasks. Alongside the benchmark, the authors propose Steward — an ecosystem manager that keeps success rates higher across state transitions while spending fewer resources maintaining skills.

A test where the textbook gets rewritten mid-exam

Most agentic benchmarks are structured like an exam with a fixed textbook: the set of APIs is static, the documentation is tidy, and there's a single correct route. Real systems look nothing like that. Services evolve independently of one another. A familiar tool disappears or gets replaced, the functionality of different services overlaps, and documentation gets noisy.

Δτ-Bench builds this dynamic into the test itself. The ecosystem moves through six states, and the transitions between them are built via capability-preserving transformations — that is, transformations that preserve the fundamental ability to complete the task. The task remains solvable; only the path to it changes. This design tests two abilities that used to be measured separately:

• proactively finding more efficient call chains rather than clinging to the first one that works;
• quickly revising accumulated procedural knowledge when old recipes break.

Why skills go stale faster than agents learn

The most troubling part of the results concerns skill-learning methods, where an agent accumulates reusable procedures. They deliver a noticeable boost at the start, but as the ecosystem evolves, that boost doesn't translate into stable adaptation. Accumulated experience starts working against the agent: it confidently follows a route that's already closed off, or misses a new tool that would do the same job in fewer steps.

The overall ceiling is also low. The strongest evaluated option reaches 42.3% successful tasks. By comparison, static benchmarks typically evaluate agents in a stable environment — here there's no stability, and a significant share of tasks goes unsolved.

Steward: a gardener for tools and knowledge

Based on these observations, the authors built Steward. The idea is that tools and procedural knowledge need to be managed together, not as separate subsystems. The manager operates through two mechanisms:

• incremental skill maintenance: only what's outdated gets fixed, without rebuilding the whole library from scratch;
• dynamic tool delivery right during task execution: the agent gets up-to-date options exactly when it needs them.

According to the experiments, Steward maintains higher success rates throughout all the ecosystem changes at lower skill-maintenance cost. It recovers broken workflows better and switches to more efficient alternatives more often.

Key parameters of Δτ-Bench

• Format: executable benchmark for agents with multi-step dialogues.
• Scale: 1,588 validated tasks.
• Coverage: 42 domains.
• Dynamics: six sequential ecosystem states generated by capability-preserving transformations.
• Interference: overlapping functionality, noisy documentation, tools disappearing and being replaced.
• Best result: 42.3% successful tasks.
• Proposed method: Steward, with incremental skill maintenance and dynamic tool delivery.
• Authors: Shuofei Qiao, Xuanguang Pan, Yijun Chen, Wendi Feng, Bin Wu, Runnan Fang, Yuqi Zhu, Shumin Deng, Emine Yilmaz.
• Materials: arXiv paper 2610.34238 and the zjunlp/delta-tau-bench repository on GitHub.

What to check in your own agent stack

If you're building agents on MCP servers, internal APIs, or a set of plugins, Δτ-Bench points to concrete questions worth asking about your own system. What happens to the skill library when one of the services changes its schema or disappears? Is there a mechanism that notices the emergence of a shorter path, not just the breakage of an old one? How much does maintaining accumulated recipes cost in tokens and human hours?

The practical takeaway is simple. Evaluating an agent with a single run against a frozen toolset isn't enough. You need regular runs against a "drifting" version of the environment, and the agent's memory needs a policy for staleness and targeted updates. The benchmark's code is open, so this kind of stress test can be assembled on your own domains right now.