AOS-Bench: Agents Separated by 0.6 Points in Isolation Deliver 54% and 68% of the Same Tasks

Coding agents are gradually shifting from executing one-off requests to acting as persistent assistants that receive a stream of work: builds, data processing, and bug fixes all at once, all on the same machine. A team of five authors led by Erle Zhu has released AOS-Bench, a benchmark that evaluates an agent system specifically as a manager of this kind of workload. The main result: success in isolation is a poor predictor of success in a stream. On heavy workloads, each of the five tested systems loses between 13 and 23 points.

Why Standard Tests Are Blind to the Queue

Existing benchmarks run each task separately, in a clean environment with ample resources. Real work looks different. Tasks arrive over time, and the agent itself decides when to pick up a new one, how to delegate it, and which tasks to run in parallel. It also has to stay within CPU and memory limits. Launch too many tasks and the machine chokes. Launch too few and some tasks are never touched.

In AOS-Bench, verified tasks arrive over time and must be delivered under shared CPU and memory constraints. The same tasks, run one at a time, serve as the baseline for each specific system. The gap between the isolated result and the streaming result shows how much of an agent's capability is lost to management.

Where the Points Are Lost

The authors tested five model–harness pairs. The picture was uneven:

• on heavy workloads, every system loses 13–23 points relative to its isolated success;
• on medium workloads, the systems differ by 11 points in isolated success, but by 26 points in streaming success;
• two systems that differ by 0.6 points in isolation deliver 54% and 68% of the same tasks;
• under sequential dispatch, 73% of the tasks in the heavy workload are never picked up;
• fixed concurrency of two trails the best agent-managed policy by no more than 4 points on both medium and heavy workloads;
• under every policy, median CPU utilization stays below 30% of allocated cores while tasks wait their turn.

Interventions in the execution policy showed that task intake explains a significant part of the gap. The agent simply fails to pick up work sitting in the queue, or decides not to. The quality of the solution itself is secondary here.

A Simple Queue vs. a Smart Manager

The most uncomfortable finding concerns complexity. One might expect an agent that plans task overlap and allocates resources on its own to clearly beat a primitive scheme. In practice, a fixed limit of two concurrent tasks comes within four points of the best agent-driven management. Meanwhile, the machine mostly sits idle: cores are busy less than 30% of the time at the median, even though unfinished work is waiting in the queue.

The bottleneck, then, is the "take it or wait" decision, not a shortage of hardware. Agents are too cautious or misjudge what can safely run together, and they leave resources unused.

The authors conclude that workload management should be treated as a separate agent capability, measured separately from the ability to solve a task. Two systems with nearly identical results in isolation behave completely differently in a stream, and ordinary leaderboards will not show this.

The practical takeaway for teams deploying persistent coding agents: choosing a system based on single-task benchmarks is not enough. Check how it behaves with several parallel assignments on a real machine. Look at the share of tasks that were picked up at all, and at CPU utilization while tasks wait. Before building complex orchestration, it is worth comparing it against a rigid two-slot queue: according to AOS-Bench, this baseline turns out to be surprisingly strong. For harness developers, the benchmark points to a specific place to improve: the task-intake policy, where up to 73% of potential work is lost.

The preprint was published on arXiv on 23 October 2026, and a second version followed on 27 October.