AOS-Bench tests AI agents: quality drops 13-23 points under parallel workloads

The practical question behind AOS-Bench

A team of researchers set out to answer a practical question: how ready are AI agents that write code and operate a computer to handle a stream of tasks rather than just one at a time — the way things actually work in practice, where a project build is running, a data-processing request comes in, and then a bug-fix request lands on top of that. Every existing benchmark for these agents (Terminal-Bench, OSWorld, and the like) evaluates tasks one at a time, so nobody had measured whether an agent can juggle multiple jobs under real resource constraints. To fill that gap, the team built AOS-Bench — a benchmark that treats the agent as a kind of "natural-language operating system" that has to decide on its own when to pick up a new task, how to split work across subtasks, and how to stay within the CPU and memory it's been given.

How the experiment was set up

The team tested five model-plus-agent-shell combinations: Claude Opus 5, GPT-5.6-sol, DeepSeek-V4-Pro, GLM-5.3, and Kimi-K3. Each system was capped at 4 CPU cores and 8GB of memory. In one condition, tasks arrived strictly one at a time — establishing a baseline for "individual" performance. In the other, several tasks arrived back-to-back at intervals, mimicking a real workflow ("stream"). Two difficulty levels were used: medium (10 tasks per run) and heavy (20 tasks).

What the numbers showed

Under heavy load, every one of the five systems lost between 13 and 23 percentage points of successful completions compared to its own performance on the same tasks solved one at a time. Claude Opus 5 held up best: its success rate fell from 75.5% on individual tasks to 62.5% in the stream condition — meaning it retained about 83% of its baseline capability. The other four systems retained only 61–75% of their baseline.

At medium load, the results were even more telling: how well a system solves individual tasks turns out to be a poor predictor of how it will handle a stream. GPT-5.6-sol posted the highest individual-task score, 85.3%, but lost 14.5 points once tasks started streaming in. Kimi-K3 started 11 points lower on individual tasks but lost just 6.3 points in the stream condition. The starkest example: two systems whose individual-task scores differed by only 0.6 points (74.9% vs. 74.3%) ended up at 68.1% and 53.9% under streaming — nearly a 15-point gap that their baseline scores gave no hint of.

Why this happens

The main driver isn't that agents get worse at solving the tasks themselves. The researchers broke down each point of lost performance by cause and found that the bulk of the gap under heavy load — 16 to 26 of the 16–23 points lost by four of the systems — came from tasks that the agent never picked up at all, didn't finish before timing out, or finished only after memory had already been exhausted. Actual execution errors on tasks the agent did take on accounted for no more than 1.3 points of loss. In plain terms: these systems aren't stumbling over solving the work — they're stumbling over picking it up in time.

GLM-5.3 and DeepSeek-V4-Pro were especially telling cases: in some runs, they simply stopped accepting new tasks and fell back to handling them one at a time, leaving 25% and 48% of incoming tasks untouched, respectively (compared to 0% for Claude Opus 5). Notably, when the same tasks were forced onto the systems all at once instead of trickling in sequentially, the agents picked up three-quarters of them — meaning the bottleneck isn't task difficulty, it's specifically the logic for accepting new work.

The counterintuitive finding about "smart" scheduling

The obvious fix would seem to be teaching agents to plan concurrency more intelligently. But the experiment found the opposite: the simplest possible rule — "always run exactly two tasks at once, no analysis involved" — came within 4 points of the best strategies where the agent decides for itself what to run and when. And regardless of which scheduling strategy was used, CPU utilization stayed below 30% of the allocated four cores while tasks sat waiting — resources sitting idle even when there was plenty of work queued up.

For teams planning to build persistent AI agents for real-world, multi-tasking workloads, the practical takeaway is this: don't rush to invest in sophisticated task-scheduling logic inside the agent — a simple concurrency cap gets you nearly the same results. The thing that actually needs fixing first is the mechanism for accepting and acknowledging new tasks, since that's where most of the lost efficiency is coming from.