Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

64 AI Agents Instead of 8 Cost 690% More and Solved One Task Fewer

Say you need to fix a bug in Click. You can run a single agent that checks its own work, or assemble a team of one lead and 63 subagents. A Habr author ran both setups on 14 fixes and 30 AtCoder tasks. Going from 8 to 64 agents made an attempt about 690% more expensive, and both models solved one task fewer. For a local fix with clear tests, it makes more sense to start with a single agent and check its first draft.

AI-processed from Habr AI; edited by Hamidun News
64 AI Agents Instead of 8 Cost 690% More and Solved One Task Fewer
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
DeepSeek on AtCoder21 of 30 with one agent, 24 with eight, 23 with 64
Qwen on fixes8 agents got 3 passes out of 14 instead of 2, but at a 679% higher cost
Hard DeepSeek tasks4 of 10 with one agent, 6 of 10 with eight

Picture an ordinary task: there is a bug in the Click library, and you have a test that catches it. Modern agent harnesses offer two paths. You can run a single agent that writes a patch, checks it, and fixes it if needed.

Or you can assemble a team: a lead, subagents, dozens of drafts and reviews. The second path looks more solid, but it is very easy to burn an extra million tokens on drafts and reviews. The author of a study on Habr tested whether this pays off: how many additional patches actually pass the tests, and what they cost.

The short answer: past a certain team size, you get no more patches, while the bill grows almost eightfold.

How the measurement works

There were two sets of tasks. The first was 14 local fixes in two Python libraries: eight in Click and six in Packaging. The agent received a bug description and the source code in a pre-specified area. The target tests were verified in advance: they fail on the old code and pass on the reference patch, so they do catch the original bug. The second set was 30 AtCoder algorithmic tasks from LiveCodeBench across three difficulty levels. The full feature-development cycle, including requirements clarification, human review, and release, was not part of the experiment.

The author used two inexpensive models, DeepSeek V4.1 Flash and Qwen3.8 Flash, at temperature 0.

8 with reasoning mode off. Team sizes were 1, 2, 4, 6, 8, 16, 32, and 64 agents. A solo agent writes a solution, checks it, and can fix it.

In a team, each agent writes its own version, and each subagent also reviews a neighbor's draft. The lead receives all the reviews and up to three full drafts, and produces one final answer. A team of eight is seven subagents plus the lead; at 64 agents there are 63 reviews.

Quality was measured as the share of final answers that passed the target tests. Cost included all calls, failed ones too, at list price with no cache discount.

Where the gains stop

At first, the team helped. On AtCoder, DeepSeek solved 21 of 30 tasks with one agent and 24 with eight. On the 10 hard tasks, it rose from four to six. At 16 and 32 agents it stayed at 24, and at 64 it dropped to 23. Qwen reached its maximum of 18 of 30 already as a single agent. Both models solved all 10 easy tasks at every team size.

Going from 8 to 64 agents on AtCoder cost 691% more for DeepSeek and 687% more for Qwen, and both models lost one solved task. The confidence intervals of the difference include zero, so on a sample this small it is hard to tell a decline from random noise. Still, for the 64-agent team, all four groups had a smaller team that solved as many tasks or more, at a lower cost. At 64 agents, the whole corpus gave 25 of 44 for DeepSeek and 19 of 44 for Qwen.

The picture on the fixes is mixed. On Click #3902, a solo Qwen did not pass, while the team of eight did, lifting the model from two passes out of 14 to three. The price was a 679% increase in cost per attempt compared with a single agent with self-checking. For DeepSeek on Packaging #1531, the same step went the other way: it lost a pass, dropping from two successful patches to one. Both rejected patches passed the baseline tests, and the defect was caught by checks targeting the specific fix.

The price of an accepted patch, not an attempt

The author suggests counting not the price of a run, but the total of all attempts, failures included, divided by the number of patches that passed. On the fixes, Qwen's team of six agents got one pass and the team of eight got three, even though a run with eight cost 36% more on average. As a result, the cost per pass for eight came out 55% lower than for six. But among these three sizes, the cheapest passes came from a single agent with self-checking. At an equal budget, seven Qwen subagents yield about 19% of the passes a single agent gets.

What to do in practice

For a local fix with clear acceptance criteria, run one agent and check its first draft with the target tests. A large team is justified where a solo agent fails: for DeepSeek on hard tasks, eight agents gave a noticeable gain. But for Qwen on hard algorithmic tasks, no team helped: with 8 or more agents, there were no passes on any of the ten.

The limitations matter: two cheap models without reasoning mode, 14 fixes in two libraries, small samples. Results may differ on stronger models. But the method is easy to repeat on your own backlog: fix the tests, sweep team sizes, and compare the cost per pass rather than the cost per attempt.

Story

Key Questions Answered

How the measurement works?

There were two sets of tasks. The first was 14 local fixes in two Python libraries: eight in Click and six in Packaging. The agent received a bug description and the source code in a pre-specified area. The target tests were verified in advance: they fail on the old code and pass on the reference patch, so they do catch the original bug. The second set was 30 AtCoder algorithmic tasks from LiveCodeBench across three difficulty levels.

Where the gains stop?

At first, the team helped. On AtCoder, DeepSeek solved 21 of 30 tasks with one agent and 24 with eight. On the 10 hard tasks, it rose from four to six. At 16 and 32 agents it stayed at 24, and at 64 it dropped to 23. Qwen reached its maximum of 18 of 30 already as a single agent. Both models solved all 10 easy tasks at every team size. Going from 8 to 64 agents on AtCoder cost 691% more for DeepSeek and 687% more for Qwen, and both models lost one solved task.

What are the key details on The price of an accepted patch, not an attempt?

The author suggests counting not the price of a run, but the total of all attempts, failures included, divided by the number of patches that passed. On the fixes, Qwen's team of six agents got one pass and the team of eight got three, even though a run with eight cost 36% more on average. As a result, the cost per pass for eight came out 55% lower than for six. But among these three sizes, the cheapest passes came from a single agent with self-checking.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…