Best AI Coding Agent Fully Solves Just 26% of Real-World Co-Design Bench Tasks

If you let an AI agent into a working repository, the picture is familiar: it does almost everything, and then you spend half an hour catching the remaining details. The new Co-Design Bench benchmark is the first to measure this systematically, and the numbers are sobering.

What was tested

A team from Alibaba and several Asian universities built tests from real development sessions. The researchers deliberately picked cases where the user had to rephrase a request or roll back changes. In other words, the test keeps tasks on which agents had already made mistakes in practice.

The tasks vary: from a small fix to a rebuild of an entire project. A typical example: extract the total from a photo of a receipt and update all dependent totals in the application. Some tasks are multimodal, with images, audio, video or documents as input.

The grading is strict. One AI judge runs the finished application in a browser, while a second re-checks every positive verdict and looks for stubs and hardcoded results. Points are awarded only for what actually works.

The results

Of the ten models, GPT-6 Astra came out on top. It meets about 70% of a task's requirements on average, but delivers a fully error-free result in only 26.4% of cases. Claude Opus 5 and Qwen3.8-Max follow, with Grok 4.6 in last place. In the multimodal track the leader fully solves 41% of tasks, and in the repository tracks less than 18%.

On older benchmarks such as WebCompass-edit, models score nearly the maximum, whereas here the gap exceeds 35 percentage points. The authors believe the earlier tests have simply become saturated: when an agent runs and fixes code on its own, differences between models are barely visible.

Another pattern: the more different types of changes in one task (layout, design, interactivity, content), the worse the result. With three to four types, all ten models dropped. And in real projects, edits are almost always mixed. More compute does not help: GLM-5.2 spends roughly six times more tokens than GPT-6 Astra on small edits and still loses noticeably.

The environment matters as much as the model

In a separate experiment, the same models were run in different harnesses, that is, the programs through which an agent reads, writes and executes code. On large rework tasks, Qwen3.8-27B scored 75% in one environment and 56% in another. The ranking of two models also depended on the environment. That is why the authors advise always specifying the "model plus harness" pair.

Limitations

The benchmark checks one fixed request per task, with no clarifications or corrections during the work. AI judges can also make mistakes. In addition, earlier METR research showed that the share of code accepted by project maintainers is about 24 points lower than results on SWE-bench Verified. The effect of the environment is ambiguous: another recent study found almost no benefit from more complex harnesses, even though they spent up to three times more tokens.

What to do in practice

First: don't choose an agent based on model rankings. Run the "model plus tool" combination on your own real tasks and compare exactly that. Second: budget time for review and polishing, especially when a task mixes layout, logic and data. Break large requests into uniform steps; this reduces the risk of unfinished work. Third: check the result in the running application, not in the diff, because that is also how this benchmark graded.

The conclusion is simple: agents save a lot of time, but the promise of "hand off the task and forget it" currently holds in roughly one case out of four.