Alibaba's new benchmark: AI agents satisfy some requirements, but not the whole task

Alibaba, together with the Chinese University of Hong Kong, Shanghai Jiao Tong University and Nanyang Technological University, has introduced Co-Design Bench. It is a set of tasks for AI agents that write code. The models are tested on what users actually ask for: take an already working web app and change it according to a description, sometimes together with a screenshot, voice recording, video or document. Here is the result for the ten models tested. The best one meets a sizable share of the requirements on average per task, but fully completes only a smaller share of the tasks. It handles individual items but cannot put everything together.

Tasks from real sessions, not synthetic data

Many tests for coding agents are built artificially: they break a reference page and ask the agent to fix it, or hand over a predefined set of edits. The authors of Co-Design Bench took a different route. They took requests from real user sessions, removed duplicates and kept those that give something to check. For each task, they restored a runnable "before" version of the app, along with the session history and attachments.

The test has two tracks:

• Multimodal Editing (ME) — local edits to styling, structure and content. The agent gets a page, a selected region and a request, sometimes with an image, audio, video or document. Example: a receipt shows a total, and the agent has to update it consistently in a table and in the summary views. • Repository Editing (RE) — coordinated changes across a frontend repository: components, routes, state and data flow. Example: a sports video clip editor where the agent must limit the number and length of clips, keep the timeline aligned when zooming, and re-enable insertion after a clip is deleted or split.

Tasks are scored along three dimensions: understanding of context (Grounded Understanding), interface transformation (UI Transformation) and implementation of functions (Functional Implementation).

Before a task enters the set, the app must build, start in a container and render in Playwright (a browser automation tool). Tasks that need unavailable external services were excluded. Requests were ranked by difficulty, verifiability and signs of past failures, such as rephrasings and rollbacks. Experts then refined each request into a clear statement and wrote hidden acceptance criteria.

How to grade without reference code

A task has no reference solution, because one edit can be made in many ways. The criteria therefore describe the outcome: what should appear and how it should behave. They do not dictate the implementation. Comparison with the original app helps establish what changed and what needed to stay the same.

The check runs in several steps:

• First, an evaluator agent examines the final app: the source diff, rendered pages and recorded browser actions. A handler in the code proves nothing if the user cannot reach it through the interface. • Then a separate LLM verifier receives the verdicts and the collected evidence and tries to overturn them. It looks for stubs, hardcoded answers, unreachable entry points, failures on other inputs and mismatches with the request. • A mandatory audit requires successful rendering and screenshots for visual claims, and browser action logs for interactive ones. If there is no evidence, scoring is aborted before any points are counted.

All verdicts and logs are saved, so any score can be traced back to its evidence. Two frontend experts also manually checked a portion of the scores. The authors openly acknowledge that this does not fully eliminate judge errors.

What the ten models showed

Ten model endpoints were run through a common agent harness: the agent can read code, edit files, build the app and run it. Several conclusions follow from the results.

• The share of criteria met and the share of tasks fully solved diverge noticeably. A user whose request was mostly implemented still ends up with a broken workflow. • Model rankings differ between tracks: the best model on local edits is not necessarily the best on repository edits. • For all ten models, completeness is worse on repository tasks with three or four types of change (structure, style, interactivity, content) than on tasks with one or two.

Separately, the authors showed that the operating mode changes the score. On WebCompass-edit tasks, moving from a single attempt to an agentic mode with tools noticeably raised the average share of functions completed, even though the rubric, judge and renderer stayed the same. This means figures from tests run in different modes cannot be compared directly.

The authors compare Co-Design Bench with other benchmarks. Design2Code, WebSight-Test and WebRenderBench test converting a screenshot into code. IWR-Bench adds videos of interaction. WebGen-Bench and FrontendBench test sites built from scratch. DesignBench, MT-Web2Code, WebCompass and FronTalk cover edits and dialogue. SWE-bench, SWE-bench Multimodal and SWE-Lancer draw tasks from GitHub and freelance work. What is new in Co-Design Bench is the combination of real sessions, multimodal attachments, a runnable starting state, grading without a reference solution and an adversarial verifier.

For teams deploying coding agents, the takeaway is practical. A "percentage of items passed" metric overstates readiness: it is more useful to count the share of tasks completed in full, and to look separately at complex, multi-part edits. To verify AI-written code, it helps to require evidence such as screenshots and recordings of browser actions. Code in a diff does not prove that a feature works.