Top AI Agents Pass Up to 98% of Tests, but Build a Fully Working Product in at Most 65% of Runs

Imagine asking an AI agent to build a taxi app. First it builds ride ordering. Then you ask for cancellation penalties, drivers with multiple cars, earnings per ride and split payments.

Each new feature sits on top of code the agent wrote earlier. If it breaks an old feature, say earnings stop accounting for penalties, the product is unusable, even when almost everything else works. This is how people "vibe-code" on Replit today.

Familiar benchmarks, meanwhile, test only a single, clearly described task, and many are nearly saturated. So Replit released ViBench 2.0.

The result: the three strongest models pass up to 98% of test scenarios, but deliver a fully working product in at most 65% of builds. The share of fully working builds on the first attempt is called pass@1. If you are building a product on an agent, that is the number to watch.

How the test works

The agent starts with an empty project and receives requirements one feature at a time in a single long conversation. Each product has 9 to 18 requests, a conversation can exceed 200 thousand tokens, and each stage is capped at 300 agent steps and two hours. In total, the benchmark contains 17 full-fledged web products modeled on Amazon, Asana, Discord, Figma, GitHub, Google Docs, Jira and Uber. They include 241 requests for new features and 544 test scenarios, and 85% of the scenarios test several features together.

Nobody checks the results between requests. The grade is assigned once, when the product is finished, by a browser agent that opens the app in a real Chromium instance and acts like a user. A build counts only if every scenario for that product passes, and each product has between 26 and 41 of them. The agent chooses its own architecture, so the tests check outcomes rather than a specific design. A label reading "Waiting at stop" instead of "At Stop" does not count as an error.

The authors also tested the checker itself. When a scenario fails, it re-checks up to four times, until three verdicts agree. It caught all 34 deliberately planted bugs, and its false-pass rate is only 0.8%. The checker and the test-data preparation both run on Opus 5.5.

The gap between "almost" and "done"

The authors tested nine models. The leaders, GPT-6.1 Sol, GPT-6 Astra and Opus 5.5, pass up to 98% of scenarios. But they deliver a fully working product in no more than 65% of builds. Most of the failed builds from these three missed just one or two scenarios. The room for improvement lies in reliability, not in the ability to produce something at all. No open model ever produced a fully working product.

Progress is fast, though. Across three releases of one model family, pass@1 on the public products rose from 3% to 53%, while the cost of a single build fell by more than half. This matters for other benchmarks too: on Vibe Code Bench, the best result rose from 61.8% in March 2026 to more than 90% for six models by the end of October. Tests where the agent does a single task hit the ceiling quickly. A long build with a check of the finished product has not hit it yet.

What to do about it

The average share of passed tests hides the main point: one broken old feature makes a build unusable. Evaluate models by the share of flawless builds. Test the product as a whole, the way a user would, not just individual steps. Replit itself uses ViBench 2.0 to choose models for its platform and to evaluate changes to its production agent.

There are caveats. Replit wrote the test and uses it itself. But an outside team at Fireworks checked the launch: it set up the published benchmark and ran every model with a single command, without modifications. Eight products are published on GitHub along with the runner and the checker. Another nine are kept hidden so that models are not trained on the test. The public files carry a special marker string, which lets them be filtered out of training data.