Replit releases ViBench 2.0: the best model delivers a fully working product in 65% of builds

Replit, the platform where users build entire products without writing a line of code, has introduced ViBench 2.0, a benchmark for long-horizon software creation tasks. The agent starts with an empty project, receives requirements one feature at a time in a single conversation, and must ultimately deliver a working web product. It is not checked with unit tests. Instead, it is tested the way a user would test it: through a browser, and only after the final request. The result is uncomfortable for the market leaders: the best model, GPT-6.1 Sol, passes up to 98% of test plans, but builds a flawless product in only 64.7% of builds.

How the benchmark works

ViBench 2.0 includes 17 full-fledged web products modeled on well-known services: Amazon, Asana, Discord, Figma, GitHub, Google Docs, Jira and Uber. Each one grows over 9–18 prompts, 241 in total. For example, the agent first builds a ride-hailing app, then receives, one after another, cancellation fees, drivers with multiple vehicles, per-trip earnings and split payments. Each new feature builds on code the agent wrote itself, and the conversation can exceed 200 thousand tokens. There is no feedback after any step, and old features must survive every later change.

A browser agent checks the result. It drives a real Chromium instance through Playwright and executes 544 test plans. 85% of the plans test several features together, for example that a driver's earnings account for fees and don't break because of split payments. A build counts (pass@1) only if every plan passes. This is a deliberately strict metric: for a user who doesn't read code, "almost works" means "doesn't work."

Conditions are identical for all models: an agent built on the OpenHands SDK with a terminal and a file editor, medium reasoning effort, and up to 300 steps and two hours per step. Six closed models were tested — GPT-6.1 Sol, GPT-6 Astra, Opus 5.5, Sonnet 5.5, Grok 4.7 and Gemini 3.8 Flash — along with three open ones: DeepSeek V4.1 Flash, Kimi K3 and GLM 5.3. Each model built each product four times, for 612 builds in total.

The gap between "almost" and "done"

The top three look like this: Sol at 64.7% pass@1, Astra at 60.3%, Opus 5.5 at 48.5%. Their share of passed plans is 93.7–98.0%. Of the 84 failed builds that reached evaluation, 42 failed exactly one plan, and 62 failed no more than two. Astra produced at least one fully working product for 16 of 17 tasks, Sol for 15, Opus for 13. The only product that failed every time across the leaders' 36 attempts was the Figma clone, where collaborative editing and undo broke.

The open models are still far from this level: none of their 204 builds passed all plans. They do differ, however, in share of plans passed: DeepSeek V4.1 Flash scored 77.4%, landing within two points of Sonnet 5.5 while costing roughly 60% as much.

Price and speed vary widely. A build costs from $2.5 to $89.9 and takes from 26 minutes to more than four hours. Astra costs about seven times as much as Sol, with a pass@1 difference of less than five points. Opus builds 2.4 times faster than Sol (73 minutes) with only 6% fewer output tokens; the authors attribute this to inference speed.

Progress is fast. Across three releases of one model family, pass@1 on the public products rose from 3% to 53%, while the cost per build fell by more than half. Notably, pass@1 grows significantly only once the share of passed plans exceeds about 70%.

How far the grader can be trusted

The most vulnerable element of such a system is the AI-based grader, so the authors tested it separately. On a single evaluation, it wrongly fails a working plan in 0.2% of cases, which is why failed plans are rechecked up to five times, until three verdicts agree.

False "passes" are rarer: GPT-6.1 Sol rechecked 477 plans that Opus had approved, and after manual review only 4 (0.8%) turned out to be false.

Of 34 deliberately injected bugs, 24 of them subtle (for example, a total off by one cent), the grader found all of them. An audit of 178 failed plans revealed 13 erroneous verdicts and two places where the specification was incomplete. All of this was fixed before release, and the remaining 159 failures are real bugs.

Separately from Replit, the Fireworks team ran the benchmark, launching it with a single command per model and no modifications. All figures in the article come from those runs.

Limitations and practical takeaway

Eight products are publicly available along with the harness and the grader. The other nine are hidden so that models are not trained on these tasks, and independent results are calculated across all 17. The grader and the "seeding" agent in the experiments are both Opus 5.5, which could in theory affect the scores, although a second grader mostly did not confirm this. In addition, the benchmark was built around Replit's own decisions about which models to integrate, so it reflects its specific use cases: web products from scratch, not work on someone else's codebase.

For teams choosing a model for agentic development, the takeaway is practical: compare not the share of tests passed but the probability of getting a fully working result. That is where the best models have the most headroom left, and where the open models have the biggest gap.