Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarVercel Blog→ original

Vercel: the agent ordered the right coffee in 60% of attempts, but only 8% of the time when it had to get it right five times in a row

An agent can successfully order a hot latte instead of a cold one: the logs look clean, but the customer is unhappy. Vercel suggests running the same task several times and measuring the share of correct outcomes. In its demo, Claude Code with Opus 5.5 succeeded in 3 of 5 attempts (60%). The chance of at least one correct order in five attempts is about 99%, while the chance of five correct orders in a row is about 8%. For daily coffee orders, the second number matters more.

AI-processed from Vercel Blog; edited by Hamidun News
Vercel: the agent ordered the right coffee in 60% of attempts, but only 8% of the time when it had to get it right five times in a row
Source: Vercel Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceVercel Blog ↗
Editor-in-Chief
Pass@5 ≈ 99%at least one of five attempts is correct
Pass^5 ≈ 8%all five attempts in a row are correct (0.6 to the fifth power)
Key metric #3With daily ordering, the wrong coffee shows up in 92% of weeks

Imagine you have released an API for ordering coffee, and your customers are not people but AI agents. One of them places a "successful" order for a hot latte, even though the user asked for a cold one. The request goes through, the logs are green, monitoring stays quiet, and the person still doesn't get the drink they wanted.

Vercel proposes catching these failures with evals: the agent is given the same task many times, and you count how often the result is correct. In a demonstration run, Claude Code with the Opus 5.5 model handled the order in 3 attempts out of 5, or 60%.

The chance of getting at least one correct order in five attempts is about 99%, while the chance of getting five correct orders in a row is only about 8%. Next, we look at where these numbers come from and why the second one matters more for the product.

How an eval differs from an ordinary test

An ordinary test answers the question "did it work or not?" An eval answers "how often, and how well?" This matters because LLM responses change from run to run: one successful manual run proves nothing. To tell whether a fix helped, you need to repeat the tasks before and after the change and compare the results.

Vercel divides agent software tests by the number of model calls. Deterministic tests use no LLM at all: for example, they check that the search step returns only the right cafés and that the payment step charges exactly the amount stated in the calculation. A one-turn eval uses a single call and checks a single decision, such as which command the agent runs first.

An E2E eval uses several calls: the agent completes the task from start to finish, and the outcome is checked. The more calls a test makes, the more expensive and slow it is, so most checks should come from the cheap lower levels. But success depends on the whole chain (find a café, filter by the conditions, place the order), and each step on its own can look reasonable.

That is why checking success requires an E2E eval. Popular benchmarks such as SWE-bench and Terminal-Bench work the same way.

The coffee example: goal and metrics

Vercel's demo uses a fictional service, SF Beans, with the team name sfbeans. The customer's request: a large cold latte made with whole milk, ready by 9:00, under $8 including tax and tip, for a user in the Mission district. Three of the four cafés don't fit: Fog City Roasters has a broken ice machine, 24th St Beans charges $9.27 for the latte with tax and tip, and Dolores Drip's earliest pickup is 9:20. Only Valencia Grind fits: $7.42 and ready at 8:50.

Vercel advises choosing a narrow goal: one outcome that matters to users. You can find it in monitoring, customer complaints, or the roadmap. For sfbeans, most agents are Claude Code and Codex, and the most common complaint is "the agent ordered the wrong drink." The readiness bar is strict: the order must be correct in every attempt, not "mostly correct."

There are three metrics. Success rate is the share of attempts that pass. Pass@k is the chance that at least one of k attempts succeeds; it answers whether the agent can do the task at all. Pass^k is the chance that all k attempts succeed; it answers whether you can rely on the agent every time. Time, steps, and cost are counted only for successful attempts, because failed attempts are shorter: the agent skips work, for example by not asking for the final price.

In five attempts, the agent ordered from Valencia Grind three times. Once it skipped the calculation and ordered from 24th St Beans for $9.27, and once it chose Dolores Drip with a 9:20 pickup. The math: pass@5 = 1 − 0.4⁵ ≈ 99%, and pass^5 = 0.6⁵ ≈ 8%. With daily ordering, a person gets at least one wrong coffee in 92% of weeks. A successful attempt takes about 9 steps and 3 minutes 50 seconds on average and costs $0.56, but counting the paid failures, one success costs $0.78.

The cost of an error determines which metric comes first. A morning coffee can't be redone, so sfbeans looks at pass^5. A coding agent with a test suite can correct itself from feedback, so pass@k suits it. An email that a person will read before sending is usually judged by the ordinary success rate.

How to build a task set, and what not to expect

A good task consists of an instruction, an environment with the same starting state for every attempt, a verifier (a check of the final state, ideally one for each constraint), and optionally a reference solution. The set should include a typical case, edge cases, and at least one request where the right action is to do nothing. The example has four tasks: a standard order; "the favorite café doesn't make the right drink"; a group order for 6 drinks at $45 with a single pickup point; and an impossible request, where the agent must tell the user that nothing fits.

For the last one, the verifier reads a command log that the agent cannot write to, to confirm that it searched before giving up. You run all of this through Harbor, an open framework from the Terminal-Bench authors: it runs agents in containers, checks the results, and records their actions.

Some honest caveats. You cannot cover every possible action an LLM might take with your product, so start with one problem and a handful of tasks. Overall averages hide a task that fails all the time, so look at results for each task. And with five attempts, 100% only means "we haven't seen a failure yet," not "it never fails." The numbers in the coffee example are illustrative.

The practical takeaway: take the most common complaint about your agents, turn it into several tasks with automated checks, and run each one several times. Choose the metric based on the cost of an error, and for recurring scenarios, aim for reliability "every time."

Story

Key Questions Answered

How an eval differs from an ordinary test?

An ordinary test answers the question "did it work or not?" An eval answers "how often, and how well?" This matters because LLM responses change from run to run: one successful manual run proves nothing. To tell whether a fix helped, you need to repeat the tasks before and after the change and compare the results. Vercel divides agent software tests by the number of model calls.

What are the key details on The coffee example: goal and metrics?

Vercel's demo uses a fictional service, SF Beans, with the team name sfbeans. The customer's request: a large cold latte made with whole milk, ready by 9:00, under $8 including tax and tip, for a user in the Mission district. Three of the four cafés don't fit: Fog City Roasters has a broken ice machine, 24th St Beans charges $9.27 for the latte with tax and tip, and Dolores Drip's earliest pickup is 9:20. Only Valencia Grind fits: $7.42 and ready at 8:50.

How to build a task set, and what not to expect?

A good task consists of an instruction, an environment with the same starting state for every attempt, a verifier (a check of the final state, ideally one for each constraint), and optionally a reference solution. The set should include a typical case, edge cases, and at least one request where the right action is to do nothing.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…