Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseAPI / Product · ActionableVercel Blog→ original

An Agent With a 60% Success Rate Orders the Right Coffee at Least Once in Five Tries 99% of the Time, but All Five in a Row Only 8% of the Time

When an AI agent orders coffee through your API, the logs can look green even though it grabbed a hot latte instead of an iced one. Vercel recommends running a single task many times and measuring the share of successes. In the demo, the agent succeeded in 3 of 5 attempts, or 60%. The chance of at least one correct order in five attempts is 99%, but the chance of five correct orders in a row is only 8%. If a user needs the right result every morning, the second number is the one to watch.

AI-processed from Vercel Blog; edited by Hamidun News
An Agent With a 60% Success Rate Orders the Right Coffee at Least Once in Five Tries 99% of the Time, but All Five in a Row Only 8% of the Time
Source: Vercel Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceVercel Blog ↗
Editor-in-Chief
Pass@5 = 99%at least one of five orders is correct; pass^5 = 8%: all five are correct
Key metric #2Cheap tests without an LLM form the base of the pyramid, while expensive E2E runs sit at the top
Key metric #3Harbor, from the authors of Terminal-Bench, runs agents in containers

Imagine you built an API for ordering coffee, and your customer is not a person but an AI agent. It placed the order, the request went through, the logs are green, but the user asked for an iced latte and got a hot one. These silent failures are exactly what the Vercel team is breaking down. Their main example: the agent completes the task in 3 of 5 attempts, or 60%. If you give it five attempts, at least one will be correct with a probability of about 99%. But it will get all five orders right in a row only about 8% of the time. Both numbers are correct; they just answer different questions.

The Agent Is Also a User of Your Product

If you build an API, CLI, MCP server, or SDK, you now have a new type of client. The agent never saw your launch or your ads, but it goes through the same funnel as a human: it finds the product, chooses it, gets access, and tries to solve the task. At each stage, fewer agents make it to the end. Vercel focused on the last step, success: did the agent do what was asked?

Checking this is harder than it seems. LLM responses change from run to run, so one successful manual test does not prove that a fix helped. You need an eval: a test that you run many times and use to assess results before and after a change.

Vercel divides tests for agent software by the number of model calls. Deterministic tests do not call an LLM at all. One-turn evals use a single call and check one decision, such as which command the agent runs first. E2E evals let the agent complete the entire task and check the final outcome. SWE-bench and Terminal-Bench belong to this category. The higher the level, the more expensive and slower the test, so most of your suite should consist of cheap tests. But success can only be verified at the top level: each step on its own looks reasonable, yet the order still comes out wrong.

Coffee at 8:30: Three Metrics Instead of One

In the example, the SF Beans service takes coffee orders in San Francisco through the sfbeans CLI. The request: a large iced latte with whole milk, ready by 9:00, under $8 including tax and tip. Of four coffee shops, only Valencia Grind fits: at Fog City Roasters the ice machine is broken, at 24th St Beans the total comes to $9.27 after tax and tip, and at Dolores Drip the order will not be ready before 9:20.

The test is built on Harbor, an open framework from the creators of Terminal-Bench. It runs agents in containers, checks the result, and records every action. The check confirms that there is exactly one order, the drink is correct, the timing fits, and the budget is not exceeded.

The metrics are as follows:

  • Success rate: 3 of 5 attempts gives 60%.
  • Pass@k: the probability that at least one of k attempts succeeds. For five attempts, it is about 99%. This metric answers whether the agent can do the task at all.
  • Pass^k: the probability that all k attempts succeed. For five, it is about 8%. This metric answers whether you can rely on the agent every time.

In Vercel's illustrative table for Claude Code with Opus 5.5, a user who orders coffee every weekday gets at least one wrong order in 92% of weeks. For morning coffee, there is no retry, so SF Beans optimizes specifically for pass^5. For a coding agent with tests, pass@k matters more: it sees its own errors and tries again. For a draft email that a human will read anyway, a plain success rate is enough.

Steps, time, and cost are calculated only from successful runs, because failures are shorter: the agent skipped work, for example by not checking the price. In the demo, a successful run takes about 9 steps and 3:50 and costs $0.56. If you count the paid failures, the cost of one success rises to $0.78.

Where to Start

You cannot cover with tests everything an LLM might do with your product. Vercel advises picking one outcome that matters to users and looking for it in observability data, customer support tickets, and the roadmap. For SF Beans, the key signal was the ticket "agent ordered the wrong drink," so the goal is a correct order on every attempt from Claude Code and Codex, which is where most of the traffic comes from.

The task set includes a typical order, two edge cases (a favorite coffee shop that doesn't have the needed drink, and a group order of six drinks for $45), and a situation where the right move is to order nothing and report that there are no options. For the last one, the check reads a command log that the agent cannot write to, to confirm it searched before giving up.

The limitation is simple: with five attempts, 100% only means "we have not seen failures yet." Aggregate averages can hide a task that fails often, so look at results for each task separately. The practical order is this: pick one painful task, run it several times, and choose your main metric based on the cost of an error for the user.

Story

Key Questions Answered

What are the key details on The Agent Is Also a User of Your Product?

If you build an API, CLI, MCP server, or SDK, you now have a new type of client. The agent never saw your launch or your ads, but it goes through the same funnel as a human: it finds the product, chooses it, gets access, and tries to solve the task. At each stage, fewer agents make it to the end. Vercel focused on the last step, success: did the agent do what was asked? Checking this is harder than it seems. LLM responses change from run to run, so one successful manual test does not prove that a fix helped.

What are the key details on Coffee at 8:30: Three Metrics Instead of One?

In the example, the SF Beans service takes coffee orders in San Francisco through the sfbeans CLI. The request: a large iced latte with whole milk, ready by 9:00, under $8 including tax and tip. Of four coffee shops, only Valencia Grind fits: at Fog City Roasters the ice machine is broken, at 24th St Beans the total comes to $9.27 after tax and tip, and at Dolores Drip the order will not be ready before 9:20. The test is built on Harbor, an open framework from the creators of Terminal-Bench.

Where to Start?

You cannot cover with tests everything an LLM might do with your product. Vercel advises picking one outcome that matters to users and looking for it in observability data, customer support tickets, and the roadmap. For SF Beans, the key signal was the ticket "agent ordered the wrong drink," so the goal is a correct order on every attempt from Claude Code and Codex, which is where most of the traffic comes from.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…