An Agent With a 60% Success Rate Orders the Right Coffee at Least Once in Five Tries 99% of the Time, but All Five in a Row Only 8% of the Time
When an AI agent orders coffee through your API, the logs can look green even though it grabbed a hot latte instead of an iced one. Vercel recommends running a single task many times and measuring the share of successes. In the demo, the agent succeeded in 3 of 5 attempts, or 60%. The chance of at least one correct order in five attempts is 99%, but the chance of five correct orders in a row is only 8%. If a user needs the right result every morning, the second number is the one to watch.
AI-processed from Vercel Blog; edited by Hamidun News
Imagine you built an API for ordering coffee, and your customer is not a person but an AI agent. It placed the order, the request went through, the logs are green, but the user asked for an iced latte and got a hot one. These silent failures are exactly what the Vercel team is breaking down. Their main example: the agent completes the task in 3 of 5 attempts, or 60%. If you give it five attempts, at least one will be correct with a probability of about 99%. But it will get all five orders right in a row only about 8% of the time. Both numbers are correct; they just answer different questions.
The Agent Is Also a User of Your Product
If you build an API, CLI, MCP server, or SDK, you now have a new type of client. The agent never saw your launch or your ads, but it goes through the same funnel as a human: it finds the product, chooses it, gets access, and tries to solve the task. At each stage, fewer agents make it to the end. Vercel focused on the last step, success: did the agent do what was asked?
Checking this is harder than it seems. LLM responses change from run to run, so one successful manual test does not prove that a fix helped. You need an eval: a test that you run many times and use to assess results before and after a change.
Vercel divides tests for agent software by the number of model calls. Deterministic tests do not call an LLM at all. One-turn evals use a single call and check one decision, such as which command the agent runs first. E2E evals let the agent complete the entire task and check the final outcome. SWE-bench and Terminal-Bench belong to this category. The higher the level, the more expensive and slower the test, so most of your suite should consist of cheap tests. But success can only be verified at the top level: each step on its own looks reasonable, yet the order still comes out wrong.
Coffee at 8:30: Three Metrics Instead of One
In the example, the SF Beans service takes coffee orders in San Francisco through the sfbeans CLI. The request: a large iced latte with whole milk, ready by 9:00, under $8 including tax and tip. Of four coffee shops, only Valencia Grind fits: at Fog City Roasters the ice machine is broken, at 24th St Beans the total comes to $9.27 after tax and tip, and at Dolores Drip the order will not be ready before 9:20.
The test is built on Harbor, an open framework from the creators of Terminal-Bench. It runs agents in containers, checks the result, and records every action. The check confirms that there is exactly one order, the drink is correct, the timing fits, and the budget is not exceeded.
The metrics are as follows:
- Success rate: 3 of 5 attempts gives 60%.
- Pass@k: the probability that at least one of k attempts succeeds. For five attempts, it is about 99%. This metric answers whether the agent can do the task at all.
- Pass^k: the probability that all k attempts succeed. For five, it is about 8%. This metric answers whether you can rely on the agent every time.
In Vercel's illustrative table for Claude Code with Opus 5.5, a user who orders coffee every weekday gets at least one wrong order in 92% of weeks. For morning coffee, there is no retry, so SF Beans optimizes specifically for pass^5. For a coding agent with tests, pass@k matters more: it sees its own errors and tries again. For a draft email that a human will read anyway, a plain success rate is enough.
Steps, time, and cost are calculated only from successful runs, because failures are shorter: the agent skipped work, for example by not checking the price. In the demo, a successful run takes about 9 steps and 3:50 and costs $0.56. If you count the paid failures, the cost of one success rises to $0.78.
Where to Start
You cannot cover with tests everything an LLM might do with your product. Vercel advises picking one outcome that matters to users and looking for it in observability data, customer support tickets, and the roadmap. For SF Beans, the key signal was the ticket "agent ordered the wrong drink," so the goal is a correct order on every attempt from Claude Code and Codex, which is where most of the traffic comes from.
The task set includes a typical order, two edge cases (a favorite coffee shop that doesn't have the needed drink, and a group order of six drinks for $45), and a situation where the right move is to order nothing and report that there are no options. For the last one, the check reads a command log that the agent cannot write to, to confirm it searched before giving up.
The limitation is simple: with five attempts, 100% only means "we have not seen failures yet." Aggregate averages can hide a task that fails often, so look at results for each task separately. The practical order is this: pick one painful task, run it several times, and choose your main metric based on the cost of an error for the user.
- Nov 21, 2026A Developer Had Claude Install 900 Apps and Built a Free Mac Cleaner
- Nov 21, 2026Apple Is Building an AI Agent and Making It Harder for Outsiders to Get Access: Permission Now Takes 14 Steps, Not 4
- Nov 21, 2026Qwen3-0.6B Fine-Tuned on a Mac mini: Errors Dropped to 3 out of 30 from 9
- Nov 21, 2026The Programmer Stopped Being an AI Agent Dispatcher: Multica Takes Over the Handoff Routine
Key Questions Answered
What are the key details on The Agent Is Also a User of Your Product?
If you build an API, CLI, MCP server, or SDK, you now have a new type of client. The agent never saw your launch or your ads, but it goes through the same funnel as a human: it finds the product, chooses it, gets access, and tries to solve the task. At each stage, fewer agents make it to the end. Vercel focused on the last step, success: did the agent do what was asked? Checking this is harder than it seems. LLM responses change from run to run, so one successful manual test does not prove that a fix helped.
What are the key details on Coffee at 8:30: Three Metrics Instead of One?
In the example, the SF Beans service takes coffee orders in San Francisco through the sfbeans CLI. The request: a large iced latte with whole milk, ready by 9:00, under $8 including tax and tip. Of four coffee shops, only Valencia Grind fits: at Fog City Roasters the ice machine is broken, at 24th St Beans the total comes to $9.27 after tax and tip, and at Dolores Drip the order will not be ready before 9:20. The test is built on Harbor, an open framework from the creators of Terminal-Bench.
Where to Start?
You cannot cover with tests everything an LLM might do with your product. Vercel advises picking one outcome that matters to users and looking for it in observability data, customer support tickets, and the roadmap. For SF Beans, the key signal was the ticket "agent ordered the wrong drink," so the goal is a correct order on every attempt from Claude Code and Codex, which is where most of the traffic comes from.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.