Airbnb switched coding-agent models — heavy users' spending dropped 7x
Airbnb chooses coding-agent models by the cost of a whole task, not by benchmarks. Model A won in tests by 3+ points, but in production each request got three times more expensive and engineers sent three times as many requests. Heavy users were moved to a model with a task cost 6 times lower, and their spending fell 7x. The takeaway: check benchmark results against the real bill after rollout.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Imagine you swap the coding-agent model your team uses for the one that looks best in tests, and a few weeks later accounting sends a bill several times larger than expected. That is exactly what happened at Airbnb. On internal tasks, Model A scored more than three points higher than the previous leader at roughly the same price.
In production, though, each request became three times more expensive, and engineers started sending three times as many requests. When the heavy users were moved to a different model, their spending on the primary model dropped about 7x. The company described in an AWS blog how to avoid this trap on Amazon Bedrock: don't track the price per million tokens, track the cost of a whole task (cost per task) — the price of all the tokens an agent spends to bring one task to completion.
Why benchmarks don't answer the main question
Thousands of employees use AirChat, Airbnb's internal tool for running coding agents, every week. All requests go through an LLM gateway on top of Bedrock. Eight new models shipped over five weeks, and by mid-October about two-thirds of requests were already going to them. Each model changes output token counts, price, latency and quota load all at once, and the charts in a launch announcement don't show how it will handle your own engineers' work.
So Airbnb runs two systems. Before rollout, it measures score and cost per task on its own tasks for every model and every level of reasoning effort — a setting that controls how much the model "thinks" before answering. After rollout, it checks what changed in production.
An agent makes many requests per task, and on each one the model re-reads the accumulated context. For that reason Airbnb counts four types of tokens: regular input, cache reads, cache writes and output. The averages include failed attempts. On one model, moving from low to high effort gave +0.8 points at a cost per task 2.5 times higher. Data like this sets the effort ceilings, which the gateway actually enforces.
Evaluation is fast: on release day, a single request launches a campaign across seven task sets, and by morning the model has a score and a cost. Each task is a folder with instructions, a Dockerfile, a verification script and a reference solution. To get started, the authors recommend the open-source Harbor framework. The hardest part is writing the rubric: long tasks run in several rounds with a simulated reviewer, and a judge model grades the work on eight criteria. If the reviewer had to repeatedly pull the agent back on track, the final score is capped at 25 out of 100.
When a good score turned into a bill
Model A was rolled out with effort capped at high, and users were asked to use low or medium. A few weeks later, per-user spending turned out to be several times higher. Output tokens made up only a tenth of the cost of a request; most of the money went to cached context that was re-read again and again. The model could work independently for a long time, and people launched long autonomous sessions.
Later, a model came out that scored higher on the same tasks at a cost per task about 6 times lower. Model A's users were moved to it with a configuration change. In the first full week, the number of requests stayed the same, while the group's spending on the primary model fell about 7x.
Monitoring, quotas and cost attribution
When latency rises, Airbnb looks at three metrics: TimeToFirstToken, OutputTokenCount and the derived generation speed (OTPS = OutputTokenCount / (InvocationLatency − TimeToFirstToken) × 1000). In the authors' example, latency rose 40% while speed stayed the same: someone had edited the system prompt, and responses became 50% longer. The authors advise marking prompt and model changes on the dashboard as deploys.
Quotas cause a different kind of confusion. Bedrock reserves quota for a request based on uncached input tokens, cache writes and max_tokens. Cache reads don't count against the quota, so a request can be cheap for the quota and expensive for the bill.
In the example, with a quota of 1 million tokens per minute and 300 requests per minute, the dashboard showed about 30%, but once the 5:1 multiplier for output tokens was applied, actual usage came to about 77%. The reservation per request was 4696 tokens, even though responses fit within roughly 800 tokens at p99. Lowering max_tokens to 1024 cut the reservation to 1624 tokens, about 65% less, and the quota let more requests through.
Truncated responses are tracked via stopReason.
For each major model, Airbnb keeps an equivalent, based on evaluation results, to which traffic can be rebalanced through configuration. For models with Legacy status, it sets an alarm on LegacyModelInvocations. Costs per team are calculated through IAM role tags and application inference profiles, while request metadata goes only to logs, not to the bill.
What to take away
- Count the cost of a whole task, not the price per million tokens.
- After rollout, check evaluation results against the cache share and the number of requests per person.
- Cap max_tokens and effort, and keep a fallback model.
Limitation: prices are taken from the public price list, not actual discounts, and you need your own set of tasks with a reliable grader.
- Nov 18, 2026AWS shows how to route Bedrock agents to private APIs through a central firewall
- Nov 18, 2026ffmpeg-skill gives Claude Code and Cursor 42 local video-editing tools
- Nov 18, 2026Huang steps down as NVIDIA CEO, company loses more than $1 trillion in market cap
- Nov 18, 2026ElevenLabs Voice Agents Reach Patients in 3.5 Minutes Instead of 2 Hours
Key Questions Answered
Why benchmarks don't answer the main question?
Thousands of employees use AirChat, Airbnb's internal tool for running coding agents, every week. All requests go through an LLM gateway on top of Bedrock. Eight new models shipped over five weeks, and by mid-October about two-thirds of requests were already going to them. Each model changes output token counts, price, latency and quota load all at once, and the charts in a launch announcement don't show how it will handle your own engineers' work. So Airbnb runs two systems.
When a good score turned into a bill?
Model A was rolled out with effort capped at high, and users were asked to use low or medium. A few weeks later, per-user spending turned out to be several times higher. Output tokens made up only a tenth of the cost of a request; most of the money went to cached context that was re-read again and again. The model could work independently for a long time, and people launched long autonomous sessions. Later, a model came out that scored higher on the same tasks at a cost per task about 6 times lower.
What are the key details on Monitoring, quotas and cost attribution?
When latency rises, Airbnb looks at three metrics: TimeToFirstToken, OutputTokenCount and the derived generation speed (OTPS = OutputTokenCount / (InvocationLatency − TimeToFirstToken) × 1000). In the authors' example, latency rose 40% while speed stayed the same: someone had edited the system prompt, and responses became 50% longer. The authors advise marking prompt and model changes on the dashboard as deploys. Quotas cause a different kind of confusion.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.