Cursor AI exposes flaws in AI coding benchmarks
Cursor has published five charts showing how it evaluates coding models, and in doing so it effectively challenges nearly every public AI benchmark. Its central argument: the share of solved tasks is not the only thing that matters. So do the cost in tokens, the type of real-world requests, and how well offline scores track a model's behavior in the product when developers use it in practice.
AI-processed from Habr AI; edited by Hamidun News
On March 11, 2026, Cursor AI published an explanation of how it compares models inside its product, and in the process unexpectedly took aim at the entire industry of AI coding benchmarks.
Why do AI benchmarks often get model choice wrong for coding?
A coding model cannot be judged solely by the share of tasks it solves. Cursor showed that the correctness of the answer and the speed of execution (the median number of tokens per completion) matter more.
Why are ordinary AI coding benchmarks wrong?
A coding model cannot be judged solely by the share of tasks it solves. Cursor showed that you need to account for the correctness of the answer and the median number of tokens per completion.
What did Cursor propose instead of leaderboards?
The company showed charts that describe real value to a developer better than the familiar percentages of solved tasks.
- Nov 1, 2026Local models lose to Opus and Codex: 0.21 on a MacBook versus 1.00 for Codex
- Nov 1, 2026Kimi scores 96.2% on a 1C database analytics test and emerges as a fallback if Claude access is blocked
- Nov 1, 2026OpenAI Rewrites GPT Behavior Rules: Five Values and a Chain of Command
- Oct 31, 2026Author puts his script before an AI jury: the verdict cost 850 thousand tokens
Key Questions Answered
Why do AI benchmarks often get model choice wrong for coding?
A coding model cannot be judged solely by the share of tasks it solves. Cursor showed that the correctness of the answer and the speed of execution (the median number of tokens per completion) matter more.
Why are ordinary AI coding benchmarks wrong?
A coding model cannot be judged solely by the share of tasks it solves. Cursor showed that you need to account for the correctness of the answer and the median number of tokens per completion.
What did Cursor propose instead of leaderboards?
The company showed charts that describe real value to a developer better than the familiar percentages of solved tasks.
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.