FrontierCode: Top AI Models Fail Maintainer Review in 42% of Attempts

Even the strongest coding models still often write changes that a maintainer would not merge. Artificial Analysis has published the first independent results for FrontierCode 1.1, a Cognition benchmark built together with maintainers of real open source projects. The leader, Claude Opus 5.5 (max with fallback), averages 52%. 58% of its attempts pass all blocking checks, and the other 42% end with a zero. Results are as of October 12, 2026.

What FrontierCode Actually Tests

Each task asks an agent to make a change in a real repository. Green tests are not the only thing that counts. The benchmark penalizes excessive edits and bloated diffs, changes to files outside the relevant area, and useless tests. Today's models solve coding tasks well, but their output often needs serious cleanup before merging. FrontierCode tries to measure exactly that gap.

Cognition built the set together with maintainers of 36 repositories, who spent more than 40 hours on each task. The benchmark was announced in June 2026. Version 1.1 came out in July and added rules for fair use of the internet. The verifier combines hidden tests and command checks, limits on the number of files touched and the amount of code changed, and a language-model review based on maintainers' criteria. Failing any blocking check scores 0. An attempt that passes all of them receives between 70% and 100%.

Artificial Analysis ran the main set of 100 tasks (the benchmark has 150 in total), with three attempts per task. All models ran in the same harness, the open agent harness Stirrup, with a limit of 250 turns. GPT-6 Luna (medium) served as the judge. An attempt in which Cognition's trajectory scanner found peeking at the solution also gets zero. The tasks are private, to rule out leakage into training data.

The Top of the Table

• Claude Opus 5.5 (max with fallback): 52% score, 58% pass rate, $10.15 per task.
• GPT-6.1 Sol (max) and Claude Fable 5.1 (max with fallback): 49% each. Sol costs $0.81 per task, Fable $12.00.
• The best open weights model is MiMo-V2.6-Pro, with 45% at $0.18 per task. DeepSeek V4.1 Flash (max) and Kimi K3 (max) follow with 44%.
• Nine models from six labs fall within 43–52%.
• Pass rate drops to 16% for Muse Glimmer (high).
• No model solved 9 of the 100 tasks in any attempt. No task was passed by every model in every attempt.

Price, Tokens, and Time

Cost per task varies by more than 100x: from $0.09 for GPT-6 Luna (max), which scored 42%, to $12.00 for Claude Fable 5.1. Four models sit on the Pareto frontier of score versus price: GPT-6 Luna (max) with 42% at $0.09, MiMo-V2.6-Pro with 45% at $0.18, GPT-6.1 Sol (max) with 49% at $0.81, and Claude Opus 5.5 with 52% at $10.15. The last step, from Sol to Opus, adds 3 points for a price increase of more than 12x.

Comparisons within model families are also telling. GPT-6.1 Sol scores 2 points higher than GPT-6 Astra (max) and costs five times less: $0.81 versus $4.22. Claude Sonnet 5.5 generates roughly 50% more output tokens than Opus 5.5, but because its price per token is half as high, it costs $8.26 versus $10.15.

GPT-6 models are the most economical in tokens. GPT-6 Astra and GPT-6.1 Sol use about 27 thousand and 33 thousand output tokens per task. Claude Opus 5.5 uses about 157 thousand, and Claude Sonnet 5.5 about 236 thousand, the most of any model. Time per task depends on both token count and output speed: from about 3 minutes for Inkling (xhigh) to about 46 minutes for Qwen3.8 Max (0902).

How to Read This Ranking

The practical takeaway for teams deploying coding agents: the 3–10 point gap between top models is often smaller than an order-of-magnitude difference in price. If an agent works autonomously and every PR still goes through human review, GPT-6.1 Sol or MiMo-V2.6-Pro look more rational on this data than the expensive flagships. Paying for the last 3 points over the leader makes sense only where the cost of an error is high.

Another useful metric: the leader's 42% failure rate means human review remains mandatory. It is worth tracking not just test passes but also diff size and side edits. FrontierCode is currently published as a separate leaderboard and is not part of the Artificial Analysis Intelligence Index. It is promised to be added to the Coding Agents leaderboard soon.