Seq@K Metric Across 7 Benchmarks and 16 Models Reshuffles AI Rankings When Feedback Is Available

Researchers at Columbia Business School and the University of California, Santa Barbara have proposed a metric called Seq@K. It measures whether a language model solves a task by the K-th attempt at the latest, when it receives feedback after each failure and can revise its answer. The authors ran the protocol on 7 benchmark variants and 16 frontier models. Their conclusion: the familiar pass@K ranking, where attempts are independent and the model never sees any critique, is a poor predictor of how a model will behave in a real agentic loop of answer, remark, and revision.

How Seq@K differs from pass@K

Pass@K counts a task as solved if at least one of K independent answers passes verification. Every candidate is generated from the same original prompt. In Seq@K, each new attempt sees the previous answers and the comments on them. If an attempt succeeds, the loop stops.

The authors stress that this is not a replacement for agentic benchmarks such as SWE-bench, WebArena or MINT. There, success depends on planning, tool use and state tracking. Seq@K isolates a single skill: how well a model revises its own work in response to remarks.

The protocol depends on the feedback mode. There are three: a bare pass/fail verdict; raw diagnostics from the verifier (traces, failing tests, a breakdown by rubric item); and critique written by a language model, either the same one or a different one, with or without access to the grading rubric.

The task set covers code (MBPP Pro and HumanEval Pro), complex instruction following (AdvancedIF), domain-specific reasoning (CLBench DKR) and medical answers graded against rubrics (HealthBench).

Rankings, model selection and cost

The order of models does change when you switch to Seq@K. On AdvancedIF, Sonnet rises from last place under independent attempts into the top group, and CLBench DKR shows similar reshuffling.

The authors tested a practical consequence on a model-selection task. An engineer takes a few calibration tasks, picks a model, and then deploys it with sequential feedback on new tasks. A short Seq@K calibration turned out to pick the best model for that scenario at comparable or lower cost. The experiment ran on MBPP Pro and HumanEval Pro, with tasks split into two groups: calibrate on one, evaluate on the other, and vice versa. The largest gain came on HumanEval Pro.

The gap between models also depends on the metric. On AdvancedIF with raw feedback, the gap between the best and worst model shrinks. On HumanEval Pro and MBPP Pro it grows instead, because models use details about failing tests in different ways. So pass@K can both understate and overstate the differences between them.

There is a cost effect too. Sequential attempts drag along an ever-larger context, but success often comes sooner. On two benchmarks, the Pareto frontier (the set of options that cannot be improved on cost without losing quality) is better for Seq@K than for independent attempts. This is most visible on DKR: independent sampling solves only a limited share of tasks, while sequential feedback climbs noticeably higher. Opus 4.7 delivered the best result there.

How models revise answers and which critique works

Attempt trajectories reveal behavior that the final percentage hides. Some models gain on the first retry, others on later ones. Some models ignore the remarks and send back the same answer unchanged. And a revision that closes every remark can break requirements that were previously met.

The quality of the feedback itself also turned out to be a variable. How much informative critique helps depends on both the benchmark and the model's capability. Critique written with the grading rubric in view helps fix far more failed requirements. If the history of previous attempts is compressed into a short summary, the success rate may drop, so the authors advise reporting how the history is stored alongside the Seq@K result.

Limitations

Seq@K assumes privileged feedback: the critic or verifier knows the correct answer or the grading criteria. That happens when a user explains to an agent what exactly is wrong, but in autonomous scenarios without a verifier the metric cannot be reproduced. In addition, a Seq@K value is defined only together with the feedback mode, the verifier, the decoding parameters and the attempt budget, so numbers from different configurations cannot be compared.

The practical takeaway for teams building agents: if your product runs in an edit loop, calibrate the model in that same loop. The feedback itself should be designed and tested separately, as part of the system.