DecisionPivot Tests 40 Solver Models: Up to a 26-Point Gap at Equal Accuracy

A solver model in your LLM pipeline may score high on a public test, but that does not mean it makes correct decisions. The authors of DecisionPivot built a benchmark of 48 tasks and 40,033 items to show the difference. Two 27B-parameter models differ by just 2.6 points on the reference track, yet score 70.1% and 44.2% on contrastive-group accuracy. The gap is 25.9 points, and a standard aggregate metric does not see it.

How the benchmark works

DecisionPivot evaluates models that handle query routing, guardrails, and output verification. It defines seven decision roles in total. The benchmark has two parts.

The primary track includes 7,800 items from 13 generated task families. The items are assembled into contrastive groups: close variants of the same task in which one key detail changes, and the correct answer changes with it. Answers are computed programmatically and checked by independently implemented solvers. The reference track consists of 32,233 items from 35 public test sets and keeps their original metrics.

• 48 tasks and 40,033 items, seven decision roles.
• 7,800 items in the primary track, 13 task families.
• 32,233 items in the reference track, 35 public sets.
• 40 solver models and 14 general-purpose models.
• Metrics: item-level and contrastive-group accuracy, probability ordering, calibration, empirical selective coverage.

Where aggregate scores break down

The main result concerns contrastive groups. A model gets credit for a group only if it answers every variant correctly. The metric therefore catches cases where a solver reacts to surface features rather than to the detail that determines the answer. The pair of 27B models with nearly identical reference-track results shows how much this matters.

The second finding concerns probabilities. On 2,400 binary contrastive pairs, Perplexity Decider ranks the positive item above the negative one in 92.3% of cases. But it answers both items in the pair correctly in only 74.1%. The model orders options well by confidence, but its decision threshold fires imprecisely. For systems where a threshold drives the decision (let a request through or block it), this matters.

Third, models with similar item-level accuracy differ in how reliable their high-confidence predictions are. In selective scenarios, where the system automatically handles only high-confidence cases and sends the rest to a human, this determines the real error rate in the automated stream.

Separately, the authors tested the effect of reasoning mode. On a subset of 1,296 items, GLM-5.3 reaches 93.5% contrastive-group accuracy at a higher reasoning level and 81.5% at a minimal one. The 12-point difference comes from the same model.

How to use the results in practice

A team choosing a model for routing, moderation, or verification should add contrastive tests to its standard evaluation. To do this, take a set of close pairs in which one deciding detail changes, and compute the share of pairs solved in full. It is also worth measuring calibration and the error rate among high-confidence predictions separately: that rate determines how many decisions can be handed to automation.

Teams should also fix the reasoning mode when comparing models. The GLM-5.3 result shows that the same checkpoint can behave differently depending on its settings, so a default-settings comparison says little about production behavior. The authors note that they evaluated models using documented, model-specific protocols, and that is a sensible approach to adopt in internal tests.