Cohere taught a 7B model to assess its confidence twice, cutting calibration error by 52.5%

Cohere has published CALIBER, a method that trains a reasoning model to estimate its own odds of success twice: before it starts reasoning and after the answer is ready. On the BigMathDigits math benchmark, a 7B model cut its expected calibration error (ECE) from 0.059, the figure for the strongest single-estimate baseline, to 0.028. That is 52.5% lower. The work's author is Cohere research scientist Beyza Ermis.

One score hides two questions

Before reasoning, the model sees only the problem statement. It can recognize the topic and gauge whether the task looks manageable. Its confidence answers the question "how likely am I to solve this?" After reasoning, it has something concrete to check, its own solution, and the question changes: "how likely is this answer to be correct?"

Cohere gives an example with two dice and a sum of at least 10. The model starts at 25% confidence, enumerates the outcomes (three ways to roll 10, two for 11, one for 12), gets 6 out of 36, or 1/6, and finishes at 88%. The reverse path is also possible: the solution looks simple, but an erroneous assumption surfaces along the way and confidence drops. Standard reasoning models do not guarantee this ability: a detailed chain of thought does not mean the model can honestly report its doubts.

The training targets for the two estimates differ. For initial confidence, the model attempts the same problem several times. If seven of ten attempts succeed, the target is 70%. For final confidence, the target is binary: a specific answer is either correct or not. If three of four attempts succeed, the initial target is 75%, while a failed attempt gets a final target of 0. Calibration rewards are added to the reward for a correct answer. Admitting doubt about a wrong answer improves the calibration reward but earns no reward for solving the problem.

What the experiments showed

The 7B and 30B models were trained on 30 000 math problems and tested on 1000 held-out problems. Three QA benchmarks were used as well. All methods were compared under the same training setup, and the baselines marked "lite" are simplified adaptations of earlier approaches. For CALIBER, metrics were computed on final confidence.

• BigMathDigits, 7B: ECE 0.028 versus 0.059 for the best baseline (−52.5%).
• AUROC 0.806 and Brier score 0.130 are the best results in the comparison.
• Transfer without additional training or task-specific checkpoint selection: ECE is lower by 26.8% on GPQA, 32.2% on TriviaQA and 36.2% on SimpleQA.
• BigMathDigits, 30B: ECE 0.022 versus 0.023 for CoCA-lite, but AUROC 0.797 versus 0.508.

The 30B result shows why two metrics are needed. On ECE, CoCA-lite is barely behind, yet its confidence hardly separates correct answers from incorrect ones: an AUROC of 0.508 is close to a coin flip. A model that always says "70%" at 70% accuracy is perfectly calibrated and still useless for deciding what to check. Cohere adds a caveat: some other baselines achieve slightly higher AUROC than CALIBER, so the advantage depends on which property is measured.

The researchers also looked separately at how confidence shifts after reasoning. On BigMathDigits, the 7B model's average estimate for correct answers rose from 78.5% to 83.5%, while for incorrect answers it fell from 62.6% to 54.5%. On other datasets, confidence sometimes decreases in both groups, but more sharply for incorrect answers. A rise or fall in itself is not a verdict: two answers that end at 70% express the same probability, even if one path went up and the other went down.

Where this is useful and what has not been tested

For product teams, the practical value lies in routing. Low initial confidence can signal that a task should go to a stronger model or a human before any tokens are spent on reasoning. Low final confidence indicates which answers to send for manual review when there is time to check only a few out of a hundred. This is where AUROC matters more than a good-looking ECE.

The limitations should be kept in mind. The experiments measure confidence before and after a single pass of reasoning. Cohere did not test using the updated estimates to drive decisions such as "verify," "look up additional information" or "abstain." Training was mostly on math, and transfer to QA is shown only for the 7B model and only by ECE. Before building such estimates into agentic pipelines, teams need to run them on their own data and check calibration on their own distribution of tasks.