Direct LLM scoring ranks papers as well as Elo, but rejects 4.3% of weak papers versus 66.4%

Major ML conferences receive more than 10 000 submissions a year, but the pool of reviewers is not growing. Researchers from Indiana University Bloomington, Renmin University of China and the University of Antwerp tested the best way to have a language model score papers. They took 232 ICLR 2025 submissions and compared five schemes.

Direct scoring (Direct) gives each paper its own score from 1 to 10. Elo derives a ranking from pairwise comparisons, in which the model picks the better of two papers. On paper ordering, both schemes track reviewer scores about equally well.

But when the scores are turned into an accept-or-reject decision, direct scoring filters out only 4.3% of the papers that reviewers rated below the acceptance threshold. Elo filters out 66.

4%.

Why good ordering does not produce a good decision

Direct scoring squeezes almost all predictions above the acceptance threshold, so weak papers pass along with strong ones. To find the cause, the authors ran a control experiment: they fitted each scheme's threshold to its own share of accepted papers. This showed that the comparative approach's advantage is placing scores correctly on the scale, not better ordering. After recalibration, direct scoring reaches higher agreement with humans than Elo while making just 24% of its API calls.

The practical takeaway: you cannot judge a review system by a single correlation. You need to look at calibration, the threshold, the choice of model, and exactly which outcome is being predicted.

How the experiment was set up

The sample was stratified by the mean reviewer score on OpenReview. The authors took 30 papers from each bin from 2 to 8, and all available papers from the sparse bins 1, 9 and 10: 7, 13 and 2 papers respectively. The sample's mean score was 5.16, with a standard deviation of 2.16.

They compared two groups of schemes. Absolute schemes assign a score on a scale: Direct, Anchor and Group Rating. Anchor adds three similar papers with known scores to the prompt, drawn from a set of 100 ICLR 2024 papers with unanimous ratings. Group Rating scores five papers at once, producing 46 groups. Comparative schemes derive a ranking from head-to-head matchups: Elo (an average of 10 comparisons per paper, 1 160 in total) and an 8-round Swiss tournament. The rankings from the comparative schemes were mapped to a 1–10 scale, so score inflation and other absolute errors cannot be directly compared across the two families.

The inputs varied too: three representations of each paper, including the title with the abstract only, and a 200–300-word LLM-written summary. All schemes were repeated on three models.

What else the study found

• The abstract alone is enough: it gives better results than a summary of the full paper.
• The choice of model affects results as much as the choice of scheme.
• Anchor performs poorly. The authors attribute this to the anchor set: papers with zero score variance cover a narrow part of the scale (standard deviation 1.66 with a mean of 5.14, maximum 8.0).
• Repeating the experiment on all 11 513 papers with scores reproduced the patterns. Absolute agreement depends on the spread of scores in the sample being evaluated, and no scheme beat the "always reject" strategy on accuracy.
• Human scores predict citations better than any scheme. The exception is Group Rating: its correlation with citations persists after adjusting for reviewer scores, and it is the only scheme that ranks papers among those already accepted.

Caveats about the "gold standard"

The reviewer scores themselves are noisy: there are 4.05 reviewers per paper on average. As a result, the ceiling for correlation with this benchmark is well below one, and results should be read against the ceiling for the sample, not against 1.

The work has a practical context. Organizers are already deploying LLMs: ICLR 2025 ran a randomized test of prompts for reviewers, and AAAI 2026 added an AI review to every paper in the main track. The study suggests that cheap direct scoring from the abstract, with recalibration, can serve as a baseline. But it is too early to base acceptance decisions on it.