Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarHabr AI→ original

Two AI Judges, 15 LLMs: Judges Favor Their Own Makers by 0.04 Points Out of 1

LLM leaderboards are often scored by other LLMs, and those judges can be biased. The author of neShemyaka!, a legal service, ran 15 models through 30 closed Russian Supreme Court cases, judged by Claude Opus 5.5 and Grok 4.7 on a 0-to-1 scale. The judges agree with each other (correlation 0.95), but each more often picks the answer from its own maker: Opus picked its own answer 4 times, Grok never. That is why a judge who is also a participant needs to be paired with a judge from another vendor.

AI-processed from Habr AI; edited by Hamidun News
Two AI Judges, 15 LLMs: Judges Favor Their Own Makers by 0.04 Points Out of 1
Source: Habr AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr AI ↗
Editor-in-Chief
Shift in favor of Anthropic models+0.040 (interval from +0.020 to +0.060)
Claude Fable 5.2 has no weak answersits worst of 30 scored 0.64
Legal-tuned GLM-5.3 FlashX Legal0.494 vs. 0.473, difference not significant

Imagine a cooking contest where one of the participants sits on the jury. Officially, the dishes are tasted blind, but a familiar chef's signature is recognizable. This is roughly what happens in language model leaderboards, where one LLM grades the answers of other LLMs, the so-called LLM-as-a-Judge approach.

The author of the legal service "neShemyaka!", which predicts court case outcomes from case materials, repeated the Lexometrica Ground Truth test in November 2026: 15 models, 30 closed cases from Russian Supreme Court practice, and two AI judges, Claude Opus 5.5 and Grok 4.

7. This time the judges were tested too, not only the models. The result: the judges broadly agree with each other (correlation 0.

95), but each more often considers the best answer to be from a model made by its own company, and the shift in favor of Anthropic models was +0.040 points on a 0-to-1 scale.

How the test works

Each of the 30 cases includes a description of the situation, a question, an expert rubric, and a reference answer. Almost half of the set (13 cases) tests the hardest skill: applying a rule to specific facts. Another 6 cases cover identifying legal issues, 6 cover drawing conclusions, 4 cover reproducing a rule, and 1 covers interpretation. The set is closed so that the cases do not end up in the models' training data.

Each judge received all 15 answers per case at once, under letter labels in random order, with model names stripped from the texts. An answer's final score is the average of the two judges, and a model's score is the average across the 30 cases. The models had no access to search or reference systems, so the results cannot be transferred directly to RAG products.

How they performed

Claude Fable 5.2 and Gemini 4 Carbon clearly outperform every model from fourth place down, but they are indistinguishable from each other: Fable comes out ahead in only 72% of bootstrap resamples. Fable is also the only model with no weak answers: its worst of 30 scored 0.64, while the next three models had minimums of around 0.4. Claude Opus 5.5 and GPT-6.2 Astra are separated by just 0.02 points. On thirty cases, only five or six quality levels are reliably distinguished, and the order of models within a group is effectively random.

Reasoning volume does not predict quality: Kimi K3.1 Pro reasons more than twenty times as much as Astra, yet scores 0.582 against 0.795. DeepSeek V4.1 Flash costs roughly two orders of magnitude less than the leader for a 0.17-point gap, and where a lawyer checks the answer anyway, the expensive model is hard to justify. The legal fine-tune GLM-5.3 FlashX Legal scored 0.494 against 0.473 for the standard GLM-5.3, and the difference is not significant (P = 0.76): the legal style did not add knowledge of Russian law. On two cases, YandexGPT replied "I cannot discuss this topic" to ordinary legal questions and received zero.

Strong models are weakest at applying rules to facts (0.74–0.85, versus 0.84–0.91 for reproducing a rule). Reasoning mode does not close the knowledge gap: from an incorrectly reproduced rule, a model derives a decision just as coherently as from a correct one.

How much you can trust the judge

To check this, the author compared the two judges' scores on 450 answers. The correlation across answers was 0.953, with a mean disagreement of 0.065; for 80% of answers the disagreement was no more than 0.10, and for 99% no more than 0.20. Re-scoring with Opus alone produced a disagreement of 0.041, while the Opus–Grok pair produced 0.066, so a judge from another vendor adds only a little noise of its own.

But the overall coefficient is misleading. Within a case, the rank correlation for the top six models is only 0.57, versus 0.92 for the bottom nine. The disagreement is the same in both groups (0.06–0.07), but the scores of strong answers sit in a narrow band, and that spread is enough to swap them. The judge reliably separates a good answer from a bad one, but it ranks two good answers poorly.

The bias toward "one's own" shows up in different ways. The difference between the average scores from Opus and Grok is positive for exactly two of the fifteen models, and both are from Anthropic: Opus 5.5 (+0.014) and Fable 5.2 (+0.012). The relative shift for Anthropic models is +0.040 with a 95% interval from +0.020 to +0.060, and the probability of randomly getting the top two positions for a single vendor is 1/105. The Opus judge named its own model's answer best 4 times, the Grok judge never, and Grok picked its own answer 5.7 times against 1.8 for Opus.

This affects the table directly. By Opus's scores alone, its model leads GPT-6.2 Astra by 0.053 (P = 1.00); by Grok's scores alone, the order reverses (0.809 vs. 0.819); and in the two-judge average the gap is 0.021 (P = 0.90). The two judges alone cannot establish where exactly the skew comes from: Opus may have inflated scores for its own model, or Grok may have deflated scores for others. Grok's own shift is not established from the averaged scores: a comparable boost also goes to GPT-6.2 Astra, YandexGPT, and GigaChat, which have no connection to it.

What to do in practice

If you build a quality evaluation on LLM-as-a-Judge, do not use as judge a model that takes part in the comparison, or at least check its scores against a judge from another vendor. Publish reliability separately for the top group, not only the overall coefficient. Show the minimum and the number of failures alongside the average: in legal work, the cost of an error is asymmetric, and a model with a good average but failures on individual cases is more dangerous than a uniform one.

There is also a caveat: 30 cases is a small sample, both judges are themselves competing in the ranking, and the author calls some of the conclusions a guideline rather than proven fact. He promises to release the full calculations as a preprint.

Story

Key Questions Answered

How the test works?

Each of the 30 cases includes a description of the situation, a question, an expert rubric, and a reference answer. Almost half of the set (13 cases) tests the hardest skill: applying a rule to specific facts. Another 6 cases cover identifying legal issues, 6 cover drawing conclusions, 4 cover reproducing a rule, and 1 covers interpretation. The set is closed so that the cases do not end up in the models' training data.

How they performed?

Claude Fable 5.2 and Gemini 4 Carbon clearly outperform every model from fourth place down, but they are indistinguishable from each other: Fable comes out ahead in only 72% of bootstrap resamples. Fable is also the only model with no weak answers: its worst of 30 scored 0.64, while the next three models had minimums of around 0.4. Claude Opus 5.5 and GPT-6.2 Astra are separated by just 0.02 points.

How much you can trust the judge?

To check this, the author compared the two judges' scores on 450 answers. The correlation across answers was 0.953, with a mean disagreement of 0.065; for 80% of answers the disagreement was no more than 0.10, and for 99% no more than 0.20. Re-scoring with Opus alone produced a disagreement of 0.041, while the Opus–Grok pair produced 0.066, so a judge from another vendor adds only a little noise of its own. But the overall coefficient is misleading.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…