LLMs Judge Translation Across 70 Language Pairs: They Beat Humans Only in Places

Language models increasingly act as judges of machine translation: they are cheaper and faster than human annotators. But how far can they be trusted? The researchers compared LLMs with humans under two evaluation schemes, on their own set of 70 language pairs with long context and on the public WMT23 and WMT25 data. The outcome is mixed. On some tasks, models agree with annotators better than annotators agree with each other. In most configurations, however, the result remains unreliable, for models and humans alike.

What was measured

The study uses two widely adopted schemes. MQM (Multidimensional Quality Metrics) requires an expert to find errors, assign them to categories and rate their severity. It is detailed and expensive annotation. ESA (Error Span Annotation) is simpler: the annotator highlights problematic spans of text and gives an overall score.

The authors compared two levels of agreement: on final scores and on the boundaries of the error spans themselves. Agreement between humans served as the reference. This makes it possible to check not only how closely a model resembles a human, but also how reproducible human annotation is in the first place.

Where humans stumble and where models do

The main finding concerns variance. Agreement changes with the annotation scheme, the language pair and the domain, so a single averaged correlation with humans says little about whether a model is suitable for a specific task. The two sides have different weak points:

• humans perform worst on detailed MQM annotation and on low-resource language pairs;
• LLMs distinguish poorly between minor errors;
• models confuse language varieties, meaning they fail to notice that a translation is written in the wrong regional norm;
• marking the boundaries of error spans is harder for models than scoring the overall quality;
• on some tasks, LLM agreement with humans exceeds agreement among humans, but this does not cancel out the overall instability.

The last point matters for interpretation. If annotators disagree with each other, the bar for a model is low. In that case, "the model agrees with a human no worse than a human agrees with another human" signals noise in the reference more than model quality.

What is missing and what the authors propose

The abstract gives neither specific agreement coefficients nor a list of the models tested, so comparing the numbers with other benchmarks requires reading the full paper. The conclusion itself is stated clearly: both sides have room for improvement, and the solution may be a joint annotation pipeline in which a human and an LLM cover each other's weak spots.

The practical logic of this approach is clear. A model can handle a first pass over a large volume of text and produce a rough assessment. Humans should step in where the model errs systematically: on minor errors, regional language varieties and precise span annotation. Rare language pairs need separate attention, because human annotation suffers there too.

How to use the result in practice

For a team building translation quality evaluation on an LLM judge, the paper suggests several steps:

• check agreement with humans separately for each language pair and domain, not against the overall average;
• do not rely on the model to annotate error spans without spot checks by humans;
• test separately whether the judge can notice a wrong language variety and minor defects;
• measure agreement among the annotators themselves, otherwise it is impossible to tell what the metric actually shows;
• for rare languages, plan a hybrid scheme with mandatory human involvement.

For those choosing a judge to rank translation systems automatically, the takeaway is pragmatic: an LLM works as a tool for speeding up evaluation, but not as an unconditional replacement for an expert. As long as reliability depends on the task, there will be no universal "default evaluation."