Apple tested LLMs as translation judges: MQM and ESA are hard for both humans and models

Apple ML Research has published a study on whether language models can replace human evaluators of machine translation. The authors tested LLMs on two widely used quality evaluation schemes, MQM and ESA, and compared their agreement with humans. The title of the paper already states the conclusion: both sides struggle, humans and models alike. One caveat: the available excerpt of the abstract breaks off at the comparison with human annotation and contains no numerical results, so what follows covers the problem setup and its implications, not numbers.

Two schemes, two levels of detail

MQM (Multidimensional Quality Metrics) is considered the most rigorous way to evaluate translation. An expert finds errors in the text, marks specific spans, assigns each one to a category (accuracy, fluency, terminology, and so on), and rates its severity. The method is precise but expensive: it requires qualified linguists and a lot of time.

ESA (Error Span Annotation) is simpler. The annotator also marks problematic spans, but without a detailed taxonomy, and gives an overall score. This is cheaper and faster, which is why the scheme is increasingly used for large-scale evaluation campaigns.

Both schemes work at the level of text spans, not just the final score. The model has to do more than say "the translation is good or bad": it must point out where exactly the error is and why. For an LLM, this is a markedly harder task.

What exactly was tested

The authors start from the premise that LLMs cannot be considered a reliable replacement for humans until they have been tested along every axis on which translation evaluation varies:

• evaluation schemes: MQM and ESA;
• language pairs: from widely spoken to less well-resourced ones;
• text domains;
• annotation granularity: from an overall score to individual spans;
• quality yardstick: the model's agreement with human annotators.

The approach is sensible. An LLM judge is often tested on a single dataset, shows good correlation, and the method is declared fit for use. Here, correlation is measured across different slices, and an overall score does not substitute for checking the accuracy of annotations of individual errors.

Why "hard for humans too"

The second half of the title is arguably more interesting than the first. Human annotation is taken as the reference, but the annotators themselves agree with each other far from perfectly, especially at the level of specific spans and error severity. One expert will flag a turn of phrase as a stylistic inaccuracy; another will skip it or treat it as a serious error.

The consequence for the whole field: "agreement with humans" has a ceiling set by how well humans agree with each other. If a model falls short of the reference, it is not always clear whether the model is at fault or the reference is noisy. Conclusions about the quality of LLM judges have to be read with an adjustment for the reliability of the annotation they are measured against.

How to use this

For a team building a translation or localization evaluation pipeline, this work yields several practical rules.

• Do not carry a result over from one language pair to another. Test the LLM judge separately for each pair and domain you need.
• Choose the scheme to fit the task. ESA-style annotation may be enough for routine quality monitoring; disputed or critical cases call for MQM and a live expert.
• Measure agreement among the human annotators themselves. Without this, it is impossible to tell where the ceiling for the model lies.
• Keep humans in the loop on a sample. LLM evaluation lowers cost but does not eliminate periodic calibration against experts.

The full text of the paper should be consulted at the original source: the final agreement coefficients and the breakdown by language will determine how confidently an LLM can be used where expert linguists are paid today.