semCER Weights OCR Errors by Severity: 61M LLM-Judge Ratings

Character Error Rate measures the distance between recognized text and the reference, then divides it by the reference length. To CER, a one-letter error in the name «Иванов» and an error in the word «и» in the middle of a paragraph cost the same. The paper's authors propose semCER, a metric that keeps CER's alignment and denominator but has an LLM judge score each grapheme-level edit. The sample is large: 61M rated edits across 16 languages, all released publicly.

How the metric works

The key engineering decision is that semCER does not break the familiar arithmetic. The alignment stays the same, and so does the denominator. Only the cost of each edit changes: instead of 1, the judge assigns it a weight based on severity. No word segmentation is required. That matters for Chinese, where spaces do not mark word boundaries, and for morphologically rich languages.

This approach keeps results comparable with earlier ones. If you have tracked your OCR pipeline's CER for years, switching does not require recomputing everything from scratch: the structure of the score is the same, and only the weights change.

What the reader validation showed

The authors compared the metric's scores with ratings from human readers in English, Vietnamese, and Chinese. semCER's weights match human ratings edit by edit, and the errors readers consider most severe land at the top of the sorted list. On pairs where CER and semCER disagreed, readers sided with semCER more often than with normalized CER in English and Chinese. In Vietnamese, the count is even.

There is also an honest caveat. For the overall score of a whole transcription, normalized CER correlates with readers as well as semCER does, and sometimes better. The value of semCER lies elsewhere: it breaks that score down into individual named edits. The metric becomes a diagnostic tool, not just another number.

Key results from the paper:

• 61M edits rated by an LLM judge, 16 languages;
• human validation: English, Vietnamese, Chinese;
• on disputed pairs, readers choose semCER more often than normalized CER in English and Chinese, and split evenly in Vietnamese;
• in Spanish, the judge rates a wrong diacritic or tone mark at almost zero; in Telugu, almost as a wrong letter;
• a distilled scorer is released with the data, so calling a large LLM for every rating is not required.

The spread across languages reflects reality: in Spanish, a lost accent rarely changes the meaning, while in Telugu the mark is part of the letter and determines which word a person reads. Flat CER cannot see this difference.

Errors that look like words

A separate result concerns the type of error. Where real words can be counted, a wrong letter in the output of a general-purpose VLM forms an existing word noticeably more often than chance. For a classical OCR pipeline, this share stays near chance. The takeaway is uncomfortable for practitioners: the language model corrects text toward the plausible, so an error becomes a phrase that reads smoothly but says something else. Such an error is harder to catch in proofreading than garbled characters.

How to use it in OCR evaluation

For teams comparing VLMs and classical engines on documents, the setup is simple. You can still compute the overall score with normalized CER. Run semCER on top of it for analysis: sort edits by severity and see which errors the model makes on names, numbers, and diacritics. This is especially useful for multilingual pipelines, where identical CER in Spanish and Telugu means different damage. The distilled scorer from the nguyenx2t/semcer repository lets you do this without calling an LLM judge for every document. The limitation is obvious: an LLM sets the weights, so for critical domains such as medicine or law, you should spot-check them against ratings from your own specialists.