semCER weights OCR errors by severity: 61 million LLM-judge scores

Character Error Rate measures the distance between recognized text and a reference, then divides it by the reference length. To CER, a wrong letter in the name "Иванов" and a mistake in the word "и" in the middle of a paragraph cost the same. The paper's authors propose semCER: it keeps CER's alignment and denominator, but an LLM judge scores each grapheme-level edit. The sample is sizable: 61 million scored edits across 16 languages, all released publicly.

How the metric works

The key engineering decision is that semCER does not break the familiar arithmetic. The alignment stays the same, and so does the denominator. Only the cost of each edit changes: instead of one, the judge assigns it a weight based on severity. No word segmentation is required, which matters for Chinese, where word boundaries are not marked by spaces, and for morphologically rich languages.

This approach keeps results comparable with earlier ones. If you have tracked your OCR pipeline's CER for years, switching does not require recomputing everything from scratch: the scoring structure is the same, and only the weights change.

What the reader checks showed

The authors compared the metric's scores with those of human readers in English, Vietnamese and Chinese. semCER's weights match human ratings edit by edit, and the errors readers consider most severe land at the top of the sorted list. On pairs where CER and semCER disagreed, readers sided with semCER more often than with normalized CER in English and Chinese. In Vietnamese the count is even.

There is an honest caveat. For the overall score of a whole transcription, normalized CER correlates with readers as well as semCER does, and sometimes better. semCER's value lies elsewhere: it breaks that score down into individual named edits. The metric becomes a diagnostic tool, not just another number.

Key results from the paper:

• 61 million edits scored by an LLM judge, 16 languages;
• human validation: English, Vietnamese, Chinese;
• on disputed pairs, readers pick semCER more often than normalized CER in English and Chinese, and split evenly in Vietnamese;
• in Spanish the judge scores a wrong diacritic or tone mark at almost zero, while in Telugu it scores it nearly like a wrong letter;
• a distilled scorer is released with the data, so calling a large LLM for every score is not necessary.

The spread across languages reflects reality: in Spanish, a lost accent rarely changes the meaning, while in Telugu the mark is part of the letter and determines which word a person reads. Flat CER cannot see this difference.

Errors that look like words

A separate result concerns the type of errors. Where real words can be counted, a wrong letter in a general-purpose VLM's output forms an existing word noticeably more often than chance. For a classic OCR pipeline, that share stays near chance. The practical takeaway is unwelcome: the language model corrects text toward the plausible, and the error becomes a phrase that reads smoothly but says something else. Such an error is harder to catch in proofreading than gibberish.

How to use it in OCR evaluation

For teams comparing VLMs and classic engines on documents, the setup is simple. You can still compute the overall score with normalized CER. Run semCER on top of it for analysis: sort edits by severity and see which errors the model makes on names, numbers and diacritics. This is especially useful for multilingual pipelines, where identical CER on Spanish and Telugu means different damage. The distilled scorer from the nguyenx2t/semcer repository lets you do this without calling an LLM judge for every document. The limitation is obvious: an LLM sets the weights, so in critical domains such as medicine or law, they should be spot-checked against the ratings of your own specialists.