Ingosstrakh Teaches a Multimodal LLM to Doubt Itself: Catching Document Errors with Logprobs

The Advanced Technologies team at the AI Development Center of SPAO Ingosstrakh has published a write-up on Habr about how to catch errors in the responses of a multimodal LLM that pulls structured data directly from incoming documents, bypassing OCR. The tool they chose needs no separate evaluator model: it is logprobs, the logarithms of token probabilities that the model already returns during generation. The question itself is a practical one: which of thousands of processed documents should go to a human, and which can pass downstream automatically.

Valid JSON without guarantees

The task belongs to the class of Intelligent Document Processing (IDP). The input is a scan or photo of a document; the output is JSON with fields such as full name, passport number, dates and amounts. The multimodal model reads the image itself, with no intermediate text recognition.

This setup has an unpleasant property. An error in a single digit of a passport number or a single letter of a surname does not break the response format. The structure is correct, the parser is satisfied, and the wrong data goes to the next system looking like reliable data. Customers noticed this quickly and started asking for a confidence score: an estimate that can be used to filter out suspicious results.

In classic OCR pipelines such as CRNN, Surya, EasyOCR or Tesseract, such a score is usually available per character, word or line. The authors add a caveat: it does not guarantee correctness and may require calibration, but it serves as a starting signal for selection. Ingosstrakh's LLM system had no such signal.

Where the signal comes from

At each step, the model chooses the next token based on the image, the instruction and the text it has already generated. A logprob can be retrieved for the chosen token. The lower the value, the less expected the continuation was for the model itself.

An important caveat from the article: logprobs do not answer the question "was the field recognized correctly?" They only show how confidently the model followed its own trajectory. A model can be confidently wrong, and a correct answer can get a low probability because of an unusual format or a poor scan. For that reason, the team does not use raw probabilities directly. It builds a risk score from them and checks it against real errors in the company's data.

What is known from the published introduction:

• task: extracting structured data from incoming documents (IDP);
• architecture: a multimodal LLM with no separate OCR step;
• business request: a confidence score for routing documents to manual review;
• signal source: token logprobs of the selected tokens;
• validation: on real data; specific quality metrics are given in the full article.

The introduction contains no numerical results, so it is too early to judge how well the risk score separates wrong answers from correct ones.

How to apply the approach

Below are my own conclusions from the problem statement, not a retelling of the authors' results.

First: score fields, not the whole document. Average confidence across a response easily dilutes a single doubtful token in a passport number, so it makes sense to look at the minimum, or another aggregation, over the tokens of a specific field.

Second: set the review threshold according to the cost of an error. For full names and identity documents it should be stricter than for optional fields.

Third: without a labeled sample, the signal is useless. To find out whether low probabilities correlate with real errors, you need ground-truth answers and a measurement of the share of caught errors against the load on operators. The calibration the authors mention in connection with OCR will be needed here too.

Fourth: re-check the logic when you change the model or the prompt version. Probability distributions differ between models, and a threshold tuned for one may behave differently on another.

For teams building LLM pipelines for document processing, the main takeaway is simple: a confidence score can be obtained almost for free, but only after validation on your own data does it turn from a number into a working routing tool.