How to Check LLMs by Hand and by Script: 12 Lessons from Translations and Size Charts

When an LLM is built into a live business process, the question is no longer whether the model can translate or look up data. The question is what happens when it makes a mistake. The authors of a write-up on Habr collected 12 takeaways from two production tasks: automatic translation of products from English to Russian and searching for size charts.

Two tasks, two strategies

The first task: tens of thousands of titles and descriptions per day. An error here is cheap, the original and photos are right there, and that volume cannot be checked by hand. So the setup combines several models, an LLM judge and script checks, plus a spot audit.

The second task: find a size chart from the combination of "brand + product type + size + gender", for example Nike, sneakers, 11 US, men's. An error is expensive: a 260 mm insole instead of 250 means a return. The volume is moderate, so a human checks every record. Two models with web search (OpenAI and Gemini) look up the chart independently, Claude acts as the judge, and a human does the final check.

Hence a simple matrix: assess the cost of an error and the daily volume, and the verification strategy becomes obvious.

What actually helps

Context matters more than clever prompts. At first only the title was translated; quality improved after adding the description, site, category and link. At Walmart, the single-letter attribute "N" (short for "No") was translated as "North" until the attribute name was added. At DoorDash, many "judge errors" turned out to be a lack of information. If the model got it wrong, first check whether it has enough data.

Put the justification before the answer. The model outputs a rule from the prompt or a verbatim fragment of the page, and only then the value. There are fewer fabrications, and it is easier for the reviewer to find the source.

The judge must answer "yes" or "no". The team dropped the five-point scale: they found no difference between scores of 3 and 5. The model's self-assessed confidence is also useless: an answer rated "10 out of 10" turned out to be a value from the women's chart instead of the men's. Agreement between two models is not enough either, since they often make the same mistake, so both signals are needed: agreement and judge approval.

Scripts beat LLMs where the task is deterministic. The model periodically fails to convert, or miscalculates, sizes in inches, pounds and ounces. Regular expressions compare the numbers in the original and the translation, and where possible the conversion is done with coefficients, without a model.

Tools and process

For manual review, the team built a Chrome extension. It takes a JSON with the results, opens the source page, highlights the found value (or launches a Google search if there is no chart), and the decision is made with hotkeys. The output JSON goes to the admin panel. Review speed increased tenfold or more.

A few more practices: record the model and prompt version for each result, run a reference set whenever the combination changes, involve category managers early (they found inaccuracies in the translation of cosmetics and medicines), and manually check a sample once a month. What must be tested is the "model + prompt" pair, because the same prompt behaves differently on different models.

Limitations and conclusion

This is the experience of two specific tasks, the article gives no accuracy figures, and conclusions about niche cases such as size charts do not necessarily carry over to other domains. But the logic is universal: LLMs can be trusted with processes where the cost of error is low or medium, while critical ones must be checked by hand until there is a history of error-free operation spanning several months. Errors and complaints should be turned into prompt fixes: error, analysis, fix, fewer such errors. This is, in essence, how Walmart works too: linguists do not edit translations by hand, they formulate a rule that is applied to the entire catalog.