Pollux from Sber AI: An LLM Judge for Evaluating Russian-Language Models

Sber AI has unveiled Pollux, a judge model for the automatic evaluation of Russian-language models. The tool tackles a problem developers have faced for years: how to check LLM quality reliably and quickly before deploying to commercial production.

From manual checks to automation

A few years ago, when language models had only just begun to produce sensible answers, quality evaluation was purely a matter of time and money. People manually reviewed every model response, flagged errors, and assessed instruction-following and factual correctness. The process was slow: reviewing hundreds of responses took days or weeks.

Today LLMs handle serious tasks: they write working code, hold conversations with customers, and plan delivery routes. But a model still has to be evaluated before it goes into a real product. Manual review has become a bottleneck in development. Companies lose time while experts check responses by hand.

Pollux: a solution for Russian

Pollux addresses this problem. It is a specialized language model trained on Russian and on the task of evaluating other LLMs. It can run in your development pipeline and check response quality automatically.

The model is openly available: developers pay no license fees and sign no contracts. You simply download it, embed it in your code, and use it.

How the judge model works

Pollux checks language model responses against several criteria: accuracy of information, completeness of the answer, match with the required style, adherence to the original instructions, and appropriateness for the context.

It works millions of times faster than a human, producing an evaluation in seconds instead of hours of manual work. It scales: you can check as many as a thousand responses at once. It is also cheaper. Where you once had to pay an expert for each response checked, the model now does it for free.

One reason Sber released the tool openly is to give the whole ecosystem a standard way of evaluating models.

The model is trained on Russian, and that matters because evaluation criteria are often language-specific. Russian is a more flexible language, its grammar is more complex, and style depends on context. Evaluation accuracy in Russian is higher than if you used a model trained on English.

Standardizing the industry

Until now, every company had its own criteria for evaluating LLMs, often improvised and incomplete. One developer checks against five criteria, another against fifteen. The results cannot be compared with each other.

Pollux creates a single standard: a shared tool that everyone can apply to their models. This should make it easier to compare LLMs with one another and reduce risks before launching into production.

This is especially significant for the Russian-speaking AI community, since most evaluation tools are geared toward English and English-language context. With Pollux, Russian-speaking developers get a tool adapted to their own realities.

What this means

Automatic LLM evaluation is becoming a development standard rather than an expensive luxury. Developers will be able to iterate faster and experiment with architecture and data without waiting in line for experts. The development cycle will speed up several times over.

For users, this means better and more reliable AI services, because models are better tested before release into production. The Russian-speaking developer community will finally get a tool adapted to the specifics of its native language.