Same LLM Benchmark Yielded 28% and 76% — Difference Is Only in How Questions Are Asked

A developer testing Russian-language LLMs on corporate documents received results ranging from 28% to 76% on the same benchmark for the same model — the difference is explained solely by the way the questions are formulated.

How the benchmark is structured

The author assembled a benchmark for Russian-language language models on real types of corporate documents — policies, orders, invoices, and agreements. The task for the models was to find the necessary document among similar ones, cite a specific line in it, and not "break" when a single date in one of the documents is changed.

• Range of results for the same model on one benchmark — from 28% to 76%
• Type of data — corporate documents: policies, orders, invoices, agreements
• Tested skills — finding the right document among similar ones, precise reference to a line, resilience to changing a single detail (date)
• Reason for variation — not the model itself, but how requests to it are organized

Where did the 48 percentage point spread come from

The key finding of the experiment is that the final result turned out to be highly dependent on how the requests to the model were organized, rather than which model was used. In other words, the same model can be "failed" on the benchmark or made to pass it almost three times better — simply by changing how the question and document context are presented to it.

Why this matters for enterprise LLM applications

Companies increasingly build internal document search and processing systems on top of LLMs — what the industry calls RAG (retrieval-augmented generation): agreements, orders, and invoices are rarely stored in a format convenient for the model, and in practice the model must select the necessary document from many similar ones, then cite the exact line. This is exactly what the author's benchmark tested.

If the variation in results is explained by the request format rather than model choice, then for teams implementing LLMs in internal processes, investments in proper organization of queries and context may yield greater effect than switching to a larger or more expensive model. This is especially noticeable in resilience to "small" changes: when a single date in one of the documents is changed, and the model either continues to cite the old version or loses the necessary document among similar ones — such failures are usually what undermine trust in LLM systems within companies, even if the model performs reasonably well on average.

What this means

A benchmark on real corporate documents shows: comparing models "head-on" without fixing the method of presenting questions is methodologically unreliable, and for practical implementation of LLMs in document workflow, it is more important not to chase a more powerful model, but to perfect how it is asked questions.

Frequently Asked Questions

What exactly did the benchmark test?

The models were tested on the ability to find the right document among documents similar to each other — policies, orders, invoices, and agreements — cite a specific line and not lose accuracy if a single date in one of the documents was changed.

Why did the same benchmark produce such different results?

Because the result turned out to be sensitive not to the model itself, but to how the requests to it were organized — the author of the experiment directly links the spread from 28% to 76% specifically to how questions were presented, rather than to changing the model.