The same LLM benchmark produced 28% and 76% — and the only difference was how the questions were asked

A developer testing Russian-language LLMs on corporate documents got results ranging from 28% to 76% for the same model on the same benchmark. The entire difference came down to how the questions were delivered.

How the benchmark works

The author built a benchmark for Russian-language language models using real types of corporate documents: policies, orders, invoices and approvals. The models had to find the right document among similar ones, cite a specific line in it, and not "break" when just one date was changed in one of the documents.

• Range of results for the same model on the same benchmark: 28% to 76%
• Data type: corporate documents, including policies, orders, invoices and approvals
• Skills tested: finding the right document among similar ones, citing the exact line, and robustness to a change in a single detail (a date)
• Cause of the spread: not the model itself, but how the queries to it were organized

Where the 48-percentage-point spread came from

The key finding of the experiment is that the final score depended heavily on how the queries to the model were organized, not on which model was used. In other words, the same model can either "fail" the benchmark or pass it nearly three times better, simply by changing how the question and the document context are presented to it.

Why this matters for enterprise LLM use

Companies are increasingly building internal search and document-handling systems on top of LLMs, an approach the industry calls RAG (retrieval-augmented generation). Approvals, orders and invoices are rarely stored in a model-friendly form, so in practice the model has to pick the right document out of many similar ones and then cite the exact line. That is precisely what the author's benchmark tested.

If a multifold spread in results comes from the query format rather than the choice of model, then teams rolling LLMs into internal workflows may get more out of investing in well-organized queries and context than out of switching to a larger or more expensive model. This shows up most clearly in robustness to "small" changes. When a single date changes in one document, the model either keeps citing the old version or loses the right document among similar ones. These are exactly the failures that most often erode trust in LLM systems inside companies, even when the model answers reasonably well on average.

What this means

A benchmark built on real corporate documents shows that comparing models head-to-head without fixing how the questions are delivered is methodologically unreliable. For practical LLM deployment in document workflows, the priority is not chasing a more powerful model but getting the way you ask it questions right.

FAQ

What exactly did the benchmark test?

The models were tested on their ability to find the right document among documents that resemble one another (policies, orders, invoices and approvals), cite a specific line, and keep their accuracy when just one date was changed in one of the documents.

Why did the same benchmark produce such different results?

Because the result proved sensitive not to the model itself but to how the queries to it were organized. The author of the experiment explicitly attributes the spread from 28% to 76% to the way the questions were delivered, not to swapping the model.