FindStatBench: новый бенчмарк выявил, где топовые ИИ-модели проваливают синтез кода

In July 2026, researchers introduced FindStatBench, a benchmark that tests large language models on combinatorial code synthesis: 2329 tasks, 24 collections, and 5.52 million hidden test objects. On this set, the strongest open and closed models differed in accuracy by less than one percentage point.

What FindStatBench Tests

FindStatBench is built on the mathematical FindStat project and requires a model to write a single Python function, solve, based on a mathematical description of the task and at most five public input-output examples. There is no search, no tool calls, no execution feedback, no voting, and no reranking — just a single attempt. The answer is run in a sandbox on hidden combinatorial objects and checked by exact execution.

The authors tested 11 systems: four closed production models and seven open-weight models served through a single inference provider.

• 2329 tasks in 24 collections, 5.52 million hidden test objects
• Two types: statistic synthesis (object → integer) and map synthesis (object → object)
• Input consists of a description and at most 5 input-output examples
• Evaluation is exact execution of the Python function in a sandbox
• 11 systems tested: 4 closed + 7 open

Why Examples in the Prompt Hurt

Examples in the prompt lower accuracy on some tasks rather than raising it. According to the paper, several classic bijections are solved perfectly by models with no examples at all, but the same models fail those tasks once given five input-output examples. Another part of the failures comes down to output-budget mechanics: the model's reasoning exhausts the visible answer before it ever reaches the code itself. Long prompts cause a sharp drop in accuracy — the authors call this the accuracy cliff.

Where Models Hit a Ceiling

Statistic synthesis is noticeably easier for models than map synthesis: mapping an object to an integer is simpler than mapping an object to another object. The strongest open and closed systems converge within one percentage point of each other on instance accuracy, and combining all systems into an "oracle" plus taking five samples from an average-strength model yield only limited gains. Some task collections remain stuck at zero.

"Statistic synthesis is much easier for us than map synthesis... and accurately inducing symbolic rules remains brittle," the

FindStatBench abstract on arXiv states.

What This Means

Even top LLMs are still poor at deriving exact symbolic rules from mathematical descriptions — and neither more examples nor more compute break through this wall. FindStatBench gives the industry a strict, executable test specifically for this type of reasoning, rather than for recalling known solutions.

Frequently Asked Questions

What is FindStatBench?

It is an executable benchmark built on the FindStat project: 2329 combinatorial code synthesis tasks where a model writes a Python function and the answer is checked by exact execution on 5.52 million hidden objects, with no hints, tools, or retries allowed.

Why doesn't more examples always help models?

According to the paper, several classic bijections are solved perfectly by models with zero examples, but break down under a prompt with five examples: the longer context leads to a sharp drop in accuracy, and reasoning sometimes exhausts the answer budget before the code is ever produced.