FindStatBench: New Benchmark Shows Where Top AI Models Fail at Code Synthesis

In July 2026, researchers introduced FindStatBench, a benchmark that tests large language models on combinatorial code synthesis: 2,329 tasks, 24 collections and 5.52 million hidden test objects. On this set, the strongest open and closed models differed in accuracy by less than one percentage point.

What FindStatBench Tests

FindStatBench is built on the FindStat mathematical project. It requires the model to write a single Python function, solve, from a mathematical description of the task and at most five public input-output examples. There is no search, no tool calls, no execution feedback, no voting and no reranking: the model gets one attempt. The answer is run in a sandbox on hidden combinatorial objects and checked by exact execution.

The authors tested 11 systems: four closed production models and seven open-weight models served through a single inference provider.

• 2,329 tasks in 24 collections, 5.52 million hidden test objects
• Two types: statistic synthesis (object → integer) and map synthesis (object → object)
• Input: a description and at most 5 input-output examples
• Scoring: exact execution of the Python function in a sandbox
• 11 systems tested: 4 closed + 7 open

Why Examples in the Prompt Hurt

On some tasks, examples in the prompt lower accuracy instead of raising it. According to the paper, models solve several classic bijections perfectly with no examples at all, but fail the same tasks when given five input-output examples. Another share of failures comes down to output-budget mechanics: the model's reasoning uses up the visible answer before it gets to the code itself. Long prompts cause a sharp collapse in accuracy, which the authors call an accuracy cliff.

Where Models Hit a Ceiling

Statistic synthesis is noticeably easier for models than map synthesis: mapping an object to an integer is simpler than mapping an object to another object. The strongest open and closed systems land within one percentage point of each other on instance accuracy, while pooling all systems into an "oracle" and drawing five samples from a mid-strength model yield only limited gains. Some task collections remain at zero.

"Statistic synthesis is much easier than map synthesis... and exact symbolic rule induction remains brittle," the

FindStatBench abstract on arXiv states.

What This Means

Even top LLMs still struggle to infer exact symbolic rules from mathematical descriptions, and neither more examples nor more compute breaks through that wall. FindStatBench gives the industry a hard, executable test of precisely this kind of reasoning, rather than of retelling known solutions.

Frequently Asked Questions

What is FindStatBench?

It is an executable benchmark built on the FindStat project: 2,329 combinatorial code synthesis tasks in which the model writes a Python function, and the answer is checked by exact execution on 5.52 million hidden objects, with no hints, tools or retries allowed.

Why don't more examples always help models?

According to the paper, models solve several classic bijections perfectly with no examples, but break under a prompt with five examples: a long context leads to a sharp drop in accuracy, and the model's reasoning sometimes exhausts the answer budget before any code is output.