OpenProblemBench: GPT-6-Astra Solves 14% of Open Math and Physics Problems

The OpenProblemBench benchmark tests whether language models can make progress on problems to which science does not yet know the answer. It contains 82 unsolved problems from the literature on mathematics and theoretical physics. GPT-6-Astra posted the best result: the average share of problems that the judges considered solved was 14.0%. Full-size open models scored 5.5–6.7%, and the lightweight Flash models 2.4–3.7%. Seven configurations were compared in total.

How the test works

Most popular benchmarks rely on known answers: a model either guesses the key or it doesn't. Here there is no key by definition, because the problems are open. For each problem, the authors supply the research context, the assumptions and the progress already made, which is everything a human would need to start working on the question.

To make the results checkable at all, the authors selected problems whose decisive mathematical or computational claims lend themselves to relatively clear verification. There are still no reference solutions. Four judge models independently grade each answer on three criteria: correctness, completeness and degree of progress.

Key parameters:

• 82 unsolved problems from the literature on mathematics and theoretical physics;
• 4 independent judge models working without reference solutions;
• 7 configurations tested;
• GPT-6-Astra: 14.0% average judged solve rate;
• full-size open models: 5.5–6.7%;
• Flash models: 2.4–3.7%.

The gap is clear. The leader scores roughly twice as much as the best open models and several times as much as the Flash class.

What sets strong answers apart

The authors examined individual cases and tied the successful results to three traits.

First, a change in how the problem is represented. A strong model recasts the question in a form that is easier to attack, instead of battering away at the original formulation.

Second, general arguments that go beyond a finite set of evidence. Checking a handful of cases or running a numerical experiment looks convincing but is not a proof. An answer that builds an argument for the whole class of objects is fundamentally more valuable.

Third, proof of the steps needed to complete the solution. Weak answers often stop at the idea and leave the key lemma unjustified. The judges rate such work low on completeness.

How to read these numbers

There are caveats. The metric is called "judged solve rate": it is an assessment by judge models, not a confirmed solution. The abstract says nothing about professional mathematicians checking each answer. In addition, the set is skewed toward problems with verifiable claims, so the 14% cannot be extrapolated to open science as a whole. It is a figure for a deliberately selected slice.

The practical takeaways for teams choosing models for research work are as follows. First, the gap between the flagship and the open full-size models is substantial here, so for proof-heavy tasks, saving money on model class means losing quality. Flash models are best kept for draft work: literature search, reformulations and checking individual derivations.

Second, if you are building a pipeline around an AI researcher, build in independent verification. A judge model without a reference can be wrong, and a formal check or a human expert remains the final filter. Third, the three traits of strong answers suggest how to write prompts: ask the model to change the formulation, demand a general argument instead of examples, and ask for a separate proof of each step. OpenProblemBench is valuable as a test bench that shows where models are already useful to co-authors and where they remain weak.