Ai2 shows LLM novelty judges break on a few words in the prompt

Ai2 has published the study Old Ideas, Novel Problems, along with a benchmark, data, code and an automated pipeline for building new evaluation sets. The takeaway is uncomfortable for anyone building automated science: LLM novelty judges are extremely unstable. A single wording change in the prompt shifts GPT-5.4's accuracy by 52.6 points, from 93.5% to 40.9%. Expensive specialized systems also turned out to be worse than a simple prompt to Claude Opus 4.6.

How the benchmark was built from ICLR 2026 reviews

Idea generators such as DataVoyager, AutoDiscovery, Theorizer and the Asta platform from the Ai2 ecosystem produce hypotheses faster than people can read them. Teams often assign the filtering to another model, a "novelty judge." There is no standard recipe for such a judge: each team picks its own model, prompt, comparison format, reasoning budget and, sometimes, literature search. Judges are usually validated on human-written papers, not on the AI ideas they will later have to assess.

To get reference labels, the authors did not treat paper acceptance as a sign of novelty. Instead, they extracted passages from ICLR 2026 reviews where reviewers explicitly discuss originality. This produced two pools:

• 154 high-novelty ideas: accepted papers from the top 10% of their field, where novelty was noted positively and nobody disputed it;
• 145 low-novelty ideas: rejected papers from the bottom 10%, where reviewers doubted the originality and nobody saw novelty.

Label check: an annotator agreed with the polarity of the signal in 47 of 50 cases, and none of the disagreements would have changed the reading of the reviewer's comment. The first scenario, Human-Only, compares human ideas. In the second, Human+Generated, the same 154 human ideas are paired with 154 ideas from a simple LLM generator. The generator received only the research area and a request to propose a new idea, with no search, tools or iteration. The labels for generated ideas are weak, but an expert reviewed 30 pairs, preferred the human idea in every one, and did not rate any generated idea as novel.

One phrase in the prompt: minus 52 points

There were six judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. The authors varied one factor at a time: the prompt, reasoning budget, access to related papers, idea format, the number of aggregated verdicts and the generator model. The prompt had the strongest effect.

GPT-5.4 was given Human+Generated pairs with two prompts. In the first, it was told that reviewers had judged exactly one idea in the pair to be novel, and it was asked to identify it. In the second, it was simply asked which idea was novel. Accuracy fell from 93.5% to 40.9%, and the verdict changed in more than half of the pairs. On Human-Only, the same change cost about seven points, although Claude Opus 4.6 lost 21 points there. A judge can look stable on human work and behave differently on AI ideas.

There is no universal prompt. If you remove the instruction not to count trivial and incremental contributions as novel, accuracy rises for some judges and falls for others. When comparing ideas from two generators (Opus 4.5 vs. Sonnet 4.5, GPT-5.4 vs. GPT-5.1), between a quarter and a third of verdicts change. This can be measured without assuming which generator is stronger.

Expensive systems lost to a cheap prompt

Another unexpected finding concerns search and reasoning. An agentic judge on GPT-5.6 Sol, with the maximum reasoning budget and iterative arXiv search, scored 0.78 macro-F1. Claude Opus 4.6 with five pre-retrieved abstracts scored 0.89. The agent cost roughly 10 times more per decision.

Specialized systems did not help either (all on Claude Opus 4.6, Human+Generated):

• simple Opus 4.6 prompt, low reasoning level, majority voting over three calls: 0.854 macro-F1, $2.59 for the whole run;
• Idea Novelty Checker (literature search with reranking): 0.642 macro-F1, $30.66;
• AI Scientist from Sakana AI (iterative Semantic Scholar search): 0.772 macro-F1, $82.05.

The simple baseline came out 12–32 times cheaper and more accurate than both.

What to check in your own evaluation pipeline

If your system ranks or filters ideas, hypotheses or research tasks using an LLM, you cannot trust the verdict of a single prompt. Run the judge on at least two or three phrasings and calculate the share of verdicts that change; this disagreement can be measured without any labels. Check it on generated ideas specifically, not just on human papers. Do not default to expensive agentic search: start with a simple prompt and three-call voting, and only then pay 10–30 times more for a measurable gain. Ai2's code and data are open, and the pipeline can be reused on reviews from other conferences and fields.