Ai2: A Few Words in a Prompt Drop GPT-5.4's Accuracy at Judging Idea Novelty from 93.5% to 40.9%
If one neural network judges the novelty of another's ideas, the outcome depends on how the question is phrased. The Allen Institute for AI (Ai2) showed GPT-5.4 pairs of "human idea vs. AI idea." When the prompt said exactly one idea was novel, the model guessed right 93.5% of the time. When asked simply which idea was novel, accuracy fell to 40.9%, worse than a coin flip. Before using an AI judge, test it across several prompt variants.
AI-processed from Ai2 (Allen Institute); edited by Hamidun News
Imagine a system where one neural network generates scientific hypotheses and another decides which of them is truly new and which merely restates what is already known. This second model is called an "LLM novelty judge." The Allen Institute for AI (Ai2) tested whether it can be trusted and got an alarming result.
GPT-5.4 was asked to pick the newer idea from a pair of "human idea and AI idea." When the prompt indicated that exactly one of the two ideas had been judged novel, the model guessed right in 93.
5% of cases. When it was simply asked which idea was novel, accuracy dropped to 40.9%, which is below random guessing.
Only a few words in the query changed; the ideas themselves stayed the same.
This matters to anyone who uses LLMs as a filter or ranker: selecting ideas for research agents, comparing hypothesis generators, or evaluating product drafts. If the verdict depends on how the prompt is worded, the numbers on your dashboard may reflect a lucky query rather than the quality of the ideas.
How the Judges Were Tested
The main difficulty is that there is no reference standard for novelty. The researchers took reviews from the ICLR 2026 conference and looked not at whether a paper was accepted, but at places where reviewers explicitly discussed originality. A paper can be accepted for strong experiments even when its idea is derivative, and the reverse can also happen.
This produced two sets. The first contained 154 ideas from the top 10% of accepted papers, where reviewers praised novelty and no one disputed it. The second contained 145 ideas from the bottom 10% of rejected papers, where reviewers doubted originality and offered no praise. An annotator agreed with the extracted rating in 47 of 50 checked cases.
For the second scenario, the researchers added 154 ideas from a simple generator to the same 154 strong human ideas. The generator was given only a research area and asked to propose a new idea, with no literature search and no iteration. These labels are weak: generated ideas were assumed by default to be less novel. An expert checked 30 pairs, preferred the human idea in every one, and did not name any generated idea as the novel one.
Six models served as judges: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6. They assigned each idea a score and chose the newer idea from each pair.
A Few Words in the Prompt Decide Everything
The result changed most with the wording of the query. For GPT-5.4, the first prompt stated that reviewers had found exactly one of the two ideas novel. The second simply asked which idea was novel. Accuracy fell from 93.5% to 40.9%, and the verdict changed in more than half of the pairs.
On pairs of two human ideas, the same change moved accuracy by only seven points. A judge can look stable on human papers and behave unpredictably on AI-generated ideas, which are exactly the ones it is being deployed to assess. For Claude Opus 4.6, the shift on human pairs was still 21 points.
There is no universally best prompt. Removing the warning "do not count trivial improvements as novel" raised accuracy for some models and lowered it for others. A successful prompt does not transfer between models. When comparing ideas from two generators, such as Opus 4.5 against Sonnet 4.5, between a quarter and a third of the verdicts changed with the prompt. This conclusion does not require knowing which generator is better.
Expensive Does Not Mean Accurate
Another result matters to those building agents. GPT-5.6 Sol with maximum reasoning budget and live arXiv search scored 78 out of 100 on macro-F1, a metric that measures how well a judge identifies both novel and non-novel ideas; higher is better. Claude Opus 4.6 with five pre-retrieved annotations scored 89. The agentic variant cost roughly ten times more per decision.
Specialized systems also lost. A simple prompt to Claude Opus 4.6 with low reasoning budget and majority voting over three runs scored 85.4 at a total cost of $2.59. Idea Novelty Checker, a pipeline with search and literature-based reranking, scored 64.2 for $30.66. Sakana AI's AI Scientist, which iteratively searches Semantic Scholar, scored 77.2 for $82.05. Both of the latter systems ran on Opus 4.6.
What to Do About It
First, any automated novelty assessment, whether for idea selection, generator ranking, or a filter inside an agent system, needs a stability check. Run several prompt wordings and count how many verdicts change. Second, do not pay for a complex pipeline until you have compared it against a simple prompt with voting. Third, validate the judge on the data it will actually evaluate, meaning AI-generated text, not published papers.
There are limitations too. The labels for generated ideas are weak, the generator was deliberately simple, and the benchmark rests on reviews from a single machine learning conference. The authors themselves call novelty assessment an open problem. Ai2 has published the benchmark, data, code, and the pipeline for building new evaluation sets, which can be applied to other reviews and scientific communities.
- Nov 22, 2026Linger Runs an AI Agent on a 16 GB Mac With No Token Fees and Promises a 5x Speedup
- Nov 22, 2026Claude Code removed ads from a TV in 90 minutes and kicked off a gadget-hacking trend
- Nov 22, 2026Agent Ignored AGENTS.md: In a Third of Pushes It Didn't Ask, and Force-Pushed 31 Times
- Nov 22, 2026Axiom Math's AI Writes 43.1 Million Lines of Lean Proof in 24 Days
Key Questions Answered
How the Judges Were Tested?
The main difficulty is that there is no reference standard for novelty. The researchers took reviews from the ICLR 2026 conference and looked not at whether a paper was accepted, but at places where reviewers explicitly discussed originality. A paper can be accepted for strong experiments even when its idea is derivative, and the reverse can also happen. This produced two sets. The first contained 154 ideas from the top 10% of accepted papers, where reviewers praised novelty and no one disputed it.
What are the key details on A Few Words in the Prompt Decide Everything?
The result changed most with the wording of the query. For GPT-5.4, the first prompt stated that reviewers had found exactly one of the two ideas novel. The second simply asked which idea was novel. Accuracy fell from 93.5% to 40.9%, and the verdict changed in more than half of the pairs. On pairs of two human ideas, the same change moved accuracy by only seven points. A judge can look stable on human papers and behave unpredictably on AI-generated ideas, which are exactly the ones it is being deployed to assess.
What are the key details on Expensive Does Not Mean Accurate?
Another result matters to those building agents. GPT-5.6 Sol with maximum reasoning budget and live arXiv search scored 78 out of 100 on macro-F1, a metric that measures how well a judge identifies both novel and non-novel ideas; higher is better. Claude Opus 4.6 with five pre-retrieved annotations scored 89. The agentic variant cost roughly ten times more per decision. Specialized systems also lost.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.