Ai2 tested five top models: they focus on the real problem in only 35–41% of answers
You ask how to thicken icing that slides off your cupcakes, but the real cause is that the cupcakes are still warm. Ai2 built XYBench, a test of 8,115 such questions, and checked five models, including GPT-5 and Claude Sonnet 4.6. On Stack Overflow questions, they put the real cause at the center of their answers in only 35–41% of cases, while focusing on the flawed approach from the question in 47–57%. So when choosing an assistant, check whether it can push back on the user.
AI-processed from Ai2 (Allen Institute); edited by Hamidun News
You ask an assistant how to thicken icing that slides off your cupcakes. A good helper will notice the problem isn't thickness: the cupcakes are still warm, so the icing just melts. A bad one will start explaining powdered sugar and cornstarch in detail.
Ai2's lab released the XYBench benchmark to measure exactly this skill and tested five strong models on it. On Stack Overflow questions (developers asking about code), the models put the user's real problem at the center of their answer in only 35–41% of cases. In 47–57% of cases, they focused on the flawed approach the person asked about.
The work was done with Carnegie Mellon, NVIDIA and Johns Hopkins and received the Best Resource Paper award at EMNLP 2026.
What the XY problem is and how it was tested
The situation will be familiar to anyone who has worked in support or answered questions on forums. A person wants to achieve goal X, decides that method Y will help, and asks for help with Y. In the benchmark, Y is called the stated request and X is the real goal, which can be inferred from context.
XYBench contains 8,115 questions from four sources:
- 1,557 questions from Stack Overflow;
- 184 questions from Stack Exchange;
- 102 hand-picked examples;
- 6,272 questions generated from WikiHow guides.
For forum questions, the goal and possible solutions were identified from the three highest-rated answers and the accepted answer. Cases where the literal question could be answered as asked were filtered out. WikiHow questions were generated by Sonnet 4.5. They were then refined so that an expert could understand the goal, while the goal itself was not named in the text.
Answers are scored on three criteria. First, whether the answer contains a solution for Y and for X. Second, what the emphasis is on: the stated request, the goal, or both equally. Third, whether the model explains that the chosen path is flawed and points toward the goal. Claude Sonnet 4.5 serves as the judge. Two people independently checked 500 answers. They agreed with the judge in 97.5–100% of cases on whether a solution was present, in 84.6–93.3% on identifying the flawed assumption, and in 74.2–79.8% on emphasis. The last figure is noticeably lower than the others: emphasis is the hardest thing to assess.
What the models showed
Ai2 tested Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro. Each model answered each question once. All five more often offered a solution for the stated request than for the real goal. On Stack Overflow, a solution for Y appeared in 75–86% of answers, and a solution for X in 59–71%.
Mentioning the right solution is not the same as making it the main point. A model may spend several paragraphs recommending a more powerful vacuum and add one line saying it's worth checking the hose. On Stack Overflow, the real goal was the focus in only 35–41% of answers, while the stated request was the focus in 47–57%. In the remaining answers, the emphasis was split roughly evenly. Human experts did better in this comparison: they more often steered the user away from the flawed path.
Knowing and doing are different skills
In two additional experiments, the authors tested why this happens.
In the first, the models were explicitly told both the stated request and the real goal. On Stack Overflow, Sonnet 4.6's share of answers with a solution for the goal rose from 59% to 67%. But the share of answers where the model explains that the assumption is flawed dropped from 47% to 34%. All models still lagged behind humans at identifying the flawed assumption. Guessing the intent is only half the job. Models still have to be willing to push back on the user.
In the second experiment, the models acted as judges. Six models (the five main ones plus GPT-4.1) were given two ready-made answers: one about the literal request, the other about the real goal. They chose the answer about the goal in 71–84% of cases. So they are noticeably better at recognizing which advice is more useful than at writing such advice themselves. A multiple-choice test would not have revealed this difference.
What to do about it
For assistant developers and product teams, the takeaway is practical. If users arrive with imprecise questions, which is almost always the case, scoring answers only on formal correctness is not enough. It's worth explicitly checking whether the model looks for the root problem. Possible approaches include a system prompt asking the model to clarify the goal first, and evaluating answers on the same three criteria. The idea of choosing the best of two answers suggests another option: have the model first generate several drafts, then pick the one that solves the real task. The authors did not test this, so it remains a hypothesis.
There are also honest limitations. The benchmark is so far English-only and mostly about procedural tasks: programming, cooking, home repair. The judge is also a model, and its agreement with humans on emphasis is no higher than 80%. Ai2 only proposes studying how training and prompting affect the results in future work.
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
- Nov 21, 2026Anthropic cuts Claude Fable 5.2 price by 20%, and on the coding test it nearly matches Opus 5.5
- Nov 21, 2026Dreamteam Raises $53M for a CRM That Fills Itself from Email and Calls
- Nov 21, 2026Claude Opus 5.5 Builds Out Dimension 9 Horror Game From the Author's 15 Rooms
Key Questions Answered
What the XY problem is and how it was tested?
The situation will be familiar to anyone who has worked in support or answered questions on forums. A person wants to achieve goal X, decides that method Y will help, and asks for help with Y. In the benchmark, Y is called the stated request and X is the real goal, which can be inferred from context. XYBench contains 8,115 questions from four sources: 1,557 questions from Stack Overflow; 184 questions from Stack Exchange; 102 hand-picked examples; 6,272 questions generated from WikiHow guides.
What the models showed?
Ai2 tested Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro. Each model answered each question once. All five more often offered a solution for the stated request than for the real goal. On Stack Overflow, a solution for Y appeared in 75–86% of answers, and a solution for X in 59–71%. Mentioning the right solution is not the same as making it the main point. A model may spend several paragraphs recommending a more powerful vacuum and add one line saying it's worth checking the hose.
What are the key details on Knowing and doing are different skills?
In two additional experiments, the authors tested why this happens. In the first, the models were explicitly told both the stated request and the real goal. On Stack Overflow, Sonnet 4.6's share of answers with a solution for the goal rose from 59% to 67%. But the share of answers where the model explains that the assumption is flawed dropped from 47% to 34%. All models still lagged behind humans at identifying the flawed assumption. Guessing the intent is only half the job.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.