Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseInsight · On RadarAi2 (Allen Institute)→ original

Ai2 tested five top models: they focus on the real problem in only 35–41% of answers

You ask how to thicken icing that slides off your cupcakes, but the real cause is that the cupcakes are still warm. Ai2 built XYBench, a test of 8,115 such questions, and checked five models, including GPT-5 and Claude Sonnet 4.6. On Stack Overflow questions, they put the real cause at the center of their answers in only 35–41% of cases, while focusing on the flawed approach from the question in 47–57%. So when choosing an assistant, check whether it can push back on the user.

AI-processed from Ai2 (Allen Institute); edited by Hamidun News
Ai2 tested five top models: they focus on the real problem in only 35–41% of answers
Source: Ai2 (Allen Institute). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAi2 (Allen Institute) ↗
Editor-in-Chief
8,115 questionsStack Overflow, Stack Exchange, WikiHow and hand-written examples
Key metric #2Given two ready-made answers, the models pick the right one in 71–84% of cases
Key metric #3Adding the goal to the prompt raised Sonnet 4.6's rate of solving the real problem from 59% to 67%

You ask an assistant how to thicken icing that slides off your cupcakes. A good helper will notice the problem isn't thickness: the cupcakes are still warm, so the icing just melts. A bad one will start explaining powdered sugar and cornstarch in detail.

Ai2's lab released the XYBench benchmark to measure exactly this skill and tested five strong models on it. On Stack Overflow questions (developers asking about code), the models put the user's real problem at the center of their answer in only 35–41% of cases. In 47–57% of cases, they focused on the flawed approach the person asked about.

The work was done with Carnegie Mellon, NVIDIA and Johns Hopkins and received the Best Resource Paper award at EMNLP 2026.

What the XY problem is and how it was tested

The situation will be familiar to anyone who has worked in support or answered questions on forums. A person wants to achieve goal X, decides that method Y will help, and asks for help with Y. In the benchmark, Y is called the stated request and X is the real goal, which can be inferred from context.

XYBench contains 8,115 questions from four sources:

  • 1,557 questions from Stack Overflow;
  • 184 questions from Stack Exchange;
  • 102 hand-picked examples;
  • 6,272 questions generated from WikiHow guides.

For forum questions, the goal and possible solutions were identified from the three highest-rated answers and the accepted answer. Cases where the literal question could be answered as asked were filtered out. WikiHow questions were generated by Sonnet 4.5. They were then refined so that an expert could understand the goal, while the goal itself was not named in the text.

Answers are scored on three criteria. First, whether the answer contains a solution for Y and for X. Second, what the emphasis is on: the stated request, the goal, or both equally. Third, whether the model explains that the chosen path is flawed and points toward the goal. Claude Sonnet 4.5 serves as the judge. Two people independently checked 500 answers. They agreed with the judge in 97.5–100% of cases on whether a solution was present, in 84.6–93.3% on identifying the flawed assumption, and in 74.2–79.8% on emphasis. The last figure is noticeably lower than the others: emphasis is the hardest thing to assess.

What the models showed

Ai2 tested Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro. Each model answered each question once. All five more often offered a solution for the stated request than for the real goal. On Stack Overflow, a solution for Y appeared in 75–86% of answers, and a solution for X in 59–71%.

Mentioning the right solution is not the same as making it the main point. A model may spend several paragraphs recommending a more powerful vacuum and add one line saying it's worth checking the hose. On Stack Overflow, the real goal was the focus in only 35–41% of answers, while the stated request was the focus in 47–57%. In the remaining answers, the emphasis was split roughly evenly. Human experts did better in this comparison: they more often steered the user away from the flawed path.

Knowing and doing are different skills

In two additional experiments, the authors tested why this happens.

In the first, the models were explicitly told both the stated request and the real goal. On Stack Overflow, Sonnet 4.6's share of answers with a solution for the goal rose from 59% to 67%. But the share of answers where the model explains that the assumption is flawed dropped from 47% to 34%. All models still lagged behind humans at identifying the flawed assumption. Guessing the intent is only half the job. Models still have to be willing to push back on the user.

In the second experiment, the models acted as judges. Six models (the five main ones plus GPT-4.1) were given two ready-made answers: one about the literal request, the other about the real goal. They chose the answer about the goal in 71–84% of cases. So they are noticeably better at recognizing which advice is more useful than at writing such advice themselves. A multiple-choice test would not have revealed this difference.

What to do about it

For assistant developers and product teams, the takeaway is practical. If users arrive with imprecise questions, which is almost always the case, scoring answers only on formal correctness is not enough. It's worth explicitly checking whether the model looks for the root problem. Possible approaches include a system prompt asking the model to clarify the goal first, and evaluating answers on the same three criteria. The idea of choosing the best of two answers suggests another option: have the model first generate several drafts, then pick the one that solves the real task. The authors did not test this, so it remains a hypothesis.

There are also honest limitations. The benchmark is so far English-only and mostly about procedural tasks: programming, cooking, home repair. The judge is also a model, and its agreement with humans on emphasis is no higher than 80%. Ai2 only proposes studying how training and prompting affect the results in future work.

Story

Key Questions Answered

What the XY problem is and how it was tested?

The situation will be familiar to anyone who has worked in support or answered questions on forums. A person wants to achieve goal X, decides that method Y will help, and asks for help with Y. In the benchmark, Y is called the stated request and X is the real goal, which can be inferred from context. XYBench contains 8,115 questions from four sources: 1,557 questions from Stack Overflow; 184 questions from Stack Exchange; 102 hand-picked examples; 6,272 questions generated from WikiHow guides.

What the models showed?

Ai2 tested Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro. Each model answered each question once. All five more often offered a solution for the stated request than for the real goal. On Stack Overflow, a solution for Y appeared in 75–86% of answers, and a solution for X in 59–71%. Mentioning the right solution is not the same as making it the main point. A model may spend several paragraphs recommending a more powerful vacuum and add one line saying it's worth checking the hose.

What are the key details on Knowing and doing are different skills?

In two additional experiments, the authors tested why this happens. In the first, the models were explicitly told both the stated request and the real goal. On Stack Overflow, Sonnet 4.6's share of answers with a solution for the goal rose from 59% to 67%. But the share of answers where the model explains that the assumption is flawed dropped from 47% to 34%. All models still lagged behind humans at identifying the flawed assumption. Guessing the intent is only half the job.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…