Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)

XYBench test: AI picks the useful advice in 71–84% of cases, but gives it itself only 35–41% of the time

Ask an AI how to thicken frosting for cupcakes, and it will suggest powdered sugar, even though the frosting slides off because the cupcakes are still hot. This is the XY problem: a person asks for help with a chosen method (Y) rather than the real goal (X). On XYBench, built from Stack Overflow questions, models pick the better of two answers 71–84% of the time, but their own answers address the user's actual goal only 35–41% of the time. That is why an assistant should be trained to check the premise of a question first.

AI-processed from Хабр: Машинное обучение; edited by Hamidun News
XYBench test: AI picks the useful advice in 71–84% of cases, but gives it itself only 35–41% of the time
Source: Хабр: Машинное обучение. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceХабр: Машинное обучение ↗
Editor-in-Chief
TestedClaude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, DeepSeek-V4 Pro
Key metric #2The benchmark has 8,115 questions drawn from code, tech support and everyday tasks
Examplesomeone asks for a regex to parse XML, when a ready-made parser is more reliable

You ask a neural network how to make frosting for cupcakes thicker, because it keeps sliding off. The assistant suggests adding powdered sugar, changing the proportions, or whipping it longer, and all the advice sounds reasonable. But if the cupcakes are still warm, the frosting will slide off in any proportions: the baked goods have to cool down first.

This situation is called the XY problem: a person wants to achieve goal X, decides that method Y is needed for it, and asks for help only with Y. Researchers measured how modern models handle this and found a gap. On Stack Overflow questions, AI picks the more useful of two ready-made answers in 71–84% of cases, but when it writes an answer itself, it addresses the user's real goal in only 35–41% of responses.

The test is called XYBench.

What the XY problem is, with simple examples

The term has long been familiar to participants in technical forums. The question mentions only the chosen method, while the real goal stays out of view.

An example from programming: a person wants to extract data from an XML document and asks how to do it with regular expressions. A model can compose a complex expression, explain the syntax and list the limitations. Formally, the request has been fulfilled. But for structured XML, a specialized parser is usually more reliable: it accounts for element nesting, attributes and the quirks of the format. If regular expressions are not mandatory, a useful answer first names their weak points and suggests a different tool.

Outside of code, it is the same. A person buys a more powerful vacuum cleaner, when the weakened suction is caused by a clogged hose. Or they reinstall the operating system without finding the cause of the application crashes. A good assistant doesn't just issue instructions; it checks whether the assumption about the path to the result is correct.

What XYBench showed

On November 13, 2026, researchers from the Allen Institute for AI (Ai2), together with colleagues from Carnegie Mellon University, NVIDIA and Johns Hopkins University, presented the XYBench benchmark. The team was led by Akhila Yerukola. The set includes 8,115 questions from programming, technical support and everyday tasks. It covers technical topics such as electronics and Linux, household situations (repairs, cooking, handicrafts) and ordinary practical instructions.

The authors tested Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro. The main result is the gap between "understanding" and "doing." When models were shown two ready-made answers to a Stack Overflow question, they chose the more useful one in 71–84% of cases. So they can tell a good approach from a bad one. But when generating answers themselves, they addressed the user's actual goal in only 35–41% of responses. Even after finding the right solution, models often kept explaining at length the originally chosen, unsuitable method.

Assembling such a set was not easy. Not every strange request is an XY problem: sometimes the chosen method is justified by circumstances, for example, regular expressions are required by the project's terms. So the authors tried to filter out questions where a literal execution of the instruction is sufficient.

From a linguistic standpoint, this is a question of pragmatics: how context and the speaker's intentions change the meaning of a phrase. For language models it is especially hard, because the goal is rarely stated separately; it is hidden inside the chosen method.

What this means for developers and product managers

If you are building an assistant, tech support or a coding helper, the conclusion is practical. The ability to choose the best option from ready-made ones does not guarantee a good free-form answer. If you test a model only with multiple-choice tests, the problem can be underestimated.

What is worth trying:

  • add to the system prompt a requirement to check the premise of the question first, and only then answer;
  • in debugging and support scenarios, allow the model to clarify what the person actually wants to achieve;
  • include in your own tests examples where a literal execution of the request does not solve the task.

There are also limitations. The available description contains no breakdown of results by model, so it is impossible to say which of the five handles this best. It is also unknown how closely the benchmark score matches the opinions of real users. Moreover, excessive insistence is also irritating: if a person really needs regular expressions, the assistant should write them rather than deliver a lecture.

XYBench gives engineers a way to measure this weakness instead of merely guessing at it.

Story

Key Questions Answered

What the XY problem is, with simple examples?

The term has long been familiar to participants in technical forums. The question mentions only the chosen method, while the real goal stays out of view. An example from programming: a person wants to extract data from an XML document and asks how to do it with regular expressions. A model can compose a complex expression, explain the syntax and list the limitations. Formally, the request has been fulfilled.

What XYBench showed?

On November 13, 2026, researchers from the Allen Institute for AI (Ai2), together with colleagues from Carnegie Mellon University, NVIDIA and Johns Hopkins University, presented the XYBench benchmark. The team was led by Akhila Yerukola. The set includes 8,115 questions from programming, technical support and everyday tasks. It covers technical topics such as electronics and Linux, household situations (repairs, cooking, handicrafts) and ordinary practical instructions.

What this means for developers and product managers?

If you are building an assistant, tech support or a coding helper, the conclusion is practical. The ability to choose the best option from ready-made ones does not guarantee a good free-form answer. If you test a model only with multiple-choice tests, the problem can be underestimated.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…