CLEAT Teaches Qwen3-4B Agents to Ask for Clarification at the Right Time: Up to +13.2 pp

Researchers have published CLEAT on arXiv, a framework that teaches an LLM agent to decide when to ask the user a question and when it is better to act on its own. On Qwen3-4B, the method achieved the highest average score in all ten test configurations across four benchmarks. Over the strongest baseline, the gain reached 13.2 percentage points on UserGym Travel and 6.5 pp in accuracy at detecting underspecified requests on IN3.

The problem: a question is not always useful

Agents are taking on increasingly complex tasks, but users often leave their goals and constraints unstated. A person asks an agent to plan a trip and never mentions a budget or unwanted layovers. The agent can guess and get it wrong. It can also flood the user with questions and ruin the experience. The authors frame the task as interactive intent alignment through clarification.

There is a subtlety here. Clarification makes sense only if the agent can turn the answer into a better solution. As training progresses, the way the model handles incomplete information and uses new answers changes, so the same question can be useful at one moment and useless at another. Static rules such as "ask if parameters are missing" do not track this dynamic.

How CLEAT works

The name stands for CLarification through valuE-guided Agent co-Training. The framework jointly trains two components: a clarification value controller and the policy that solves the task. Both learn from shared interaction outcomes.

The central concept of the work is residual query value. It measures how much a question helps the task compared with continuing under the current policy. The authors estimate it with paired rollouts from the same dialogue history: in one, the agent branches into asking a question; in the other, it acts immediately. The difference in returns is the value label.

These paired returns serve two purposes. The controller learns to predict the value of clarification. The policy learns to pick useful questions and rely on the answers. The loop then closes: the controller guides the collection of the next rollouts, and the updated agent's results produce fresh labels. Clarification prompts adapt to the model's growing capabilities.

What the experiments showed

Main results in version v1 of the preprint:

• base model: Qwen3-4B;
• highest average score in all 10 evaluation configurations across 4 benchmarks;
• up to +13.2 pp over the strongest baseline on UserGym Travel;
• +6.5 pp in accuracy of judging underspecification on IN3;
• ablations: joint training improves the policy even if the controller is switched off at inference;
• adapting the prompts to the policy's evolution gives an additional gain.

The last point matters most for practice. It means the controller works not only as a filter at inference time but also as a teacher: it changes the way the agent itself handles incomplete data.

Where this is useful

Teams building agents for booking, support or coding get a clear principle: measure the marginal value of a question rather than impose a fixed threshold. If an assistant asks too often, users leave. If it asks too rarely, it makes mistakes silently. Paired rollouts offer a way to calibrate this balance on real task outcomes.

The limitations are also visible. The results come from a small 4-billion-parameter model and have not yet been peer reviewed. Paired rollouts from the same history imply extra compute during training. The abstract does not say how the method will behave on larger models or in real products with live users. Still, the scheme itself, in which a question's value is re-estimated as the agent grows, looks applicable beyond this particular implementation.