In the HALO Benchmark, 61.20% of Rejected AI Agent Responses Came Too Early or Were Too Intrusive

A proactive agent has to decide on its own whether to help, when to do it and how far to go, all without an explicit request. Researchers at the University of California, Davis and the University of Southern California built the HALO benchmark to test this. Its main finding: agents most often fail not by doing nothing, but by showing too much initiative. Of the responses that human raters rejected, 61.20% (2188 of 3575) suffered from intervening too early or acting too forcefully.

How HALO Is Built and Who Annotated the Responses

The scenarios cover four types of needs: dependency (a precondition for a future task is missing), blocking (the current activity cannot continue), risk (a possible problem has not yet happened) and opportunity (a useful but optional step). They are spread across 10 everyday domains, from study and work to family, health, travel, finance and hobbies. Each domain has five personas, for 50 personal environments in total. Each environment contains eight scenarios, which makes 400.

An environment consists of email, a calendar, chat, contacts and a wallet, and the evidence sits in SQLite databases and text files. The agent has 29 commands for searching data and 19 commands for taking actions. At each checkpoint it sees part of the event chain, can gather data over several rounds, and then must produce a single executable response or decline to act. Actions are split into levels: L0 is observation, L1 is drawing attention, L2 is preparation without external commitments, and L3 is execution with real consequences. A higher level does not mean better help. GPT5.6-Sol generated the scenarios, which were then checked by a validator and reviewers.

The environment feeds the HALO-8K set: 8000 responses from 10 models, 800 per model. Twenty experts ruled on whether each response was acceptable and, for rejected responses, gave a reason: premature, late, insufficient, excessive and so on. Sensible inaction counted as a correct response.

What the Results Show

The spread is wide: from 30.38% for GPT-OSS-20B to 72.88% for Gemini3.8-Flash. Even the best model gives an unacceptable response in more than a quarter of cases. Among open models, DeepSeekV4-Pro leads with 56.63%, and it beat the closed Gemini3.1-Pro and GPT5.6-Luna. On average, closed models scored 63.08% and open models 47.55%, but the three strongest open models fall within 2.50 points of each other, and a weak tail drags the average down.

Agents most often chose level L1 (69.20% of responses), L3 accounted for 16.09%, and L2 for just 4.18%. The share of execution is nearly identical in both groups, at about 16.1%. Closed models choose L0 twice as often. Premature intervention occurs almost three times as often as late intervention, and excessive force about twice as often as insufficient force. The two best models make the fewest such errors.

Inaction alone does not guarantee quality. Qwen3.8-27B almost never errs by inaction (0% Inaction), but it picks L0 in only 0.38% of cases and reaches just 55.13% acceptability.

Data Search Matters More Than the Volume of Reasoning

Open models spend an average of 4.77K reasoning tokens and run 1.21 search rounds per response. Closed models use 5.53K tokens and 2.55 rounds. Open models use only 14% fewer tokens but roughly 53% less search. Among reminders, consulting local content is associated with higher acceptability and less often leads to an "insufficient" rating. Claude-Opus5.5, next in the ranking, spends about 23% of the leader's reasoning tokens, so a high score is possible at very different resource costs.

Even when the timing of a reminder is right, reminders are often insufficient and execution is often excessive.

Limitations

Acceptability varies little by need type (54.35–56.00%) and ranges from 50.88% to 58.88% by domain. The evaluation uses a sample of 800 responses per model, and tokens are counted under different providers' rules, so cost comparisons should be made with caution. A model generates the scenarios, although humans check them. The takeaway for developers: measure a proactive assistant not by how many needs it detects, but by how well it judges the timing and scale of its actions, and keep L0 and L1 as the safe default modes in a product.