Proactive agents in HALO are intrusive: 61.20% of rejected responses came too early or were excessive

Researchers from the University of California, Davis and the University of Southern California have introduced HALO, a benchmark for proactive agents: assistants that decide on their own whether to help, when to do it and how far to go, without waiting for a direct request. The authors enlisted 20 experts to manually evaluate 8000 responses from 10 models. The main finding is an uncomfortable one for the whole industry: even the best model provides acceptable help in only 72.88% of cases, and the main cause of failure is excessive initiative.

How HALO works and what it tests

The benchmark defines four types of needs. Dependency is a missing condition for a future task. Blockage is a situation in which the current activity cannot continue. Risk is a possible adverse outcome that has not yet occurred. Opportunity is an optional benefit whose absence breaks nothing.

All of this is spread across 10 everyday domains: study, work, home, family and caregiving, health, travel, personal finance, community life, social and cultural life, and hobbies and sports. Each domain has five characters, which makes 50 personal environments, and each environment has eight scenarios, for 400 in total.

The HALO-Arena environment includes email, a calendar, chat, contacts and a wallet, backed by SQLite databases and text files. The agent gets 29 commands for retrieving information and 19 commands for taking actions. Before responding, it has to find evidence in structured or unstructured data and only then decide whether help is justified. The paper gives an example: a trip entry in the calendar may prompt the agent to check the database for needed items and a file with a purchase authorization. Those facts justify a reminder, but not the purchase itself.

Responses fall into four levels: L0 is observation without intervention, L1 is drawing the user's attention, L2 is preparing an action without external commitments, and L3 is execution that changes commitments or has an external effect. A higher level does not mean better help: when restraint is appropriate, L0 is accepted too.

The scenarios were built with GPT5.6-Sol and checked by a validator, a reviewer model and two human auditors; a third resolved disputed cases. For labeling, the team took 2160 checkpoints, 800 responses per model. An expert first decides whether the response is accepted. For rejected ones, the expert then specifies what went wrong: too early, too late, insufficient, excessive, or other problems such as a lack of grounding in data.

Results: a wide spread and a leader with no real margin

Acceptability ranges from 30.38% for GPT-OSS-20B to 72.88% for Gemini3.8-Flash, a gap of 42.50 points. Among open models, DeepSeekV4-Pro leads with 56.63%. Closed models average 63.08% and open models 47.55%, but the groups overlap: DeepSeekV4-Pro outperformed both Gemini3.1-Pro and GPT5.6-Luna.

Models most often choose L1, simply drawing the user's attention: 69.20% of all responses. Preparation (L2) is rare at 4.18%, and execution (L3) accounts for 16.09%. The share of execution is nearly identical in both groups, at about 16.1%, so it does not explain the quality gap. Closed models choose inaction about twice as often.

The difference in data search is notable. Open models run an average of 1.21 search rounds per response versus 2.55 for closed models, roughly 53% fewer, even though reasoning-token consumption is comparable: 4.77K versus 5.53K. Meanwhile, Claude-Opus5.5, which placed second among closed models, uses about 23% of the leader's reasoning tokens, so a strong result does not require maximum spend.

The main problem is overreach, not passivity

Of 3575 rejected responses, 2188 (61.20%) involved either a reaction that came too early or excessive intensity. The share by model ranges from 44.23% to 79.66%, and for six of the ten it is above half. Premature intervention is almost three times as common as late intervention, and excessive intervention is about twice as common as insufficient intervention. The pattern holds across all four types of needs and all ten domains. The two best models also have the lowest rates of both prematurity and excess.

Restraint contributes too: Qwen3.8-27B never erred by inaction, but it chose L0 in only 0.38% of cases and scored just 55.13%. Among reminders, access to the contents of local data is associated with higher acceptability and a smaller share of "insufficient" responses. When the timing is right, reminders more often turn out insufficient, and execution more often turns out excessive.

Limitations

The data covers sampled scenarios and recorded inference settings, and token counts depend on the providers' accounting rules. The scenarios are model-generated, although humans checked them, and a single annotator handles each scenario. For developers, the takeaway is practical: when designing proactive assistants, measure not only whether they trigger but also the timing, scope and authority of the intervention, and make the agent look for supporting evidence in the user's data first.