iPrefBench: iOS agents get 91.6% of actions right but solve only 47.7% of tasks

An agent that "knows you" and acts on your behalf remains the central promise of AI assistants. The new iPrefBench benchmark shows how far that promise is from reality. On 130 deliberately incomplete tasks across 26 iOS apps, seven frontier agents reach 91.6% on the action-level score, but fully complete no more than 47.7% of tasks. Personalization eats the rest.

Two situations where an agent has to think

Modern agents that operate a user interface handle fully specified commands with confidence: open an app, move through the screens, tap the right buttons. But real people don't talk that way. A request like "order my usual" or "book it like last time" leaves out half the parameters. To fulfill it correctly for that particular person, the agent has to choose one of two strategies.

The first is to infer what's missing from the user's activity history. The second is to ask about what the history doesn't contain and can't contain. The agent must neither invent assumptions with no data behind them nor flood the person with unnecessary questions. The benchmark's authors build their evaluation around this balance. By construction, every missing user-dependent requirement is either history-inferable (it can be deduced from the history) or ask-only (it can be learned only by asking).

A persona with 3 051 events and a simulated conversation partner

The environment is executable: agents work inside an iOS environment rather than answering questions about screenshots. At its core is a single coherent persona with 3 051 events of past activity. This is the history from which an agent can draw preferences. For dialogue, the benchmark uses an interactive simulated user: if the agent decides to clarify a detail, a simulator answers it rather than a pre-recorded reply.

Benchmark parameters:

• 130 incomplete user tasks;
• 26 apps;
• 3 051 history events in one persona;
• 2 types of missing requirements: those inferable from history and those that require a question;
• 7 frontier agents tested;
• 2 groups of metrics: personalized task execution (whether it was done correctly for this particular user) and agent-user interaction (whether the agent asked when it needed to);
• best result on the action-level score: 91.6%;
• best result on full task success: 47.7%.

Where agents break

The main finding is the gap between "I can tap" and "I understand what the person wants." The execution mechanics of the best agents are nearly polished, and by action scores they look almost perfect. But a task counts as solved only if the outcome matches the user's preferences, and here the result drops by almost half.

Ask-only information is a particularly weak spot. Agents lack the discipline to stop in time and ask a question: they under-request the data they need and press on with guesses. That is a dangerous mode for real-world scenarios such as bookings, payments and messages, where a wrong assumption turns into a wrong action. For history-based tasks the picture is milder in general, but even there agents have to pick out what they need from thousands of events.

The paper is published on Hugging Face, on Apple's organization page, and its authors are Yuting Ning, Ruijia Cheng, Sushobhan Ghosh, Peitong Duan, HungShi Lin, Abhishek Sundararajan, Shreyash Pandey, Huan Sun, Jeffrey Nichols and Eldon Schoop. The choice of iOS and a simulated user in itself shows where developers are looking: a personal agent on a phone must not only execute but also know the limits of its own knowledge.

For those building or choosing mobile agents, the takeaway is practical. Step-level and action-success metrics no longer tell you anything about quality: you need to measure separately whether the agent filled in the missing parameters correctly and how many times it asked where it couldn't proceed without a question. It is worth testing the clarification policy, meaning when the agent must stop, and not just click accuracy. And until agents learn to tell "I can find this out" from "only the user can say this," fully delegating personal tasks to them will remain risky.