OpenDS-World: The Best AI Agent Solved 39% of Deep Search Tasks on the Live Web

Deep Search agents have learned to answer questions over static document collections convincingly, but the live internet works differently. The OpenDS team proposes testing agents where they will actually operate, and has released OpenDS-World, a benchmark of 200 hard tasks. The results are sobering: of the 12 frontier models tested, the best reached 39.0% accuracy. Humans, across 1000 independent attempts, scored 21.1%.

Why static tests no longer reflect reality

Most earlier deep search evaluations relied on a fixed corpus or simplified retrieval. The real web is harder: the needed facts are scattered across independent sources, stored in different modalities and file formats, and sometimes reachable only by interacting with a page. An agent has to click, scroll, open attachments, extract data from documents, and cross-reference what it finds.

The authors therefore define the OpenDS task as multi-step research on the live web. The agent searches through a search engine on its own, works with a browser, processes the evidence it finds, links pieces from several sources, and in the end must give one factual answer. There is no room for "approximately" or "probably": the answer must be unambiguous and verifiable.

How OpenDS-World is built

The benchmark consists of 200 tasks. They cover different information modalities, heterogeneous file formats, and scenarios where reaching the data requires interactive actions. Humans independently checked each task to confirm that the chain of evidence is valid and the answer is unique. For a live-web benchmark, this is a key requirement. If a question has several correct answers, the metric loses its meaning.

All 12 models ran in the same environment, with a single set of tools and a single execution protocol. The differences in results therefore reflect the quality of the agents themselves, not a lucky scaffold.

Key parameters of the study:

• 200 tasks with independent human verification
• 12 frontier models in a unified environment
• best model result: 39.0% accuracy
• human evaluation: 1000 independent attempts, 21.1% accuracy
• analysis along four axes: tool usage, search cost, termination behavior, recovery from failures

Where agents break down

The most interesting part of the work is not the headline number but the analysis of behavior. The authors compared how models use tools, how much they spend on search, when they decide to stop, and how they get out of a dead end after a failed step. They found that the models' research strategies differ widely. Some dig deeper into sources, others reformulate queries more often, and still others give up too early or, conversely, keep searching with no chance of success.

There are two common weak spots. The first is tool coordination: search, browser, and file processing have to be joined into a coherent chain, and agents lose information at the seams. The second is long-horizon management. When a task takes dozens of steps, an early error carries through to the end, and the recovery mechanism does not always kick in.

The comparison between 39.0% and 21.1% calls for caution: conditions for humans and agents differ in time, constraints, and familiarity with working through interfaces. But the fact that the tasks proved hard for both sides points to a high bar for the benchmark. It is not a test of erudition but a test of endurance and research discipline.

What to change in agent pipelines right now

For teams building research agents, OpenDS-World offers a practical reference point. First, evaluating a product on static datasets is no longer enough: you need a run on the live web, with non-fixed content and interactive access. Second, metrics should be broadened. Accuracy that ignores search cost and the number of steps hides what each correct answer costs. Third, stopping and rollback policies are critical: an agent must know when to change strategy and when to admit that no evidence exists.

For model developers, the benchmark works as a diagnostic tool. It shows exactly where quality is lost: in source selection, in file handling, or in planning a long chain. That is where effort will have to go if agents are to stop being demos and become a working tool for researchers.