AgentVQA Tests 22 VLMs on Agentic Tasks: The Best Scores Just 58.5%

The best of 22 vision-language models tested reached only 58.5% accuracy on the new AgentVQA benchmark. It is an offline set of 1000 hand-picked hard examples collected from 23 source datasets. It covers five areas where VLMs work as agents: web agents, robotics, egocentric video, games, and spatial planning. The authors say the model ranking on AgentVQA mirrors the ranking in the corresponding online evaluations. That would mean a fast, cheap test can replace some of the expensive runs in simulators.

Why online agent evaluations run into cost limits

VLMs can do a lot: they describe images, read interfaces, and answer questions about video. But how they behave inside an agentic loop, where they have to look at a screen or scene, choose an action, and see the consequences, is much less clear. The direct way to find out is an online evaluation in a simulator or a real environment. It is necessary, but it is expensive, slow to scale, and its results are hard to compare across teams.

Offline question-answering benchmarks (VQA) could close this gap. The AgentVQA authors say existing sets are fragmented and limited: each covers its own narrow domain, and there is no common scale for agentic scenarios. AgentVQA tries to build such a scale in a single set.

How the 1000 "hard" examples were assembled

The team did not scrape data automatically. It picked examples by hand from 23 sources. The criterion was realistic, genuinely difficult scenes from interactive environments. The examples were then adapted so they can be scored reliably without running the environment. A real situation from an interactive task becomes a question with a verifiable answer, and the original context is preserved.

The main argument for this method is the correlation with online results. If model A beats model B in a simulator, it usually beats it on AgentVQA too. For labs, that means a quick indicator of progress. For customers, it is a way to compare models before an expensive integration.

Key benchmark parameters:

• 1000 hand-picked examples;
• 23 source datasets;
• five domains: web agents, robotics, egocentric video, games, spatial planning;
• 22 models evaluated;
• best result: 58.5% accuracy;
• the model ranking agrees with online evaluations.

Where models break: perception, not logic

The most interesting part of the work is the cross-domain error analysis. The authors examined why models miss and concluded that the cause is not a misunderstanding of the task or weak logic at the next step. The main problem is grounding the question in the image and maintaining an accurate representation of the current, changing visual state.

Specifically, there are four types of failure:

• locating the right interface elements on the screen;
• interpreting motion;
• tracking state changes over time;
• inferring three-dimensional spatial relationships.

All four show up across domains. A web agent has to find a button, a robot has to work out where an object moved, a game agent has to keep track of what changed between frames, and a planner has to judge the relative positions of objects in 3D. These tasks share a root cause: perception remains the bottleneck.

What teams building agents should do

The first practical takeaway: boosting a model's reasoning is not enough if it sees the scene wrong at the input. The better investment is visual grounding: training on data with precise element localization, video with dynamics, sequences of states, and spatial annotation.

The second takeaway concerns the development process. AgentVQA can serve as a fast pre-filter: run candidates offline, keep the best, and send only those to expensive online environments. The correlation with online rankings makes this approach sensible, although it does not replace final testing in a real environment.

The third takeaway is for those choosing a model for a product. A 58.5% result for the leader among 22 models shows that on hard agentic scenarios, current VLMs are far from reliable. If an agent has to click through interfaces, control a robot, or plan in space, it will need perception checks, rollbacks, and human involvement at critical steps. The breakdown of errors by type also points to where engineering effort should go first.