SpatialScout: Best VLM Agent Builds Only 41.71% of a Room's 3D Map

Models can answer questions about space, but that does not mean they can measure it. The authors of the SpatialScout benchmark tested exactly this: they gave VLM agents a camera, a list of objects, and a limited number of steps, then required them to say where everything in the room is located. Claude Opus 5.5 posted the best result, with an average overlap score of 41.71%. 47.23% of target objects cleared the IoU threshold of 0.5, and only 3.25% cleared 0.9.

An agent with a camera and a limited step budget

Most spatial intelligence tests work like a quiz: they show an image or a video and ask about the relative positions of objects. An answer like "the lamp is to the left of the sofa" can be guessed from the rough composition of the frame, and it says nothing about whether a model can build an accurate map. SpatialScout closes that loophole. The agent decides for itself where to point the camera, which frames to request, and how to match the same object across different shots.

The agent's input is an object inventory, RGB observations, camera calibration, and camera poses. Its output must be object centers in a shared coordinate system. The dataset includes 80 diverse indoor scenes and 3140 target objects. Each scene allows only a limited number of camera interactions, so the agent cannot circle the room endlessly and recheck everything.

The task requires four skills at once: choosing a viewpoint, linking observations across views, estimating geometry, and revising the map when a new frame contradicts an earlier hypothesis. A weak link in any of them breaks the result.

Where the map breaks down: from room to centimeters

The metric is strict. The score accounts for both localization accuracy and target coverage, and missed predictions are penalized. An object the agent fails to find counts as a failure rather than simply dropping out of the statistics. Comparison uses overlap scores scaled to the object itself: for a small mug, the permissible error in meters is noticeably smaller than for a wardrobe.

Key numbers for the best of the tested models:

• average overlap score: 41.71%;
• share of targets with IoU above 0.5: 47.23%;
• share of targets with IoU above 0.9: 3.25%;
• scenes in the benchmark: 80, target objects: 3140.

The gap between 47.23% and 3.25% shows where the boundary lies. The model broadly understands that a room contains a sofa, a table, and a window, and roughly estimates where they are. But when the task demands accuracy good enough for navigation or manipulation, the result falls apart. The authors describe this as a gap between recovering a rough layout and building a complete, accurate metric map.

It is also worth noting that the researchers tested frontier models, not weak baselines. If even the leader covers fewer than half of the targets at an acceptable level, the problem runs deeper than prompt tuning or a lucky choice of viewpoints.

What to change in your pipelines now

For teams building robots, AR assistants, and agents for physical environments, SpatialScout is useful as a diagnostic. It shows at which stage a model loses accuracy: when choosing viewpoints, when matching objects between frames, or during geometric estimation. That matters because each ailment needs its own cure.

Practical takeaways:

• do not rely on a VLM as the sole source of a metric map; it is better to entrust geometry to specialized modules such as SLAM or multi-view reconstruction, and leave the language model to semantics and planning;
• evaluate an agent's spatial abilities with IoU-threshold metrics, not question answering: an average score hides a tail of inaccurate predictions;
• build a map-revision mechanism into the system, because revising hypotheses in light of new observations remains the weak spot;
• watch coverage: for a robot, a missed object is worse than an object with a small coordinate error.

The benchmark code is published on GitHub, so any lab can run its own agent on it and compare against the Claude Opus 5.5 numbers. For now the bar is low: to produce a map you can rely on, models will have to improve markedly in accuracy at the upper IoU thresholds.