SpatialBrain Teaches a Frozen VLM to Accumulate Experience, Winning on 4 of 5 Benchmarks

The SpatialBrain team proposes a way to improve a vision-language model's spatial reasoning without fine-tuning: the weights of the base VLM stay frozen, and a layer of memory, skills and accumulated experience is built around it. On five spatial reasoning benchmarks, the system outperforms the same model's direct answers on four and ties on the fifth.

When a glance is enough and when you need to measure

The core idea of the work is simple. Giving a model geometric tools is not enough, because the question of choosing remains. Sometimes the answer is visible in the image. Sometimes the scene has to be reconstructed, a distance measured, or object positions recomputed in a different reference frame. An agent that calls reconstruction on every question wastes resources. An agent that always answers by eye gets it wrong where precision is needed.

The authors separate two kinds of knowledge. The first is evidence about a specific scene: where the table is, how far away the door is. The second is experience in how to reason with that evidence: under what conditions direct observation is enough, and when a measurement is needed. The first is tied to one scene and useless in another. The second transfers between tasks, and that is what SpatialBrain tries to accumulate.

Three layers around a frozen model

The architecture has three components that run on top of an unchanged VLM:

• 3DFusionMemory merges observations from the current scene into spatial relations that can be queried.
• Spatial Skill Library stores reusable procedures for working with tools: how to reconstruct, how to measure, how to change the reference frame.
• Spatial Cognition links task conditions to experience: when to bring in memory, which skill to call, and how.

The first two layers supply resources; the third teaches the system how to manage them. In effect, it is an attempt to pull into a separate module what usually hides in prompts and heuristics in agent code.

Experience without answer leakage

The accumulation mechanism is carefully designed. Completed trajectories are not stored as they are. Instead, they are abstracted into process records: which decisions were made and what they led to. Between task blocks, these records are consolidated, and later decisions draw on earlier runs.

The key constraint: neither scene coordinates nor reference answers carry over into subsequent tasks. This matters for evaluation. The quality gains cannot be written off as memorization of specific examples, because only knowledge of the reasoning procedure is passed on.

The overall picture across the five benchmarks is uneven. Four improvements and one tie mean there is no universal win: if the scene and question are such that a direct answer is enough, the add-on contributes nothing. But it does no harm either, which is a decent result for an agent layer.

The authors also provide component comparisons, experience coverage curves and traces of resource decisions. These show how the system accumulates experience and in which situations it turns to memory and skills. The coverage curves are especially useful: they show how much experience has to build up before it starts to affect quality.

What to take away for agent pipelines

For those building agents on VLMs that are closed or expensive to fine-tune, SpatialBrain offers a working template. You can leave the model alone and improve around it: a separate scene memory, a catalog of procedures, and a decision log that persists across individual tasks. The principle of separating "evidence about the scene" from "reasoning experience" applies beyond spatial tasks, for example in robotics and navigation, where a model chooses between fast perception and costly measurement.

The practical takeaway: treat the tool-call budget as part of quality. If experience suggests that a glance is enough for a given type of question, reconstruction is postponed. Such systems should be tested on several benchmarks at once, since the tie on the fifth shows that the gain depends on the type of task.