GeoSpatialBench Tests MLLMs on 28 Types of Spatial Relations Across Six Cities

GeoSpatialBench is a new open benchmark for multimodal large language models (MLLMs). It tests 28 types of spatial relations: topological, directional, metric and ordinal. The test is built on real urban data from six geographically diverse cities. The authors' main finding is that models read structured GeoJSON noticeably better than map images. When asked not to answer a question but to change the placement of objects while respecting geometric constraints, most failed edits violate at least one of the conditions.

How the benchmark differs from familiar tests

Most existing evaluations of MLLM spatial reasoning are simple: static question-and-answer tasks on small or synthetic scenes. The model looks at an image and says what is further left and what is closer. Such tests can barely check whether a model can change a configuration without breaking the given geometric conditions.

GeoSpatialBench is built as a ladder of three steps: understanding spatial relations, geometric reasoning, and constrained spatial editing. Each case is available in two representations: as structured GeoJSON and as a map image. The reference answers and the constraints themselves were verified through GIS operations. In other words, correctness is determined by a geometry engine, not by another language model acting as a judge.

Editing matters more here than it might seem. An agent that has to, say, move an object so that it stays inside a parcel and does not cross a road must hold several conditions at once. For embodied models and robotics, this is a basic operation, not an exotic one.

What the results showed

The authors tested frontier MLLMs. The abstract gives qualitative conclusions rather than per-model tables of numbers:

• GeoJSON consistently outperforms map images.
• The multimodal combination (JSON plus image) does not reliably beat GeoJSON alone.
• On qualitative topological and ordinal relations, models perform decently.
• On quantitative metric relations, where distances and sizes are needed, accuracy drops.
• On ordinal-relation tasks, editing performs worse than question answering.
• Most failed edits violate at least one of the given spatial constraints.

The multimodality result stands out. The image does not help even when precise geometry sits right next to it. The abstract does not explain why, but the working hypothesis is obvious: the visual channel introduces errors into estimates of distances and positions, while text coordinates contain none. The authors call the gap between "recognizing a relation" and "building a configuration that satisfies it" the main takeaway of the work.

Practical takeaways for developers

If you are building a geo-assistant, a GIS agent or a planning system for a robot, the paper suggests three working rules. First, pass geometry to the model in structured form, not as a map screenshot. Second, hand any metric calculations (distances, areas, buffers) to external tools such as PostGIS or Shapely rather than trusting the model's eyeballing. Third, run every edit through a constraint check and send it back to the model for revision until the conditions are met. A single pass without validation, judging by the test, often produces a violated constraint.

For researchers, the benchmark is useful as a diagnostic testbed: it is open and lets you measure understanding, calculation and editing separately. There is one caveat: the abstract gives no figures for specific models, so choosing a model for a given task means reading the full paper and checking its results tables.