TerraAgent checks whether the AI actually looks at the image: hallucinations drop from 17.1% to 9.8%

The authors of TerraAgent took on a problem that plain accuracy cannot see: a model can answer a question about a satellite image correctly without ever relying on the image itself. The framework (called AVA in the paper) wraps a frozen multimodal model for remote sensing in three modules and requires no fine-tuning. On the RSVQA-LR, RSVQA-HR, VRSBench and LHRS-Bench benchmarks, average accuracy improved by up to 2.40 points over the strongest baseline methods under comparable conditions. The share of answers built on prior knowledge alone fell from 46.2% to 21.7%. The share of hallucinations dropped from 17.1% to 9.8%.

Accuracy that proves nothing

In RS-VQA, questions like "is there a body of water near the road" often have predictable answers. The model knows what cities and countryside usually look like and guesses from statistics. The accuracy metric counts that luck as success. The authors identify three sources of the problem. In-context examples drawn from different sources match poorly in resolution (GSD), sensor type and urban texture. Real queries drag in a long visual context: tiles at different scales, frames taken at different times, retrieved reference data. Evidence gets diluted in that context, and errors accumulate along the reasoning chain. Finally, standard metrics simply do not register an answer that does not rely on the image.

Three modules around one model

GDR handles the selection of demonstrations. It groups candidates by GSD, retrieves them through scene graphs, and discards those whose answers are not supported by their own images. A bad example no longer gives the model a wrong reference point.

MACE breaks the query into a directed acyclic graph of sub-questions, each with the type of evidence it needs. The sub-questions go to the appropriate perception tools. Verified results, with their provenance recorded, are written to a shared evidence registry, and repeated perception is capped by a limit on the number of attempts. This turns a long context into a set of short, verifiable steps.

CVA acts as an auditor. It takes the evidence cited by the final answer and perturbs it by removing or substituting it. If the answer does not change, it does not rest on the image. From this the framework computes a visual-dependency score, which is used to calibrate confidence and to trigger a repeat query that forces the model to consult the evidence.

What the measurements showed

• Average accuracy: up to +2.40 points over the strongest baseline methods under the same conditions.
• Visual-dependency score: from 0.28 to 0.49.
• Share of prior-only answers: from 46.2% to 21.7%.
• Share of hallucinations: from 17.1% to 9.8%.
• Cost: 2.6x a single pass of the base model.
• Training: not required; the base MLLM stays frozen.

The accuracy gain is modest. The main result lies elsewhere: almost half as many answers that the model gives without looking at the image.

Where this is useful and what to check

The most obvious use cases are infrastructure monitoring, damage assessment after floods and fires, agriculture, and urban planning. In these areas an answer given "out of habit" can be costly, and the evidence log and visual-dependency score give a basis for auditing. The approach can be transferred to other frozen models when you need to verify that answers rely on the image without fine-tuning.

There are limitations, too. The results come from four academic benchmarks, not from closed production data. The cost of 2.6 passes is noticeable when processing images at scale. Counterfactual verification is only as useful as the model is accurate about which evidence it relied on. Before deploying, measure visual-dependency on your own data and weigh the reliability gain against the added latency and cost.