Qwen3-VL-4B Gains 18.33 Percentage Points on the Drone-PUR Benchmark After Fine-Tuning on Drone Imagery

A group of researchers from Tianjin University, Nankai University and Southeast University has published Drone-PUR, a dataset and benchmark that tests how multimodal models (VLMs) find, interpret and connect visual cues in drone imagery. The authors also presented UAV-PUR, a base model fine-tuned from Qwen3-VL. Its 4B and 8B-parameter versions improved overall accuracy on multiple-choice tasks by 18.33 and 17.52 percentage points over the original pretrained models.

Three levels: perception, understanding, reasoning

The benchmark rests on the idea that a model's final answer depends on the quality of the "evidence" in the frame. The authors define a visual cue as a task-relevant image region containing one or more objects, tied to a specific modality, viewpoint and capture time. Perception finds such regions and extracts basic information: category, attributes, count and position. Understanding matches them against semantic conditions and spatial concepts. Reasoning links several cues through logical inference.

An example from the paper: in an image of ducks, the model must spot the birds and a fence, work out the structure of the enclosure and the traces of human presence, and then explain why all the ducks have gathered in the pond.

Drone imagery makes these tasks harder than usual. Objects are small and densely packed, important attributes occupy only a few pixels, and a change of viewpoint alters both the appearance of objects and the system of spatial reference points. In Drone-PUR, more than 90% of annotated regions cover less than 1% of the image area.

What is inside the dataset

The benchmark includes 11 tasks, 175,213 training examples and 20,051 test examples. The inputs vary: single frames, sequences, pairs of RGB and thermal infrared (TIR) images, and synchronized views from multiple drones. The authors took images from eight of their own previously released datasets and added 66,940 new ones. Ten tasks are scored through multiple choice, while detection, grounding (linking an object to a description) and counting are evaluated in an open format, that is, by direct quantitative prediction.

Four reasoning tasks cover causal explanations and conclusions about a situation within a single frame, as well as combining cues from different modalities and viewpoints. The examples were generated with VLMs, then cross-checked by other models and by hand, which took more than 800 person-hours. The Position Spatial Judgment (PSJ, assessing the relative position of objects) task was annotated entirely by hand, and all test examples passed human review.

Results for 22 models and the effect of fine-tuning

The authors tested 22 general-purpose VLMs. Within a single model family, larger models usually handle all three levels better. Spatial judgment is the exception: its quality does not improve steadily with size, and predictions show systematic directional biases. Models struggle most with PSJ and multi-view reasoning (MVR).

Across models, strong perception correlates with strong reasoning. Single-task fine-tuning experiments showed selective transfer in both directions: training on finding and interpreting cues improves reasoning, while training on combining cues improves perception. Masking key local regions and disabling modalities or viewpoints confirmed that reasoning suffers without the important fragments and complementary observations.

UAV-PUR was trained in three stages: perception adaptation, joint training on all skills, and a final consolidation stage with language-only training. In the first two stages, the authors used three auxiliary "heads" with geometric and numerical supervision, which help ground visual representations in aerial-capture tasks. At inference, only the original language head is used. The largest gains came in target matching and multi-source grounding.

Limitations

UAV-PUR still trails the best API model on pure reasoning tasks. Fine-grained spatial interpretation, reliable open-format predictions and combining cues from different sources remain difficult even after fine-tuning. The available description does not name the leading models or give their absolute scores, so for comparisons it is best to consult the project materials at drone-pur.github.io.

For teams building navigation, inspection or monitoring systems on VLMs, the main takeaway is practical: general-purpose models without adaptation to aerial imagery make mistakes on small details and spatial relations, and targeted fine-tuning delivers a noticeable gain but does not close the reasoning gap. The open dataset with a training split makes it possible to reproduce this recipe on your own models.