ACT3D teaches a VLM to explore 3D scenes and beats Gemini 3.6 Flash: 53.2% across six benchmarks

The authors of a paper featured on Hugging Face Daily Papers have proposed ACT3D, a vision-language model that answers questions about a 3D scene by moving through it on its own. Its average accuracy across six 3D benchmarks reached 53.2%. That is 22.0% higher than the base Qwen model and higher than the proprietary Gemini 3.6 Flash. ACT3D also uses 50–70% fewer input frames than existing open 3D methods.

One frame instead of a point cloud

VLMs usually get 3D understanding through brute force: the model is fed a set of views, a video, or a dedicated 3D representation. The authors took a different route and kept the VLM's native visual input: one image at a time, as a human sees. The 3D scene itself becomes an environment. The model interacts with it through actions such as "move left" and receives rendered images in response.

Each episode begins with a first-person view and a visual question. The model then reasons, moves, changes its gaze direction, and only after that commits to an answer. Training has two stages: supervised fine-tuning on ready-made trajectories, followed by reinforcement learning. The reward is tied mainly to the correctness of the final answer.

The key detail is that RL does not tell the model which viewpoints to visit or which actions to choose. The model has to derive its exploration policy from a single signal: whether interacting with the space helped it answer better or not.

What the numbers show

• Average accuracy across six 3D benchmarks: 53.2%.
• Gain over the base Qwen: 22.0% (the abstract does not specify whether this is in percentage points or relative terms).
• Result is higher than the proprietary Gemini 3.6 Flash.
• Input frame savings relative to open 3D methods: 50–70%.
• On unseen domains: +5.9% on average over the supervised initialization.
• Gain over the base model on questions solvable from the first frame: 0.6%; on questions requiring extended exploration: 16.3%.

The last pair of figures is arguably the most revealing. When the answer is visible right away, ACT3D adds almost nothing to the base model. When the model has to search, the difference becomes substantial. That suggests the model does not drive the camera around the scene needlessly, but adapts its exploration to the question.

The authors describe three behavior patterns. ACT3D looks for objects hidden behind other items. It avoids extra movement when it already has enough information. And it collects complementary viewpoints for spatial comparisons, for example when it needs to compare two objects that do not fit into a single frame.

Where this could be useful

The idea reaches beyond the specific benchmarks. The model learns not only to reason over visual data, but also to decide when and how to acquire it. For robotics and embodied agents, this is a natural setup: a robot also has no pre-built point cloud, only a camera and a set of movements. The outcome-only reward approach removes the need for expensive annotation of "correct" routes.

For engineers building pipelines on VLMs, the takeaway is practical. Saving 50–70% of frames means less load on context and compute at inference, and a base open model can be raised to the level of a closed flagship without changing its architecture. SFT on trajectories followed by RL on task outcome is a fairly reproducible recipe, and it is reasonable to expect it to carry over to other interactive environments.

There are limitations, too. The abstract describes simulated 3D environments with rendering, not real cameras, noisy sensors, or dynamic scenes. The transition to the physical world remains an open question, and future work will answer it.