RealityBench: 26 VLMs Tested on Documenting Physical Work in Blender

Researchers have introduced RealityBench, a set of eleven environments in Blender. They used it to test 26 vision-language models on whether they can look at a frame from a worker's camera or smart glasses, understand what the person is doing, and help with documentation. The best models almost fully solve tasks where the answer is written as text right in the frame. But they read measurements poorly and judge small details and spatial relationships badly. On frame acceptance, the top models are statistically indistinguishable from a single non-specialist assessor.

Why a synthetic benchmark

The idea of helping a worker in the physical world through prompts has preoccupied AI agent research for decades. One of the basic skills here is understanding what a person is doing by seeing through their eyes, and helping them record the result. There is almost no real data for testing such systems: most recordings of physical work are closed, and the variability is too large for a clean comparison.

Kitchens and warehouses offer predictable lighting, geometry, and sets of objects. Installing solar panels on a roof or repairing a car is different: conditions are unconstrained and the spread is huge. The authors modeled such scenarios in Blender and built eleven environments with two types of tasks.

• Acceptability: whether a given viewpoint works as evidence of completed work.
• Documentation: whether the agent understands, from that viewpoint, the state of the environment that matters for the task.

What the 26 models showed

• Coverage: 11 Blender environments, 26 VLMs.
• Documentation: top models mostly cope when the answer is text in the frame.
• Weak spots: reading measurements, small details, spatial relationships, such as the tilt of a tape measure.
• Frame acceptance: the best models are statistically indistinguishable from a single non-specialist assessor.
• Cost of a run: within the price range of human review.
• Transfer to reality: on three questions with real photos, the model rankings from the renders held.

Frame acceptance is harder than documentation. Human assessors disagree with each other, so the ground truth is blurry. The authors used an evaluator that agrees with the majority opinion about as often as an individual human does. On this scale, the best models do not differ from a single assessor who did not take part in writing the paper. Running them costs roughly the same as a human pass.

In three environments, the authors were able to break down the causes of acceptance errors. Most false rejections come from misreading the scene, not the requirement. The model understands what needs to be shown but sees incorrectly what is in the frame. So weak vision on small details directly hurts the usefulness of such a system.

The match between rankings on renders and on real photos is an argument for the approach itself. The sample is small, just three questions with real images, so this is more a feasibility check than final proof.

What teams building field AI tools should consider

For products with smart glasses and photo reports, the takeaway is pragmatic. Recognition of inscriptions and labels can be automated today. Tape measure readings, gaps, and tilts are better left for a human to check or confirmed with a second shot.

When evaluating frame acceptance, compare the model with a single live reviewer rather than with an imagined ground truth. When people themselves disagree, this measurement is more honest. Since most errors come from misreading the scene, it makes sense to improve the system at the perception level: viewpoints, close-ups, retakes. Rewriting checklists will yield less.

Synthetic Blender environments make it possible to test models where real data is closed. A new generation of VLMs can be run through them before field pilots appear.