TARSphere Tests MLLM-Powered Robots Across 360 Episodes in 30 Scenes

The TARSphere team has released a full software stack for evaluating robots controlled by multimodal language models in a world where the environment changes and people change their minds. At its core is the TARSphere-Bench benchmark: 360 episodes in 30 indoor scenes, five paired test conditions, and two base tasks, "go and report" and "find and deliver." A run of eight MLLMs painted an uncomfortable picture. General-purpose models can outperform specialized embodied models, and collisions and falls remain frequent even when every model gets the same action interface and decision budget.

A simulator where people want things too

Most existing systems for embodied agents evaluate completion of a predefined goal. Others simplify the physics down to symbolic actions such as "pick up an object" or "go to a room." In reality, a robot in an apartment or office faces something different: a door is suddenly closed, a person asks to change an order, and a single phrase does not always make clear what they actually want.

So the bottom layer, TARSphere-Infra, runs robot policies in a physics simulator. Alongside the robot are embodied NPCs, virtual humans the robot can talk to within a limited dialogue. The outcome is scored by an event-driven finite-state machine (FSM). It tracks completed actions and information handed over, updates task states, and issues a verdict. The FSM reacts to what actually happened in the simulation, not to the agent's own account of its successes.

A brain on a frozen model

The second component, TARSphere-Harness, serves as the robot's "brain harness." The frozen MLLM is not fine-tuned for the task. It acts through multi-level executable skills and draws on three sources: observations, hierarchical memory, and execution feedback. The last one matters: the model sees that its action did not work and revises its decision.

This approach makes comparison straightforward. When all eight models share one skill set and one decision budget, differences in results come from the model itself, not from the quality of the harness. The benchmark itself was built with an agent in the loop: agent-in-the-loop curation aligns the meaning of tasks, the causal dependencies between steps, and the geometry of the scenes. That keeps out episodes where an object is physically unreachable or an instruction contradicts the layout.

Where robots break

The five paired conditions test different aspects of robustness:

• perception robustness;
• adaptation to disturbances and failures;
• task updates during execution;
• coordination with a human;
• intent understanding.

The main finding of the evaluation is that general-purpose MLLMs can outperform models tuned for embodied scenarios. That challenges the assumption that narrow fine-tuning automatically yields a more reliable robot. Broad world knowledge and strong reasoning in general-purpose models, judging by the results, sometimes matter more than specialization.

The weak spots are shared. Task updates and interaction with people remain difficult for every model. An agent that follows a plan well gets lost when the plan has to be rewritten on the fly or when it must clarify what the person wants. The authors separately note frequent collisions and falls. A successfully delivered object means little if the robot knocked over furniture on the way.

What to change in robot evaluation

For teams building agents on top of MLLMs, TARSphere offers several practical pointers:

• Measure execution safety alongside the task completion rate. Two systems with the same success rate can differ sharply in the number of collisions.
• Test task changes and dialogue with a human separately. By the benchmark's results, this is where models fail most.
• Compare models with a fixed action interface and decision budget. That shows what the model contributes and what the harness contributes.
• Do not discard general-purpose MLLMs in favor of embodied models without testing on your own scenarios.

There is also a limitation: all of this happens in simulation, in 30 internal scenes, and the NPCs speak within a limited dialogue. Moving to real hardware will add sensor noise, mechanical wear, and the unpredictability of people. Still, as a working framework for comparing robot "brains" before they enter the physical world, TARSphere looks practical. It pushes evaluators to look not only at whether the robot reached its goal, but also at what it did to the people and objects around it on the way.