EmbodiedLab: robots tested on experiments, and the best one solves less than half

Researchers from HexaFuture, whose authors include Hongcheng Gao, Bryan Hooi, Yinpeng Dong, and Wentao Zhang, have presented EmbodiedLab. It's a benchmark for embodied agents that must independently run an experiment and extract knowledge from it. The set includes 30 quantitative experiments across five scientific disciplines on 14 lab setups. Even the strongest of six tested agents solves fewer than half the tasks under native control.

Discover, not rearrange

Most robotics benchmarks test the ability to change the world: pick up a cup, open a drawer, stack blocks. The target state there is known in advance, and the agent just needs to reach it. Scientific discovery benchmarks, by contrast, test knowledge, but the physical experiment in them is replaced by symbolic actions or numeric queries. The agent calls a function and gets back a number, rather than fiddling with an actual instrument.

EmbodiedLab closes the gap between these two approaches. The agent receives a goal and a description of the setup. It chooses the procedure itself, controls a robot in a physical lab simulator, and reads instrument readings via cameras. Changing the physical state here is a means, not an end: the agent moves the system, observes the response, and investigates it until it obtains the sought result — for example, a physical law.

How to make an honest exam

The main problem with such a test is that the answer must not already be in the training data. The authors solved this through quantitative experiments: the system's parameters are resampled for each task instance. The result can't be recalled or guessed, but it can be precisely evaluated.

The evaluation checks two things:

• what the agent ultimately learned, where the answer must be grounded in readings recorded during the episode; • whether the physical state required by the task was achieved.

Each experiment has also been verified for solvability, so a failure can't be blamed on a broken task.

Six models, three modes

The authors ran six frontier models in coding agents built by their respective developers. Each was tested in three configurations:

• native observation and control, with no add-ons; • with an added perception and motion-planning module; • with manipulation skills extracted from the strongest agents.

Under native control, even the best agent succeeds in less than half of the experiments. For all six agents, most failures stem not from reasoning but from the interaction with the setup itself: the manipulator has to reach the right point, handle the instrument correctly, wait for the system to settle, and take a reading. A model may understand exactly which experiment to run and still fail to carry it through.

Another finding concerns what actually helps. Manipulation skills obtained from the strongest agents give weaker models more benefit than added perception and motion planning do. The bottleneck, it appears, lies in well-honed operational technique rather than in seeing the scene better or computing trajectories better.

Who this test is for

For agent developers, EmbodiedLab provides a metric that didn't exist before: the ability to close the loop of "hypothesis — action — observation — conclusion" in a physical environment. This is the capability expected of lab automation and of autonomous assistants in science. For teams building such systems, the result suggests a priority. Before scaling up reasoning ability, it's worth investing in reliable instrument control and libraries of manipulation skills.

The benchmark and evaluation code are open on GitHub in the HexaFuture/EmbodiedLab repository, so new models and agents can be run on the same 30 experiments and compared against the six baseline systems. The work was published on arXiv under number 2610.35385. One limitation remains obvious: everything happens in simulation, and transfer to real labs will need separate verification. But as a diagnostic for exactly where embodied agents break down, the benchmark looks useful.