EmbodiedLab: Robots Tested on Lab Experiments, and the Best One Solved Fewer Than Half

Researchers at HexaFuture, whose authors include Hongcheng Gao, Bryan Hooi, Yinpeng Dong and Wentao Zhang, have introduced EmbodiedLab. It is a benchmark for embodied agents that must run an experiment on their own and extract knowledge from it. The set contains 30 quantitative experiments from five scientific disciplines on 14 lab setups. Even the strongest of the six agents tested solves fewer than half of the tasks under native control.

Finding Out, Not Rearranging

Most robotics benchmarks test the ability to change the world: pick up a cup, open a drawer, stack blocks. The target state is known in advance, and the agent only has to reach it. Scientific-discovery benchmarks, by contrast, test knowledge, but they replace the physical experiment with symbolic actions or numerical queries. The agent calls a function and gets a number back instead of wrestling with a real instrument.

EmbodiedLab fills the gap between these two approaches. The agent receives a goal and a description of the setup. It chooses the procedure itself, controls the robot in a physics-based lab simulator, and reads instrument values through cameras. Changing the physical state is a means here, not the goal: the agent pushes the system, watches how it responds, and probes it until it finds what it is looking for, such as a physical law.

How to Build a Fair Exam

The main problem with a test like this is that the answer must not be in the training data. The authors solved it with quantitative experiments: system parameters are resampled for every task instance. The result cannot be recalled or guessed, yet it can still be scored precisely.

The evaluation checks two things:

• what the agent ultimately learned, where the answer must rest on readings recorded during the episode; • whether the physical state the task requires has been reached.

Every experiment has also been verified as solvable, so a failure cannot be blamed on a broken task.

Six Models, Three Modes

The authors ran six frontier models inside the coding agents built by their own developers. Each was tested in three configurations:

• native observation and control, with no add-ons; • with an added perception and motion-planning module; • with manipulation skills extracted from the strongest agents.

Under native control, even the best agent succeeds in fewer than half of the experiments. For all six agents, most failures stem not from reasoning but from interacting with the setup itself: the manipulator has to reach the right point, handle the instrument correctly, wait for the system to settle, and take the reading. A model may understand which experiment to run and still fail to carry it through.

Another finding concerns what helps. Manipulation skills obtained from the strongest agents give weaker models more than added perception and motion planning do. The bottleneck, it seems, lies in practiced operational techniques rather than in seeing the scene better or computing trajectories.

Who This Test Is For

For agent developers, EmbodiedLab offers a metric that did not exist before: the ability to close the "hypothesis, action, observation, conclusion" loop in a physical environment. This is the ability expected of lab automation and of autonomous assistants in science. For teams building such systems, the result suggests a priority. Before pushing reasoning capabilities further, it makes sense to invest in reliable instrument control and libraries of manipulation skills.

The benchmark and evaluation code are open on GitHub in the HexaFuture/EmbodiedLab repository, so new models and agents can be run on the same 30 experiments and compared with the six baseline systems. The paper is published on arXiv under number 2610.35385. One limitation remains obvious: everything happens in simulation, and transfer to real labs will need separate validation. Still, as a diagnostic of where exactly embodied agents break down, the set looks useful.