LIBERO-Industry tests 5 robot policies on 216 factory tasks

The authors of LIBERO-Industry assembled 216 tasks that test generalist robot policies under assembly-line conditions: a moving conveyor, a limited window for action, and tight geometric tolerances. The five models they tested, including vision-language-action and world-action models, lose performance as constraints are added and stacked on top of each other. The benchmark is built on top of the popular LIBERO and answers a question that lab tests have sidestepped: what happens when the product changes and the belt does not stop.

Where conventional evaluations break down

Flexible manufacturing, in particular high-mix, low-volume (HMLV) electronics assembly, runs on constant change. Batches are small, product variants replace one another, and conditions on the line drift. A robot here has to adapt rather than repeat a single memorized trajectory. The authors point to a gap: existing evaluation frameworks do not capture industrial constraints systematically when it comes to generalist policies. A model can perform confidently on standard tabletop tasks without ever encountering a belt that carries a part away within seconds.

LIBERO-Industry turns these constraints into controllable task conditions and measurable success criteria. Each factor can be adjusted on its own or in combination with others.

How the benchmark is built

The set consists of three classic manufacturing scenarios. They differ in which side of the operation is moving: the source, the destination, or both.

• Static-to-Dynamic (S2D): loading, where an object is picked from a static position and placed onto a moving flow.
• Dynamic-to-Static (D2S): unloading, where an object is taken off a moving flow and placed at a static point.
• Dynamic-to-Dynamic (D2D): transfer between two moving flows.
• 216 tasks in total.
• Varied parameters: object geometry, conveyor speed, interaction window duration, and the size of the permitted geometric region.
• Five generalist policies were evaluated, including VLA models and world-action models.

The parameters are chosen to reproduce real line requirements. A short interaction window means the policy has little time to grasp. A small permitted region means an approximate hit already counts as a defect. A fast belt combines both pressures.

What the experiments showed

The main result is predictable in direction but useful in structure: policy performance declines gradually as industrial constraints are introduced, and falls further when they are combined. Degradation does not come as a sudden drop on a single factor. The constraints compound, and it is the combinations that prove hardest. For teams choosing a model for a production line, this matters more than an average score on a classic benchmark.

The second finding concerns training. Under a unified fine-tuning and evaluation protocol, the co-training condition with LIBERO-Industry data produced a higher observed share of completed target-object transfers than the condition with the original LIBERO. The abstract gives no exact numbers, so the size of the gain cannot be judged from it. There is also a caveat: LIBERO remains a simulation benchmark, and the transfer of these results to real hardware with actual conveyors is not demonstrated here. Moreover, a gain from training on data close to the test data is expected. What is more interesting is something else: the training mix needs data specifically about dynamics and tolerances, not just about variety in objects and instructions.

Practical takeaways

If you are evaluating a robot policy for manufacturing, do not stop at success on static tasks. Run it across the axes of flow speed, window duration, and tolerance size separately, and then in combinations: the weak spot usually shows up where factors intersect. Developers should build scenes with moving objects and narrow tolerances into their fine-tuning datasets rather than hoping that generalization from tabletop data will work on its own. Finally, numbers obtained in simulation should be treated as a lower bound on validation requirements. Testing on a real line with real product variants remains mandatory.