Humanoids Learn to Serve in Tennis Like Pros: Planner and Tracker Split Apart

Humanoid robots have made significant progress in performing ball-sport tasks, but combining a professional movement style with reliable task execution remains an open problem. Noitom Robotics and IEEE Spectrum have published a white paper describing an adaptive planning-and-tracking architecture for motion. This architecture lets humanoids learn to perform tennis serves and rallies while preserving the style of professional athletes.

Architecture: splitting the functions

The traditional approach combines kinematic trajectory generation and execution tracking into a single system. This creates an inherent conflict: the requirements for aesthetic, stylistically correct movement clash with the requirements for precise tracking. The solution is to explicitly split the system into two hierarchical levels.

The high-level planner generates kinematic trajectories that preserve the characteristic style of a professional athlete's movement. The low-level tracker is responsible for precisely physically embodying these trajectories with minimal distortion. Source data is extracted from broadcasts of sporting events and motion-capture recordings of professional tennis players. This data goes through a process of reconstruction, retargeting onto the robot's anatomy, and physical correction to ensure safe and realistic execution of movements on the humanoid platform.

Bridging the sim-to-real gap

One of the key challenges in transferring a policy trained in simulation to real hardware is that motion-tracking errors accumulate exponentially. In autoregressive planning systems, a small error at one timestep leads to cumulative errors at subsequent steps, risking instability and task failure.

The researchers implemented a two-tier approach: the low-level tracker is trained to execute movements at various, randomly varied execution speeds, which increases robustness to perception noise and real-world inaccuracies. At the same time, the high-level planner is trained on a learned motion-speed representation, allowing it to adaptively select speed depending on state and conditions. This combination of approaches significantly reduces the compounding errors typically seen when transferring from simulation to real hardware.

Validation and practical results

The architecture was tested on two different humanoid platforms: a research robot and a full-size humanoid. The most significant result is the full-size robot performing in-the-wild tennis serving in an unconstrained environment without using a motion-capture system. This demonstrates practical applicability in real-world conditions, where the robot relies solely on onboard sensors and visual perception.

The white paper concludes with practical recommendations for engineers developing humanoid robot-athlete systems. The research covers adaptive motion tracking, robot sports in ball games, hierarchical motion control architecture, humanoid motion control, learning from video and motion-capture data, motion planning and tracking, motion retargeting across different anatomies, movement style, real-world robot deployment, reinforcement learning, robot perception, sports robotics, sim-to-real transfer, and whole-body robot control.