Coding agent replaces teleoperator: robots score 93 successes out of 100 versus 84

Collecting demonstrations through teleoperation remains the standard in robotics, but every episode takes robot time and operator time, and every new task starts from scratch. Researchers from the DAGroup at Peking University have proposed PyRUA-DataFarm, a "data farm" in which a coding agent takes the teleoperator's place. The agent receives a data request and one calibration episode, meaning a short recording of the robot in the target scene. The output is synchronized camera views labeled with the robot's real actions. Policies (control models) trained only on this data succeeded in 93 of 100 attempts on five real-world tasks. Policies trained on the same number of teleoperated episodes succeeded in 84 of 100.

How the agent replaces the operator

The agent builds the task in the simulator the way an engineer would. It reconstructs the scene and cameras from the calibration episode, checks that the simulator can reproduce the required skills, and writes an expert controller together with a success judge. The calibration episode defines the scene and cameras, but it is not the source of actions.

The simulator then runs the expert from many initial states and outputs trajectories with exact actions and object states. A video model conditioned on real frames renders each run for all cameras at once, so the views are consistent with one another. A new appearance, such as a different tablecloth or lighting, is specified with a text description.

There is a separate question: who checks the data? An agent that writes both the expert and the success check can approve its own mistakes, and a video model can copy the layout of its references, paint in extra objects, or drift away from the simulated motion. So certification is kept separate from production. The expert is developed against a judge it cannot see, and it is admitted only on held-out initial states. The renderer may change appearance, but not geometry, objects, or timing. Every rendered episode is checked against the simulation state by automated checks and by an independent reviewer.

Results on a real robot

The experiments covered five bimanual tasks involving rigid, articulated, and deformable objects, as well as objects with a display. In the original scene, the farm's data produced 93 successes in 100 attempts, against 84 for teleoperation. The gap widened when conditions changed. Under a tablecloth and colored light that appeared in none of the training sets, the farm's policies kept 87 successes, while the teleoperation-trained policies dropped to 46. The authors attribute this to the appearance variation the farm adds around fixed, exact trajectories.

Rendering from real images proved decisive: the same trajectories with standard domain randomization produced only 26 and 21 successes. By image statistics, the farm's episodes are more than twice as close to real camera frames as the simulator's own rendering. The farm's renderer also keeps all views consistent with the simulated motion, while calling a video model directly handles this less well.

How much it costs

Task adaptation is fully automatic and costs an average of $21 in API expenses, paid once per task, plus $5 for a rendering contract. Each subsequent episode (simulation, rendering, re-rendering, and review) costs an average of $0.80 and requires neither a robot nor an operator. At 100 episodes per task, the average comes to $1.06 per episode.

An hour of synchronized three-camera data for a task costs $107, including all of these expenses. For comparison, the quoted price of teleoperation in China is $117, and labor costs alone in North America come to $290. The code and project page are published, so the approach can be tested on your own tasks.