A coding agent replaces the teleoperator: 93 successes out of 100 versus 84 for humans

To teach a robot to open a drawer, say, a human operator with a controller demonstrates the right movements many times, and after each failure manually resets the objects. Every such episode costs operator time and robot time, and a new task or a new scene means starting data collection from zero. The DAGroup team at Peking University has proposed PyRUA-DataFarm (a "data farm"), a system in which a coding agent replaces the operator: from one short scene recording, it assembles the task in a simulator on its own and produces labeled videos for training.

On five real-world tasks, a robot trained only on this data succeeded in 93 of 100 attempts, while one trained on the same number of teleoperation episodes succeeded in 84. For teams that collect data for robot manipulators, it is a way to cut operator costs and the cost of reconfiguring for new tasks.

How the agent stands in for the operator

The input is just two things: a text description of the task with a success condition, and one calibration episode, a short multi-camera recording of the robot in the target scene. The actions in this recording are not copied. It is used to work out where the cameras are, which objects are in the scene, and how big they are.

From there, the agent works like an engineer. It reconstructs the scene in the simulator, checks that the simulator can actually perform the required skills (for example, pulling a drawer by its handle), and writes three things: a scene description, an expert program that solves the task, and a success check. The expert is run from many starting positions, and the simulator outputs exact actions, object states, and contacts.

That leaves the problem of appearance: a simulation looks like a simulation, and a robot trained on it stumbles in the real world. So every run is passed to a video model, which receives real frames from the calibration recording. It renders photorealistic video for all cameras at once, preserving the arm pose, objects, and contacts from the simulation. A new tablecloth or colored lighting is specified with an ordinary text prompt.

Quality control is handled separately. An agent that writes both the expert and the success check can approve its own mistakes. So the expert is developed against a judge whose code it cannot see, and is accepted only on held-out starting states. The video model is allowed to change appearance, but not geometry, objects, or timing. Automated checks and an independent reviewer compare each finished episode against the simulation, and only then does it enter the dataset.

What the experiments showed

The tests covered five real-world bimanual tasks involving rigid, articulated, and soft objects, including cloth and a screen. Robots trained only on the farm's data achieved 93 successes in 100 attempts. Robots trained on the same number of teleoperated episodes achieved 84.

The gap widens when conditions change. When an unfamiliar tablecloth was placed on the table and colored lighting was switched on, the farm-trained policies kept 87 successes, while those trained on teleoperation dropped to 46. The authors attribute this to the farm adding appearance diversity around fixed, precise trajectories.

Real frames in the video model turned out to be critical: the same trajectories with ordinary randomization of textures and lighting produced only 26 and 21 successes. By image statistics, the farm's episodes are more than twice as close to real frames as the simulator's own render.

What it costs

Adapting to a new task costs an average of $21 in API fees, paid once per task, plus $5 for a rendering contract. Each subsequent episode costs an average of $0.80 for simulation, rendering, re-rendering, and verification, with no robot and no operator. Over 100 episodes, the average comes to $1.06 per episode. An hour of synchronized three-camera data works out to $107, with all of these costs included. For comparison, a quote for teleoperation in China is $117 per hour, and labor costs alone in North America run $290.

Limitations and takeaway

The difference from Chinese teleoperation is small: the main savings will show up at large volumes and with frequent task changes, when the one-time $26 setup cost is spread across many episodes. The authors calculated the prices themselves, and there is no independent audit. Only five tasks were tested, and everything depends on the simulator being able to reproduce the required skills: for contact-rich operations with complex physics, this may not work. This is a preprint, and no one has yet reproduced the results.

The practical takeaway: the pipeline of "the simulator supplies the actions, the video model supplies the appearance, and an independent reviewer admits the data" is already comparable to teleoperation in quality and holds up better when the scene changes. The code is on GitHub in the PyRUA-Lean repository, DataFarm branch.