RoboArchitect Lifts Manipulator Success from 9.7% to 43.3% on New Scenes Without Fine-Tuning

The authors of RoboArchitect have shown that the harness for a robotics policy does not have to be written by hand: a coding model can do it. The system takes a frozen vision-language-action (VLA) policy, looks at recordings of its failed runs, and rewrites the control code around it on its own. With no weight updates at all, average success on unfamiliar scenes rose from 9.7% to 43.3% (4.5x), and on unfamiliar tasks from 12.7% to 40.2% (3.2x). These figures come from held-out episodes that the harness did not see during optimization.

What exactly evolves

Here, the harness is a program that sits between the environment's observations and the low-level commands to the controller. At each step it receives camera frames, depth maps, the pose and gripper state, and a text instruction. The starting version simply passes the frozen policy's actions through unchanged, which is equivalent to evaluating the "bare" model.

Then a loop familiar to any engineer begins: run, find the failure, understand the cause, fix the code, check again. Each evaluation logs full traces: frame-by-frame images, robot readings, and the outputs of all models and tools. The coding model (the authors ran it through the Claude Code client at maximum effort level) reads the traces, diagnoses the failures, and proposes a new version. The candidate is re-checked on the same training episodes and accepted only if it beats the current version, so quality on the training set rises monotonically. The cap is 5 iterations.

The harness has helper tools available: Qwen3-30B-A3B-Instruct as the language model, Molmo2-8B for vision and pointing, Grounding-DINO for detection, SAM 3 for segmentation, and GraspGen for generating grasp poses. There are also geometry utilities that convert pixel-depth pairs into 3D coordinates in the robot's base frame.

What the system came up with

The strategies it found are quite human. If the policy keeps pushing against a juice carton until it tips over, the harness notices the jam and lifts the arm. When a shelf gets in the way from above, the gripper rotates and the arm approaches from the side. If a toaster door won't close because the push is too weak, the harness scales up the action amplitude and latches it shut. Each edit targets a specific kind of failure, so it is easy to read and build on, unlike changes to weights.

Where the policy provides no useful signal at all, the harness stops querying it and guides the arm using its own perception. In effect, the result is a code-as-policy agent, and the authors argue that it empirically unifies two approaches that are usually treated separately.

The harness also develops persistent memory on its own. The authors did not specify the schema or the storage rules: the system decides what to write down and how to retrieve it. As a result, the memory holds pragmatic details, such as an effective grasp depth, which are taken into account in later decisions.

Results and limitations

Evaluation ran on LIBERO-Pro-10 (LIBERO-10 with rearranged objects), SimplerEnv Google and WidowX, and VLABench. The policies used were MolmoAct-2 and X-VLA. For VLABench, the authors used the official progress score, which gives partial credit for grasping the right object and meeting conditions, although the paper also calls it success for simplicity. For comparison with fine-tuning, the authors took public checkpoints of the same models tuned on the target benchmark.

The gain is largest where the benchmark is far from the policy's training data: a generalist model in an evolved harness often outperforms a fine-tuned version of itself. The authors also built a single harness for different platforms (Franka Panda, WidowX, and Google's mobile manipulator). It was trained on procedurally generated episodes outside the standard evaluations. Around the generalist X-VLA, it noticeably improves results, and around fine-tuned specialists, it preserves their quality.

The caveats are clear: all experiments were run in simulators, and testing on real hardware is not described in the available text. The result depends heavily on a powerful coding model and a set of ready-made tools, and the material gives no data on the number of iterations needed or the cost of the runs. The practical takeaway for teams adapting policies to a new arm or scene: instead of collecting demonstrations and fine-tuning, it is worth trying automated harness refinement first. The authors claim that this is where a significant share of today's headroom for manipulation lies.