CodeActionBench: GPT-6 Astra in Codex CLI Hits 73.3% Success on Two-Handed Tasks

The CodeActionBench benchmark, reported by Jiqizhixin, asks a simple question: if an agent can look at an image and write code, can it control a robot? In the two-handed simulation, the best named result belongs to GPT-6 Astra running through Codex CLI. Its attempt success rate was 73.3%. Claude Opus 5 was tested in the Reference harness, but in the available portion of the publication its result cuts off, so the models cannot yet be compared by the numbers.

What the benchmark actually measures

Most robotics tests check individual links in the chain: scene recognition, planning, or low-level control. CodeActionBench combines the whole chain in a single run. The agent receives a visual observation, builds an action policy (in this case, as code), executes it in a physical environment, and decides for itself whether the task is done.

The last step matters more than it seems. An agent that confidently claims success after a failure is more dangerous on a real production line than one that honestly reports a fault. By building completion assessment into the loop, the authors test the agent's self-monitoring, not just its motor skills.

The second feature is uniform conditions. General-purpose agents like Codex CLI come with their own harness: a set of tools, a prompt format, rules for working with files. Unless those parameters are aligned, the comparison measures not the model but the packaging around it. The source's mention of a Reference harness suggests the authors have a reference harness for this kind of alignment.

What the published data shows

• Benchmark: CodeActionBench, an evaluation of general-purpose multimodal agents in robotic manipulation.
• Loop: observation, policy generation, physical execution, judgment of completion.
• Environment: two-handed (bimanual) simulation.
• Best named result: GPT-6 Astra in Codex CLI, 73.3% attempt success.
• Second participant: Claude Opus 5 in the Reference harness; the available text gives no figures.
• Not disclosed: the number of tasks, the number of attempts per task, the full list of models, the spread of results.

How to read the 73.3% figure

The metric is called attempt success, and it is not the same as the share of tasks solved. If an agent is allowed several attempts, the final share of solved tasks is usually higher than the success rate of a single attempt. Once the authors publish the full table, it will be worth looking at both numbers, and at how the agent uses feedback between attempts.

Simulation offers reproducibility and cheap runs, but it does not close the gap with the real world. Friction, backlash, sensor noise, and imperfect grasps on real manipulators often break policies that work cleanly in a virtual environment. So 73.3% shows the ceiling of the agent's capabilities under controlled conditions, not readiness for the factory floor.

Still, the approach itself deserves attention. Code as an interface to a robot is convenient to verify: a policy can be read, debugged, and replayed. That is more transparent than an end-to-end neural policy, where you have to guess the cause of an error.

What practitioners should check

Teams evaluating general-purpose agents for robotics should check three things. First, wait for the full methodology: the number of tasks, the retry rules, and the harness configuration. Second, compare models in the same harness rather than by figures claimed in different sources. Third, separately measure the accuracy of completion self-assessment: how often the agent declares success when it has actually failed. That number will determine whether such a system can be left without human supervision.