Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents: 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →

The Best Coding Agent Controls a Robot in 77% of Tasks, but With Sensors Alone Only 30%

Coding agents like Codex write code confidently where you can inspect any variable and restart the process. A robot is different: you can't rewind a dropped part. The Embodied First Exam (EFE) benchmark gave agents 1156 tasks across 47 simulated robots. The best agent, GPT-6 Astra in Codex CLI, solved 77% of the hard tasks with access to simulator state and unlimited resets, but only 30% with sensors alone and a limited number of resets. The takeaway: agents need a harness with safe recovery.

AI-processed from TG: @gonzo_ML (gonzo-обзоры ML статей); edited by Hamidun News
The Best Coding Agent Controls a Robot in 77% of Tasks, but With Sensors Alone Only 30%
Source: TG: @gonzo_ML (gonzo-обзоры ML статей). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @gonzo_ML (gonzo-обзоры ML статей) ↗
Editor-in-Chief
Key metric #1Agents issued actions directly, wrote controllers, or trained policies
Precise contact is especially hardit takes repeated attempts
Key metric #3All figures come from single attempts and simulation only; the code is open on GitHub

Coding agents like Codex or Claude Code write and fix programs with confidence, so it is natural to want to trust them with controlling a physical robot. But in software, a developer can inspect any variable and, if something breaks, simply restart the process. With a robot, you can't rewind a dropped part, and the only thing it sees is what its own sensors show.

The new Embodied First Exam (EFE) benchmark measured the cost of that difference. The best agent tested, GPT-6 Astra in OpenAI's Codex CLI, solves 77% of hard tasks when it has the full simulator state and unlimited resets. When only onboard sensors and a limited number of resets remain, it solves just 30% of the same tasks.

That is a drop of 47 percentage points.

What Was Tested

EFE is a containerized benchmark of 1156 tasks across 47 simulated robots. The agents are used as they are, with no robotics-specific tuning, and are given three routes: issue commands to the robot directly, write a controller for it, or train a control policy. The benchmark compared the same hard tasks in two modes: "developer mode" and "real-robot mode."

A large team led by Manling Li worked on the project; co-authors include Fei-Fei Li, Pieter Abbeel, Eric Xing and Dawn Song. The benchmark code is published on GitHub, so you can run your own agent on it.

Where the Agent Breaks

Imagine plugging a plug into a socket. In developer mode, you have an overhead camera, the precise coordinates of every object, and an unlimited "undo" button. In real-robot mode, you see only through the arm's own eyes, and you get few attempts, because after a failure something may have fallen or shifted. That is exactly how the two modes differ in EFE.

In the first mode, GPT-6 Astra handles 77% of the matched hard tasks. In the second, it handles 30%. The authors attribute this primarily to stages that require precise contact: finding the right force and position takes several rounds of trial and error. When attempts are scarce and a mistake is irreversible, the agent can't fall back on its usual "write, run, fix" loop, the way it does with code.

The main point is that for coding, familiar privileges (a full view of state and cheap restarts) are implicitly built into how agents work. The physical world takes them away.

What to Do in Practice

The author of a write-up on the gonzo_ML channel calls harnesses (execution scaffolding) for physical robots one of the next big topics. The conclusion from EFE is indeed that one strong model is not enough. You need a layer that decides which actions are safe, checks results against sensors, and returns the system to a working state after a failure.

For teams building robots on top of LLM agents, this suggests a few simple rules:

  • do not evaluate an agent only in simulation with full state: such a metric greatly overstates real-world chances (77% vs. 30% in EFE);
  • build in safe recovery and rollback rather than counting on a restart;
  • test stages with precise contact separately, because that is where success drops the most;
  • treat the number of available attempts as part of the task conditions: the fewer the resets, the lower the result.

Limitations

The authors note that all scores in the benchmark come from single attempts and only in simulation. The variance between runs remains unknown, and transfer to real manipulators will bring its own difficulties: friction, sensor noise, and part wear. So 30% is not a forecast for a live robot. It is a gauge of how hard the "sensors only and limited resets" mode hits an agent that performs very well in developer mode.

The practical lesson for product managers and founders: if you promise an "agent that controls hardware," test it under conditions as close to production as possible. The 47-percentage-point gap between a convenient simulation and honest sensors is the zone where it is being decided today whether such a product will work.

Story

Key Questions Answered

What Was Tested?

EFE is a containerized benchmark of 1156 tasks across 47 simulated robots. The agents are used as they are, with no robotics-specific tuning, and are given three routes: issue commands to the robot directly, write a controller for it, or train a control policy. The benchmark compared the same hard tasks in two modes: "developer mode" and "real-robot mode." A large team led by Manling Li worked on the project; co-authors include Fei-Fei Li, Pieter Abbeel, Eric Xing and Dawn Song.

Where the Agent Breaks?

Imagine plugging a plug into a socket. In developer mode, you have an overhead camera, the precise coordinates of every object, and an unlimited "undo" button. In real-robot mode, you see only through the arm's own eyes, and you get few attempts, because after a failure something may have fallen or shifted. That is exactly how the two modes differ in EFE. In the first mode, GPT-6 Astra handles 77% of the matched hard tasks. In the second, it handles 30%.

What to Do in Practice?

The author of a write-up on the gonzo_ML channel calls harnesses (execution scaffolding) for physical robots one of the next big topics. The conclusion from EFE is indeed that one strong model is not enough. You need a layer that decides which actions are safe, checks results against sensors, and returns the system to a working state after a failure.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…