AA-Robotics v0.1: GPT-6 Astra controls a robot arm and scores 69 out of 100

Artificial Analysis has released AA-Robotics v0.1, a research preview of a benchmark that tests how well frontier language models understand a physical scene, plan, and act. Each model gets a six-axis arm (five joints and a gripper) and a simple goal. GPT-6 Astra leads with 69.4 out of 100. It is followed by GPT-6.1 Sol (64.6), Gemini 4 Argon (55.3), Claude Opus 5.5 (48.8) and Claude Fable 5.1 (46.3). But the average task takes Astra 30 minutes and $16.36, while a human finishes in 20 seconds.

How the test works

The models work in an isolated container with a short instruction. They are told how to reach the arm through a simple HTTP API, that it is a Hiwonder NexArm (with no additional specifications), and how to submit an attempt. They see the scene through four cameras: three stand around the table, and one is mounted on the wrist. The model writes the code for planning and sending movements itself. All attempts are zero-shot, and a human judges the final state: full completion earns 100 points, and partial completion earns points for progress. GPT models ran in Codex, Claude in Claude Code, and Gemini 4 Argon in Antigravity, all with a high reasoning effort level.

v0.1 includes eight tabletop tasks:

• press a red button;
• place a toy crocodile in a rectangle;
• pour a drink from a bottle into a cup;
• put four blocks into a box and close the lid;
• assemble shapes to match a picture;
• write hello world with a marker inside a frame;
• build the tallest tower from foam blocks;
• measure the length, width and height of a box to within 2 mm.

All five models scored 100 on the button and the crocodile. Nobody managed to measure the box: the maximum was 17 points, for Sol and Opus 5.5, and the rest scored zero.

Time, money and strange errors

Four of the five models take roughly 30 minutes per task. Gemini 4 Argon is the slowest at 52 minutes, of which an average of 21 minutes goes to observing and thinking before the first action. The spread across tasks is huge. On the crocodile, GPT-6.1 Sol declared it was done in five minutes, while Claude Opus 5.5 spent 45, nine times as long. The fastest of the best attempts belongs to Astra: it pressed the button in 2.2 minutes.

On cost, the gap with humans is smaller. The most economical model, Sol, costs $1.93 per task. For a human at an average hourly rate of $37.81, the same task would cost roughly ten times less, but the time gap is nearly a hundredfold. Astra costs more than eight times as much as Sol for an advantage of just four points (69 versus 65). About 70% of the spending goes to input context: the models have to process a large number of images.

The main weakness is judging their own results. Sol built a passable tower in about 17 minutes, but kept acting and then failed the task. Another model got the crocodile into the target position by minute 23, then spent 22 minutes moving it and knocked it off the edge three times. In the lid task, a model declared the lid flat, although it was resting on a wedge on top of a cube. There are also physics errors: the arm bumps the tower with its forearm, drags the crocodile by its tail, and the stream of liquid pushes the cup four centimeters.

How to read the results

The most honest accuracy metric is the share of successful grasps. For GPT-6 Astra it is about 45% (n = 95), for Claude Fable 5.1 it is 44%, for Opus 5.5 it is 40%, for Gemini 4 Argon it is 36%, and for GPT-6.1 Sol it is 28%. The best models miss or drop the object in more than half of their attempts.

The practical takeaway for anyone building robotics agents: an LLM paired with a coding agent can already control a manipulator on simple tasks, but it is not suitable as a standalone controller in production. The models are too slow and expensive, and their stopping criterion is unreliable. It makes more sense to use them as planners and put an independent check of the result next to them. The authors expect rapid progress, and it is worth watching how the models' grasp success rate and time to first action change.