Robots Can No Longer Bluff: Why Success Rate Is Not Enough

Imagine hiring a driver who tells you: "I reach the destination in 90% of cases." In robotics, that was long considered an excellent result. We got used to measuring progress by success rate, ignoring how that success was achieved. A robot could perform a series of clumsy, energy-hungry movements, nearly break its manipulator, and still touch the right button. Voilà: the coveted 1 goes into the report.

Chinese researchers have decided it is time to end this pretense of busy activity, and they have presented a new paradigm for evaluating embodied intelligence. This is not a cosmetic change to the metrics but a fundamental shift in how we understand "smart" machines.

The problem with the old approach was its binary nature: either victory or defeat. In the real world, outside clean laboratories, the price of that victory matters. The new evaluation methodology (jùshēn cāozuò), for embodied manipulation, introduces a multidimensional scale. It now accounts not only for the final result but also for the movement trajectory, the time spent and, most importantly, robustness to external disturbances. If you nudge the robot slightly or change the lighting, and its "success rate" drops from 90% to zero, there was no intelligence there. There was only a rigid program, optimized for one particular video for investors.

Why does this matter right now? We are on the threshold of humanoids and manipulators entering unstructured environments en masse: our homes and offices. Conditions there are not ideal. There are children, pets and a constantly shifting mess. The old metric (chénggōng lǜ) is useless when it comes to safety and predictability. The new paradigm forces developers to focus on generalization rather than memorizing specific scenarios. It is a harsh filter that will quickly separate startups peddling pretty renders from companies building real technology.

The move to complex metrics also changes the rules of model training. When a neural network is rewarded not simply for "reaching the goal" but for "reaching the goal efficiently and safely," its behavior changes. It comes to resemble a living creature that conserves energy and avoids unnecessary risks. That is what we call genuine embodied intelligence.

The researchers stress that abandoning the dictatorship of a single indicator will finally let the industry compare different approaches objectively. Previously, every lab boasted about its own figures, which could not be compared. Now a single ruler is emerging, and it is a fairly strict one.

For the market, this means a temporary slowdown in "paper" successes but a sharp acceleration in real progress. We will see fewer loud headlines about "robots that do everything" and more boring but important charts on robustness and control quality. This is the industry growing up. You can no longer simply record the hundredth take, in which the robot happened to complete the task, and pass it off as a breakthrough. The system must now prove its effectiveness in motion, under load and under uncertainty.

Bottom line: The era of marketing videos without editing seams is coming to an end. Robots will now have to prove their professional competence with numbers that cannot be faked by simple luck. Are today's market leaders ready for that level of transparency?