PhysRSI-eval Measures How VLM Agents Learn from Mistakes Across 18 Manipulation Tasks

The authors of a paper featured on Hugging Face Daily Papers have proposed PhysRSI-eval, a set of metrics for evaluating physical recursive self-improvement (RSI) in VLM-based agents. The idea is simple: give a model 18 manipulation tasks, cap the number of interactions with the world, and watch how the agent learns from its own failures. The conclusion is awkward for conventional leaderboards. Two systems with nearly identical average scores on public benchmarks can behave completely differently in a physical loop.

Five axes instead of one score

A typical benchmark answers the question "did the model succeed?" PhysRSI-eval asks how fast and how far an agent can improve. To do that, it breaks the metric into five components:

• First-Attempt Performance: the result of the first attempt, before any feedback;
• Efficiency: how economically the agent spends attempts on the way to success;
• Success: the final share of tasks solved;
• Budget: how many environment interactions were needed out of the fixed limit;
• Improvement from failure: the gain in quality after failed rounds.

The practical point of the split is that a high final Success does not have to come with early or efficient improvement. An agent can reach a good result by burning through the entire budget, or it can squeeze out a decent result almost immediately and then stop. For a real robot, where every attempt costs time, wear, and sometimes broken objects, these are different risk profiles.

The barrel hypothesis and predicting from early attempts

Based on their experiments, the authors formulate the Physical Barrel Hypothesis. The name refers to Liebig's barrel: the water level is set by the shortest stave. Here the staves are the model's capabilities, and different ones limit different parts of improvement. One governs the quality of the first attempt, another the ability to learn a lesson from a failure.

The paper also studies observational scaling from two directions. The first task is to predict the relative performance of VLM agents from their public capabilities. The second is to forecast late progress from early physical feedback. Together, they address how quickly and how far an agent can improve. The abstract gives no specific coefficients or dependency parameters, so the accuracy of the forecasts can only be judged from the full text of the paper.

The second part of the study looks at interventions in the agent's workspace, that is, in what remains after rounds of improvement. There are three results:

• removing history lowers success in later rounds, meaning the accumulated context genuinely does work;
• frozen workspaces remain useful on unfamiliar scene layouts;
• in three cross-task cases, success comes earlier.

The authors draw a cautious conclusion: adaptation to a specific scene and reusable experience can coexist. In other words, what the agent accumulates does not necessarily "stick" to a single table layout.

What teams should check in practice

For those building robots or agents on VLMs, the work offers several practical pointers. First, picking a model by its average public score is risky: evaluate it by its five-axis profile on your own tasks and with a realistic attempt budget. Second, if costly failures are critical for you, look at First-Attempt Performance and Efficiency, not just final Success. Third, keep the history and workspace between rounds: according to the authors, results are worse in later rounds without history, and a frozen workspace helps on new layouts.

The limitations are also clear. Eighteen manipulation tasks make for a narrow sample, and the barrel hypothesis so far looks like an observation, not a proven law. But the framing itself is useful: it shifts the conversation about agents from "how smart is the model" to "how does it behave when it gets things wrong."