Researchers Propose PAPAV Framework: Only 3 of 64 Benchmarks Test Verification

When a multimodal agent moves from controlling a browser to controlling a robotic arm, the environment changes, and so does the cost of a mistake. A new paper from Hugging Face Daily Papers tries to describe this transition in a single shared language. The authors introduce the concept of multimodal embodied agents (MMEA) and propose PAPAV, a framework of five functions. The main empirical finding: of the 64 benchmarks reviewed, only 5 evaluate the ability to anticipate the consequences of actions, and just 3 check whether the result was actually achieved. Almost all the rest measure whether the task was completed.

Three classes of systems and a common denominator

The authors start by separating the categories. Multimodal agents (MMA) conduct interaction through reasoning, memory, tools, and feedback. The most visible example is agents that operate a computer. Robotic systems (RS) tie perception to execution under physical dynamics. MMEA sit at the intersection: they are goal-directed systems that connect multimodal reasoning about a task with physical action and revise their decisions based on the outcome.

The central question of the paper is simple: what changes when an agent moves from computer-use to robot-use? Answering it requires a comparison scale that is not tied to a specific architecture. The authors choose functions rather than model design.

PAPAV: five functions instead of a single metric

The acronym breaks down as follows:

• Perceive: sense the current state;
• Anticipate: predict the effects of actions;
• Plan: build a feasible plan;
• Act: act through an interface or a body;
• Verify: check the outcome.

This definition makes it possible to identify what all three classes have in common, and then compare MMEA with ordinary MMA and with classical robotics. An agent clicking buttons and an industrial robotic arm on a factory floor both perform all five functions. What differs is how hard each function is and how costly a failure becomes.

Where physics takes away freedom

The authors identify five physical constraints. The list runs from partial observability to unverifiable outcomes. The overall effect is the same: in the physical world, fewer decisions can be fixed in advance, and there are also fewer ways to undo a mistake. A wrong click in an interface can usually be rolled back. A dropped object or a wrong movement cannot.

This is why Anticipate and Verify carry so much weight. An agent that cannot assess consequences in advance and then honestly verify the result is dangerous in a physical environment, even if its planning is good.

What the analysis of 64 benchmarks showed

The paper's key figure describes the state of evaluation in the field:

• 64 benchmarks were analyzed in total;
• explicit evaluation is concentrated on the Act function;
• only 5 benchmarks evaluate Anticipate;
• only 3 evaluate Verify;
• both functions are mostly hidden behind the overall task success score.

The problem is that task success blends all five abilities into one number. A model may have completed a task thanks to good perception rather than good prediction. Or it may have failed because of weak verification even though the plan was sound. Without separate measurements, it is hard to tell what actually improved between system versions and what needs fixing.

The authors build a list of open problems on these same constraints. They include designing reliable physical agents and measuring progress across all five abilities.

How to use the framework in practice

The paper has practical value for teams building agents for robots. PAPAV can serve as an audit checklist: for each function, identify which test measures it and find the ones left uncovered. If a pipeline has no separate check of the result after an action, that is a reason to add one before scaling up.

For benchmark developers, the takeaway is even more concrete. They need tasks that evaluate outcome prediction and outcome verification separately, not just final success. For investors and customers, it offers a reference point for reading demos: an impressive task-completion rate does not yet show how a system behaves when the outcome cannot be verified or the error cannot be undone.

The paper does not propose a new model and does not promise a gain on any specific test. It is an attempt to bring order to the terminology and metrics at the intersection of agents and robotics, while the boundary between them is rapidly disappearing.