ProAS speeds up LLM agent training by 2.44x, pushes ALFWorld to 94.3%

The LINs-lab team has published ProAS, short for Progress-Aware Advantage Shaping: a plug-in add-on for reinforcement learning-based training of LLM agents. The method requires no separate critic model and pushes the results of group-based baseline algorithms to a new level. On Qwen2.5-7B-Instruct, the agent solves 94.3% of tasks in ALFWorld and 70.8% in WebShop, while the 1.5-billion-parameter version reaches 91.9% and 66.7%. Training also converges up to 2.44x faster.

The problem: one score for the entire trajectory

RL has become the main way to fine-tune LLM agents. The most popular critic-free algorithms, GRPO among them, are cheap because they don't need to maintain a second large network for evaluating states. The cost of that savings is that the advantage is calculated at the level of the whole trajectory and distributed equally across every action within it.

The ProAS authors show what that leads to in multi-step tasks, where an agent takes dozens of actions.

• In a successful trajectory, everything gets rewarded, including structurally meaningless steps: redundant transitions, repeated checks, loops. The model reinforces them as "correct" simply because the outcome was successful.
• In a failed trajectory, all actions are penalized equally, even though some steps nearly reached the goal while others took the agent far off course. The signal doesn't distinguish between a mistake on step two and a miss on the last step.

In long agentic scenarios, this slows down training and drags down final success rates.

How ProAS redistributes reward

Instead of a critic, the method uses an Agentic Reference Graph (ARG), a dynamically evolving reference graph. It allows each action's progress to be evaluated relative to the task. Two mechanisms are built on top of that evaluation:

• Masking Unreasonable Incentives (MUI) masks the positive signal for actions that don't make sense, even if they belong to a "good" trajectory.
• Asymmetric Penalty Redistribution (APR) redistributes the penalty in bad trajectories depending on how far the remaining path deviates from the correct one. Steps that drag the agent further from the goal receive a larger penalty.

This is a plug-and-play scheme: it calibrates the advantage calculation within existing group-based algorithms without changing the rest of the training pipeline.

The numbers and robustness

Experiments were run on two models from the Qwen2.5-Instruct family, on the ALFWorld and WebShop benchmarks:

• Qwen2.5-1.5B-Instruct: 91.9% success on ALFWorld, 66.7% on WebShop.
• Qwen2.5-7B-Instruct: 94.3% on ALFWorld, 70.8% on WebShop.
• Training convergence speedup: up to 2.44x.
• The results were obtained on top of several common group-based baseline methods, and the authors claim ProAS lifts all of them to a new state-of-the-art.

The team also separately tested robustness to the quality of the reference graph. The method retains its effectiveness with both imperfect and distilled ARGs. That matters because obtaining a perfect graph for real agentic tasks is difficult, and working well with approximate versions lowers the barrier to adoption.

Who benefits from this in practice

Teams training agents with GRPO-style schemes typically run into compute costs and unstable signal over long trajectories. ProAS addresses both problems at once: no critic is added, and a speedup of up to 2.44x means fewer GPU-hours for the same result. The method suits teams building agents for environment navigation, web scenarios, and multi-step automation.

The code is published in the LINs-lab/ProAS repository on GitHub, so the hypothesis can be tested on your own tasks. One caveat: the numbers come from two benchmarks and models up to 7 billion parameters. How the method performs on larger models and in less structured environments remains to be shown by independent reproductions. But the idea of evaluating each step's contribution without a critic looks like a workable way to make agentic RL cheaper and more precise.