ProAS speeds up LLM agent training 2.44x and lifts ALFWorld success to 94.3%

The LINs-lab team has published ProAS, Progress-Aware Advantage Shaping: a plug-in add-on for reinforcement learning of LLM agents. The method needs no separate critic model and lifts the results of group-based baseline algorithms to a new level. On Qwen2.5-7B-Instruct, the agent solves 94.3% of tasks in ALFWorld and 70.8% in WebShop, while the 1.5B-parameter version reaches 91.9% and 66.7%. Training converges up to 2.44x faster.

The problem: one score for the whole trajectory

RL has become the main way to fine-tune LLM agents. The most popular critic-free algorithms, GRPO among them, are cheap because they do not keep a second large network for evaluating states. The price of that saving is that the advantage is computed at the level of the whole trajectory and split equally among all actions in it.

The ProAS authors show what this does in multi-step tasks, where an agent takes dozens of actions.

• In a successful trajectory, everything gets rewarded, including structurally meaningless steps: redundant moves, repeated checks, loops. The model reinforces them as "correct" only because the outcome was a success.
• In a failed trajectory, every action is penalized equally, even though some steps nearly reached the goal and others took the agent far away from it. The signal cannot tell a mistake on step two from a miss on the last step.

In long agentic scenarios, this slows training and drags down final success.

How ProAS redistributes the reward

Instead of a critic, the method uses the Agentic Reference Graph (ARG), a dynamically evolving reference graph. It makes it possible to evaluate each action's progress relative to the task. Two mechanisms are built on that evaluation:

• Masking Unreasonable Incentives (MUI) masks the positive signal for actions that make no sense, even if they sit inside a "good" trajectory.
• Asymmetric Penalty Redistribution (APR) redistributes the penalty in bad trajectories according to how far the remaining path deviates from the desired one. Steps that pull the agent further from the goal receive a larger penalty.

This is a plug-and-play scheme: it calibrates the advantage calculation in existing group-based algorithms and leaves the rest of the training pipeline unchanged.

Numbers and robustness

The experiments ran on two models from the Qwen2.5-Instruct family, on the ALFWorld and WebShop benchmarks:

• Qwen2.5-1.5B-Instruct: 91.9% success on ALFWorld, 66.7% on WebShop.
• Qwen2.5-7B-Instruct: 94.3% on ALFWorld, 70.8% on WebShop.
• Training convergence speedup: up to 2.44x.
• The results were obtained on top of several common group-based baselines, and the authors claim that ProAS lifts all of them to a new state of the art.

The authors also tested robustness to the quality of the reference graph. The method stays effective with both imperfect and distilled ARGs. This matters because a perfect graph is hard to obtain for real agentic tasks, and the fact that it works with approximate versions lowers the barrier to entry.

Who can use this in practice

Teams that train agents with GRPO-style schemes usually run into compute cost and signal instability on long trajectories. ProAS hits both problems: it adds no critic, and a speedup of up to 2.44x means fewer GPU-hours for the same result. The method suits those building agents for environment navigation, web scenarios and multi-step automation.

The code is published in the LINs-lab/ProAS repository on GitHub, so the hypothesis can be tested on your own tasks. There is one caveat: the numbers come from two benchmarks and from models of up to 7B parameters. How the method behaves on larger models and in less structured environments will be shown by independent reproductions. Still, the idea of evaluating each step's contribution without a critic looks like a workable way to make agentic RL cheaper and more precise.