Lean4AgenticRL: Multi-Level Process Rewards Push LAR-27B to 77.3%

Long-horizon software engineering tasks — whether development, debugging, or terminal automation — require agents to learn at two levels: the final outcome and the intermediate actions. However, standard outcome rewards provide limited information about which specific steps were actually useful, making efficient reinforcement learning difficult. Amazon researchers have introduced Lean4AgenticRL — the first framework to ground multi-level process rewards in a unified Lean4 formalization scheme for open agent training. Using this framework, they trained the LAR-9B and LAR-27B models, which outperform comparable counterpart models and reach 77.3% on TerminalBench-2.1.

Three annotation levels instead of one

Existing approaches rely mostly on high-level rewards for the final outcome, largely ignoring information about low-level tool-call behavior and mid-level progress through workflow stages. Lean4AgenticRL addresses this by adding three interconnected reward levels:

• Outcome reward — evaluates the success of the task's final result • Workflow-stage reward — evaluates progress at each stage of plan execution • Tool-call reward — evaluates the correctness of calls to external tools and functions

Lean4 formalization as the basis for verification

The framework starts by building a problem-specific workflow — a solution plan that balances the agent's own capabilities against the controllability of its execution. This workflow is then formalized within a unified Lean4 scheme, and all three reward levels are annotated within the same system. This formalization makes it possible to verify the agent's behavior within a shared system, guaranteeing logical consistency of its actions.

Training relies on a modified GRPO algorithm with stage-wise normalization (workflow-stage-normalized GRPO). Instead of processing each step independently, the algorithm distributes credit across comparable stages from different execution attempts. This significantly improves training efficiency, since the agent receives more precise information about which solution phases and actions actually contribute to success.

LAR-9B and LAR-27B: benchmark results

Using Lean4AgenticRL, the researchers trained two model families:

• LAR-9B (a 9-billion-parameter model) • LAR-27B (a 27-billion-parameter model)

The test results are impressive:

• LAR-27B achieved 77.3% on TerminalBench-2.1 • LAR-27B achieved 84.2% on SWE-Bench-Verified • Both models in the LAR family outperform the baseline model by an average of 9.89% • LAR-27B outperforms the best known comparably sized counterpart by 5.07%

These results demonstrate the potential of integrating formal methods into open reinforcement learning for agents. Lean4AgenticRL shows that more granular process annotation and formal verification can substantially improve agent performance on complex engineering tasks.