Prefix-GRPO: Training Small Language Model Agents on Trajectory Prefixes

Researchers have proposed Prefix-GRPO, a reinforcement learning method published on arXiv (cs.LG) in July 2026. It helps small language-model agents learn from the trajectories of a strong teacher model without reducing them to one-shot imitation. On three environments — TextCraft, BabyAI and ALFWorld — it outperforms both classic distillation and standard RL.

Why distillation stalls on long-horizon tasks

Direct distillation collapses the teacher's rich multi-turn behavior into single-step imitation targets. That is inefficient where early decisions determine future states and rewards. Small models are attractive as the basis for interactive agents: they are cheap and fast. But when an agent has to work through a long chain of actions (collect an item, reach a goal, follow an instruction), training that amounts to "repeat the teacher's answer" loses the causal structure. The model memorizes the final turns but does not learn how the teacher reached the right state.

How Prefix-GRPO works

Prefix-GRPO splits each teacher trajectory into two components: replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to restore a valid intermediate state. The student then continues the interaction on its own and receives a task reward.

The key difference from response-only GRPO is that the method also applies clipped policy updates to the historical assistant tokens inside the replayed prefix. A separate SFT checkpoint, obtained through policy distillation, estimates the old log-probabilities of those tokens. This merges prefix learning and continuation learning into a single policy-optimization form.

• Method: Prefix-GRPO, reinforcement learning built on top of GRPO
• Idea: teacher trajectory = replay prefix + online continuation
• Difference: clipped updates for assistant tokens inside the prefix as well
• Old log-probabilities are estimated via a policy-distilled SFT checkpoint
• Test environments: TextCraft, BabyAI, ALFWorld (3 benchmarks)

What the experiments showed

On all three environments, Prefix-GRPO improved small-model agents compared with both distillation and standard RL baselines. A separate series of ablation experiments tested what actually drives the gain: prefix replay alone proved insufficient without explicit optimization of the tokens inside the prefixes.

Prefix replay alone is not enough without explicit optimization of the

tokens inside the prefix.

— wording from the study's abstract on arXiv. In other words, restoring the intermediate environment state is not enough: the policy must also be updated on the historical tokens, or the method does not deliver its full effect. According to the authors, the implementation and scripts for reproducing the results are publicly available on GitHub.

What this means

Prefix-GRPO is an attempt to transfer part of the "intelligence" of large teacher models into compact models without losing the multi-turn logic of their behavior. If the approach scales beyond game environments, cheap local agents could handle long tasks noticeably better than they do with ordinary distillation.

Frequently asked questions

What is Prefix-GRPO?

It is a reinforcement learning framework for small language-model agents. It splits a teacher model's trajectories into prefixes, replays them in the environment to restore the intermediate state, and fine-tunes the student through online interaction with a task reward.

How does Prefix-GRPO differ from standard GRPO?

Unlike response-only GRPO, which updates the policy only on new responses, Prefix-GRPO also applies clipped updates to the historical assistant tokens inside the replayed prefix. It estimates their old log-probabilities through a separate policy-distilled SFT checkpoint.

Where was the method tested?

The authors tested Prefix-GRPO on three agent environments — TextCraft, BabyAI and ALFWorld — where it outperformed distillation and standard RL baselines. The code and reproduction scripts are available on GitHub.