STI-OPD: Teacher Probabilistically Swaps in Its Own Actions and Beats OPD Baselines

The authors of arXiv preprint 2610.10878 propose STI-OPD, a variant of on-policy distillation for agents that work over multiple turns. With some probability, the teacher replaces the student's action with its own, and that probability is computed from the divergence between the two policies. According to the authors, the method outperforms the strongest previous OPD baseline on every evaluated benchmark and at every student size. The abstract gives no specific percentage gains; those have to be looked up in the full paper.

Where on-policy distillation breaks down

OPD is simple. The student generates its own trajectories, and the teacher provides a dense signal at every token: how far its distribution differs from the student's. This approach has already worked well on hard tasks such as mathematical reasoning, where the whole generation fits into a single long answer.

Agentic scenarios are harder. The student picks an action, the environment responds with an observation, and that observation determines the next step. An early mistake drags later steps down with it. After a few turns, the trajectory ends up in a region the teacher itself would never have reached. The teacher's token-level labels on such states become less reliable and sometimes harmful: the student is taught to fix a situation that simply does not exist in a normal run of the task.

How STI-OPD works

During multi-step interaction, the method measures the divergence between the teacher and student policies via KL divergence and converts it into an intervention probability. At each step, it then randomly decides whether to keep the student's action or replace it with one generated by the teacher. The further the trajectory drifts from the teacher, the higher the chance that the teacher takes control and pulls it back into the zone of reliable signal.

The authors contrast this with earlier schemes that use a hard divergence threshold or a fixed intervention schedule. In their words, the probabilistic variant strikes a better balance between teacher control and student exploration: the student keeps its own steps but does not stray too far.

Mixed trajectories carry a technical cost. Some of the tokens were written by the teacher and do not match the student's distribution, so the standard OPD objective becomes biased. To correct for this, the authors introduce Importance-Weighted Reverse KL. It accounts for the gap between the teacher's token sampling and the student's policy, and it preserves the original OPD objective.

Key elements of the method:

• intervention is driven by the KL divergence between the teacher and student policies;
• the divergence is mapped to a probability, and the decision is made by sampling;
• action replacement happens at the level of a single step in a multi-turn interaction;
• training on mixed trajectories uses Importance-Weighted Reverse KL;
• the evaluation covers tool-integrated reasoning and long-horizon interactions.

What the experiments showed

According to the abstract, STI-OPD outperforms the strongest previous OPD baseline on every benchmark and at every student model size. Ablations show separately that both components contribute: divergence-guided intervention and importance weighting. Remove either one and the gain shrinks.

The limitations are visible too. The method requires a teacher that is available during training and can generate actions on demand, which means paying the inference cost of a strong model at every step. The results so far appear only in the preprint, there are no independent reproductions, and the announcement does not include the full table of metrics.

Who can use this

Teams that distill large models into compact agents for working with tools, code, or long dialogues get a concrete recipe against trajectory drift. You can test it on your own pipeline: it is enough to log the KL between the policies at each step and compare threshold-based intervention with the probabilistic one. If your teacher is expensive to run, it makes sense to estimate up front what share of steps will end up using its actions. Add the corrective importance weight from the start; otherwise, swapping in teacher tokens can easily break the match with the original OPD objective.