STI-OPD: A Probabilistic Teacher Intervention for Multi-Step Agentic Tasks

Authors Junnan Liu, Linhao Luo, Zhijun Chen, Qianren Mao, Thuy-Trang Vu, and Gholamreza Haffari have introduced STI-OPD, a stochastic teacher-intervention framework for distillation-based multi-step agentic learning. The paper is registered on arXiv as 2610.10878 in the Computation and Language category.

The problem with on-policy distillation in agentic systems

On-policy distillation effectively transfers capabilities from a strong teacher to a weaker student through dense, token-level supervision. The approach has shown promise on complex tasks, including mathematical reasoning. In multi-step agentic systems, however, the student's decisions alter subsequent observations. An error made early on accumulates across the entire interaction trajectory. As a result, the student's trajectories drift far from the teacher's distribution — its supervision becomes unreliable or even harmful to learning, undermining the effectiveness of the whole method.

STI-OPD's architecture

The teacher's intervention is guided by the divergence between the student's and teacher's policies. During multi-step interaction, the system occasionally swaps the student's action for the teacher's action — this maximizes the accumulation of reliable supervision for training.

The key innovation is the stochastic intervention strategy. Instead of rigid threshold conditions or a fixed intervention schedule, the system: estimates policy divergence via KL divergence between the teacher's and student's action distributions; converts this divergence into an intervention probability; and probabilistically samples the intervention decision at each time step. This allows it to adaptively balance teacher control against the student's independent exploration, depending on the current state of divergence.

Training on mixed trajectories

The resulting trajectories contain a mix of actions from both the teacher and the student. To learn correctly from such mixed data, the authors introduce an importance-weighted reverse KL objective. This component corrects the token-sampling mismatch between responses generated by the teacher and the distribution of the student's policy. This preserves the original objective of on-policy distillation and avoids introducing bias into the training process.

Results and applicability

The experiments covered two categories of tasks: tool-integrated reasoning (tasks involving external tools) and long-horizon interaction (multi-turn interaction with an environment). Across every evaluated benchmark and every student model size, STI-OPD outperforms the strongest prior on-policy distillation baseline. Ablation studies confirm that both key components — divergence-guided intervention and the importance-weighted correction — contribute to the overall quality gain. The paper is marked as work in progress and is available in PDF, HTML, and TeX Source formats on arXiv.