Researchers Test 16 Types of Hints for OPSD and Find a Selection Metric

On-policy self-distillation (OPSD) lets a language model learn from privileged information without a separate teacher model. A new paper featured on Hugging Face Daily Papers shows that the hint itself (PI) shapes the result as much as the algorithm does. The authors ran 16 PI configurations through three multi-turn agentic benchmarks and three student models that differ in scale and reasoning mode. The main practical takeaway: you can assess a hint's quality before training using KL metrics, and this approach is more reliable than directly testing how useful the hint is.

Why PI turned out to be the weak spot of OPSD

The setup is familiar. The model sees the task together with extra information that the "student" does not have at inference time, and its token distribution serves as the training signal. The student trains on its own samples, hence on-policy. No extra teacher is needed.

Until now, most papers have treated PI as a fixed design decision. Few have studied what to put into it and how to build it. In multi-turn agentic tasks the question is sharper: a trajectory consists of actions, observations, and reasoning, and there are dozens of ways to turn it into a hint. You can keep only the outcome, add the steps, or highlight the erroneous ones.

What the 16 configurations showed

PI effectiveness depends on three factors at once: its content, its generation source, and the student model itself. The general patterns are as follows:

• information-dense trajectories used as PI give larger gains on average;
• PI produced by a stronger generator is usually more useful;
• thinking models benefit from PI less consistently than models without an explicit reasoning mode;
• there is no universal hint recipe: the best configuration changes from task to task and from student to student.

The authors give no numerical gains in the abstract, so you will have to check the full text of the paper for the size of the effect.

How to evaluate a hint before training

Running a full training run to test every PI is expensive. So the authors compared a direct assessment of PI usefulness against two metrics based on KL divergence:

• sampled-token reverse KL (D): reverse KL over sampled tokens;
• error-step KL contrast (Δd⁻_err): a KL contrast tied to the steps where the agent makes mistakes.

Both metrics track final post-training quality more consistently than the direct usefulness assessment does. The latter has a weak spot: information leakage distorts it. If a hint effectively contains the answer, the model with that hint looks strong, but the student gains nothing from it.

The authors also tested a practical scenario. Choosing PI for each task by Δd⁻_err improves post-training results compared with random selection. The gain is largest where candidates differ strongly from one another. If all hint variants are roughly equal, the metric adds almost nothing.

OPSD plus GRPO

A separate check covered compatibility with RL. Effective PI stays useful when OPSD is combined with GRPO, and the combined objective outperforms a strong GRPO baseline. In other words, self-distillation does not compete with reinforcement learning but complements it.

What to take into practice

A team that trains agents through self-distillation should stop treating the PI format as a constant. A sensible order of work looks like this:

• build several PI variants from the same trajectories, from compressed to dense;
• where possible, generate hints with a stronger model;
• rank candidates by KL metrics, not by how well the model solves the task with the hint in context, otherwise leakage will mislead you;
• for thinking models, do not expect the same gain as for regular ones, and test them separately;
• if you already have a GRPO pipeline, add OPSD with the selected PI as a second term of the objective.

The limitation is also clear. The work is empirical, its conclusions come from three benchmarks and three students, and transfer to other domains needs to be verified. Still, as a guide to prompt engineering for self-distillation, the result is useful: it replaces intuition with a cheap metric that you can compute before launching expensive training.