OPMD Teaches Student LLMs Not Just the Teacher's Answers, but Its Internal Computations

A team of researchers has proposed OPMD (On-Policy Mechanism Distillation), a language model distillation method that transfers not only predictions but also intermediate computations from teacher to student. The authors claim that OPMD achieved the highest average accuracy among the evaluated distillation baselines in all three teacher–student configurations. It also carries over to multi-teacher distillation and to supervised fine-tuning across different model families. The work is published on Hugging Face Daily Papers. The abstract contains no absolute metric values, model names or benchmarks, so the size of the gain cannot yet be assessed.

Where on-policy distillation hits its limit

On-policy distillation (OPD) works like this: the student generates its own prefixes, and the teacher scores what it would have said at each step. The student thus learns from its own state distribution rather than from someone else's reference texts, which reduces the mismatch between training and inference. The approach has a limitation: supervision stays in the output space. The student learns which tokens the teacher predicts, but it does not see how the teacher arrived at them.

The obvious move is to make the student's hidden states resemble the teacher's. The authors consider this insufficient for two reasons. First, teacher and student representations may differ geometrically, so direct matching means little. Second, even well-aligned hidden states are not guaranteed to suit the student's subsequent layers. They can look "right" and still be useless for the student's own computation.

How OPMD works

The method has two steps. First, it builds a "student-compatible teacher". This component converts the real teacher's representations into intermediate states that the student's downstream layers can use. Then those states are internalized on on-policy trajectories: the student generates its own continuations and learns to reproduce the target internal states. How the converter is implemented is not clear from the abstract; that will have to be checked in the full paper.

The key idea is that compatibility with the receiving side becomes part of the problem formulation. The authors call the result a "mechanism": the student gets an internal computational structure it can actually apply, rather than a cast of someone else's.

What the results claim

• Three teacher–student configurations: OPMD leads the evaluated distillation baselines in average accuracy in each one.
• Multi-teacher distillation: the approach generalizes to several teachers.
• Cross-family SFT: the method works when training across different model families.
• Reasoning analysis: mechanism supervision yields more efficient reasoning.
• Internal changes: the student's intermediate computations and problem-solving strategies change, meaning information is transferred beyond what the outputs contain.

The last point matters methodologically. An accuracy gain on its own could be explained simply by an extra training signal. The authors say they see structural shifts in the internal computations, which is a stronger claim than a line in a table.

Limitations and open questions

There is no data yet on what the method costs. The extra stage of building a compatible teacher and the training on hidden states almost certainly increase compute costs and pipeline complexity. Another question is access: transferring intermediate computations requires the teacher's internal representations. That is possible with open weights, while models available only through an API do not fit this approach. Finally, the lead is claimed among the "evaluated baselines", and the authors themselves choose that set of baselines.

What to check in practice

Teams that compress reasoning models from open teachers into compact students should add OPMD to their candidate list once code and full tables appear. The first thing to check is which model pairs and tasks produced the gain: math, code and general benchmarks behave differently. Next, compare training cost against plain OPD at an equal GPU-hour budget. The cross-family scenario is the most interesting. If independent replications confirm it, developers will have a way to transfer capabilities between architectures that share no training history.