Apple identified when on-policy distillation helps model training
Apple ML Research published a study on the limits of on-policy distillation, a method that provides dense per-token control when training reasoning models. The question is not the method itself, but when to use it. Which teacher model should be chosen? What context should be used for self-distillation? The optimal choice differs from token to token, but computing that in practice is expensive. Apple proposes a training-free approach to address these questions without costly experiments.
AI-processed from Apple ML Research; edited by Hamidun News
Researchers from Apple ML Research have published an analysis of the boundaries of on-policy distillation — a training technique where a teacher model provides per-token guidance to reasoning models. The conclusion: the method can be a powerful tool, but only when applied correctly.
What is on-policy distillation and why is it needed
On-policy distillation is a way to train a new model by giving it very detailed guidance at each step. Imagine: a smart teacher solves a problem aloud, explaining each step, and a student learns not just from the final answer, but from how the teacher reasoned along the way.
This is especially important for reasoning models — models that solve complex problems step by step, unfolding the logic. OpenAI and other labs use similar approaches when training models like o1 and Claude 3.5 Sonnet. At first glance, the more detailed the guidance, the better the student should learn.
When it really helps and when it can hurt
Apple figured out that it's more complex. Key questions remain open:
- Which model to choose as a teacher (stronger is not always better)
- What context to use during self-distillation, when a model learns from itself
- The optimal choice may differ from token to token
The current approach to these questions is usually this: run expensive training, compute for several hours (or days) on GPU, and look at the final metrics. But the problem is that these aggregated indicators hide the truth: at the level of individual tokens, methods often work completely differently.
How Apple proposes to solve this
The research group presented a training-free approach — a way to understand the effectiveness of on-policy distillation WITHOUT expensive experiments. This allows ML engineers to understand whether the method works in their specific scenario before running full model training.
Such tools are critical for large labs: every hour of GPU time costs money, and the ability to predict training results in advance saves resources and development time.
What this means for the ML community
On-policy distillation remains a powerful technique, but this research shows that it cannot be applied mechanically. Different tasks, different models, different data require different solutions — and Apple provided a tool to make these decisions based on evidence, without costly trial-and-error.
Want to stop reading about AI and start using it?
AI News is a curated feed of AI/tech news. Hamidun Academy teaches you to use AI systematically in your work.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.