RGR Protects LLMs from Skill Loss in Self-Distillation by Preserving Representation Geometry

A model that teaches itself can raise its average benchmark score while quietly forgetting tasks it used to solve. The authors of the preprint arXiv:2610.12855 found exactly this effect in the family of on-policy self-distillation (OPSD) methods. They also propose a remedy: representation geometry regularization, or RGR.

How OPSD works and where it breaks

In OPSD, the same language model plays two roles: teacher and student. The only difference is the context it is given. No strong external model is needed for distillation, because the model gets its training signal from itself. This is convenient for post-training: it is cheaper, scales more easily, and does not depend on anyone else's weights.

The cost became visible when the authors analyzed training query by query instead of looking at the average result. Queries the model answered correctly in early stages began to be answered incorrectly as optimization progressed. The overall metric kept rising: new wins outweighed the losses, and the chart looked healthy. If you track only the final average, the problem is easy to miss.

Hidden-representation geometry as an indicator

The authors link the degradation to a change in the structure of hidden representations. In their observations, representations inside clusters become less compact, and the clusters themselves become less well separated from each other. Put simply, the model starts to "blend" things it used to distinguish, and loses some of its earlier skills along with that.

RGR is built on this observation. The method adds a regularization that, during post-training:

• keeps representations compact within a cluster;
• preserves the separation between clusters;
• plugs into existing OPSD methods without changing their basic optimization procedure;
• has been tested on several OPSD methods and benchmarks, and the code is open on GitHub (repository supermachinelearner/RGR).

The conclusion in the abstract is worded cautiously: RGR "generally" mitigates degradation and preserves the gain in capabilities. The abstract gives no concrete numbers, so the scale of the effect can only be judged from the full paper and the repository itself.

What matters here, and what remains unclear

The value of the work lies as much in the diagnosis as in the method itself. Forgetting during self-improvement is recorded at the level of individual queries, and representation geometry serves as a measurable proxy for it. Such an indicator can be computed during training, catching drift before benchmarks sag.

Several questions remain open that the abstract does not answer: how exactly the clusters are defined, how much compute the regularizer adds, and how the method behaves on larger models and longer runs. The wording "generally" also means the effect may be weaker for some combinations of methods and tasks.

What to check in practice

If your team is experimenting with self-distillation or other self-improvement schemes, add two steps to your evaluation. First, track not just the average score but also the share of queries the model solved at early checkpoints and then stopped solving. Second, compare runs with and without RGR on your own data. Because the method does not change the core optimization loop, it should be straightforward to plug into an existing OPSD pipeline, and the open code lets you verify the claimed effect before deciding to adopt it.