Bridged Self-Distillation teaches small models to reason from any teacher, even one behind an API

Distilling reasoning from a strong model into a small one has long run into two problems: the student cannot reproduce the teacher's solutions, and the teacher's token probabilities are either incompatible with the student's vocabulary or not available at all. The authors of Bridged Self-Distillation (BSD) propose a single scheme that covers three scenarios at once: a teacher with the same tokenizer, a teacher with a different tokenizer, and a closed model behind an API. The teacher has to do just one thing: produce reasoning text.

Where standard distillation breaks down

On-policy distillation is built on a sound idea. The student generates its own answers and the teacher evaluates them, so training happens on the distribution the student actually produces. This removes the mismatch between the training data and the model's behavior at inference time. But the method has limitations.

First, a strong teacher's solutions may be too complex for a small model. The student cannot reproduce the chain of reasoning a giant model produces, and training turns into a chase after an unreachable target. Second, a token-level signal requires the teacher's probabilities in the student's token space. With different tokenizers, these map onto each other poorly, and closed APIs often do not expose them at all. Until now, each situation was handled with its own form of supervision, and no universal recipe existed.

How BSD gets around token incompatibility

The idea is to take the teacher out of the loop and keep only its text. The external teacher writes a reasoning trace in advance, and this can be done offline, once. The trace becomes privileged context.

Two copies of the same model then do the work. The student generates rollouts without access to the trace, just as it will on real tasks. The second copy, the self-teacher, receives the same trace in its context and evaluates those very rollouts. The evaluation is expressed in the student's own token space, so the vocabulary-compatibility problem disappears: the model compares itself with itself, once with a hint and once without.

The verifier makes the final decision. The self-teacher's token-level guidance is used to correct updates on failed rollouts. Successful attempts are confirmed by the external outcome signal, while failed ones get a pinpoint hint about where the reasoning went off track.

The external teacher is queried only for its text, so the training procedure is the same in every scenario. There is no need to rewrite the pipeline for a specific pair of models.

What is known about the setup and results

• Three teacher scenarios: shared tokenizer, different tokenizers, closed API.
• Input from the teacher: reasoning text only; it can be generated offline.
• Roles: the student generates rollouts without the trace in context; the self-teacher (a copy of the student with the trace) evaluates them.
• Signal: token-level guidance plus the verifier's final verdict.
• Application: correcting updates on failed rollouts.
• Result, according to the authors: BSD improves the base model and outperforms baseline methods tailored to each of the scenarios.
• The code is published on GitHub under the TUMLegalTech account.

Who can use this right now

The main practical value is that the method removes the dependence on the teacher's internals. A team that wants to boost a small open model in math, code, or another verifiable domain can take traces from any frontier model through an ordinary API. It needs no logits, no compatible tokenizer, and no special arrangements with the vendor.

The second advantage is cost. Traces are generated offline and only once, so there is no need to keep an expensive teacher in the training loop. You pay only for the text, and most of the compute goes to the small model itself.

The limitation is also obvious: the scheme relies on a verifier. Where an answer can be checked automatically (tasks with an unambiguous result, tests for code), the method fits naturally. For open-ended tasks without a verifiable outcome, you will have to come up with a substitute for the verifier.

For the industry, this is one more step toward knowledge from closed models flowing into compact open ones. API providers cannot shut this door simply by hiding logits: it is enough that they return the reasoning text. And anyone building their own small models should look at the repository and test the scheme on their own domain task.