TraceForge trains models on "their own thoughts": +7.1 points on MBPP+

The TraceForge team has proposed a distillation method for the case where the teacher returns only an answer and hides its reasoning. Instead of trying to extract the hidden chain of thought, the system takes the context and the teacher's final answer and writes the reasoning itself, in the style of the student model's family. Across five benchmarks, with everything else held equal, general tasks showed almost no difference, but coding gains are noticeable: with the same 5,000 questions and the same teacher answers, HumanEval+ rose 3.7 points and MBPP+ rose 7.1.

Hidden reasoning and a fragile workaround

Having one LLM generate training data for another remains one of the most versatile ways to train models. But some frontier teams have closed this channel for their own systems: what comes out is an answer or a short summary, without a detailed reasoning trace. The TraceForge authors point out the irony: these models themselves grew up on distillation of human knowledge, while distilling an open model is still routine. The restriction applies only to their own outputs.

Recent work has learned to recover hidden reasoning by replaying encrypted blocks between models. The method works, but it violates providers' terms of use. You cannot build a product on it for the long haul, because any change to the rules or to the technical protection wipes out the entire pipeline.

Generator and critic: building someone else's thoughts with your own hands

TraceForge approaches the problem from a different angle. The input is the context and the teacher's answer, and the reasoning is written by a generator trained to reproduce traces from the target model family.

Training has two stages. First, supervised: the generator learns to reconstruct a family's real traces from the context and the answer. Then adversarial: a critic learns to judge how authentic a trace looks, and the generator adapts to it. The result is a generator that, given the teacher's answer, writes the reasoning the way a model from the student's family would have written it.

Plausibility was measured by an independent judge that estimates the probability that a trace is genuine, on a scale from 0 to 1. The authors stress that these are authenticity scores, not the share of correct solutions.

• Generated traces: 0.68 on average
• Real traces: 0.66
• Traces picked by the critic from four candidates: 0.82
• Training set: 5,000 questions with the same teacher answers
• HumanEval+: +3.7 points at the final checkpoint
• MBPP+: +7.1 points at the final checkpoint, with the advantage holding at every checkpoint

Keep in mind that the judge measures resemblance to the original. Generated traces scored 0.68 versus 0.66 for real ones, which means only that the judge cannot tell them apart from real traces. The number says nothing about whether the reasoning is correct. Best-of-four selection, however, lifts the score to 0.82, so the critic is useful not only during training but also as a filter on the output.

Where the gain shows up and where it doesn't

The experiment is designed so that only one variable changes. Five training corpora were compared across five benchmarks. On general tasks, the corpora came out roughly equal. On coding, the generated reasoning beat Fable 5's own thinking content, with the same questions and the same teacher answers.

The authors only speculate about why: reasoning written in the student family's style is easier for the student to absorb. If the hypothesis holds, what matters is not the quality of someone else's thoughts but their compatibility with how the student builds its own chain. That is plausible for code, where reasoning structure is stricter and style errors show up more visibly in tests.

What to change in your distillation pipeline

For teams that fine-tune models on answers from closed systems, the takeaway is practical. A teacher without traces is not a dead end, and there is no need to circumvent provider restrictions. The context and the answer are enough, and the reasoning can be synthesized locally and without breaking any rules.

The limitations are also clear. A gain on general tasks has not been shown; it appeared on coding. In addition, the authenticity score remains a proxy, and you cannot rely on it without checking trace correctness on your own tasks. A sensible first step: take one coding task, generate four candidates per answer, pick the best with the critic, and compare against the baseline sample on MBPP+. If a gap of around seven points reproduces, switching to this pipeline will pay off.