MIT: A Weakened Copy Teaches a Model to Reason — Up to +13 Points of Accuracy

A team at the Massachusetts Institute of Technology has found a way to improve a reasoning model without an external teacher, reference solutions or a verification environment. The method is called WISD. Researchers take a base model and deliberately build a weaker copy of it, either by coarsening the weights (quantization) or by dropping the smallest ones (pruning). They then train the original on the difference between the two versions. The most striking result comes from Qwen3.5-4B. On HMMT 2025, the Harvard–MIT math competition, its accuracy rose from 57.7% to 70.5%. On LiveCodeBench v6, a collection of programming problems, it climbed from 39.1% to 56.7%.

Why weaken a model at all

Self-distillation, where a model teaches itself, has become routine. But the popular schemes require "privileged information": reference solutions, expert write-ups or feedback from an environment. Such a teacher knows more than the student because it sees a hint. If all you have is a pile of problems without answers, these schemes don't work.

The authors approach it from the other side. The teacher doesn't get stronger. The original model remains the strong member of the pair, and the weak partner is created artificially by corrupting its weights. The difference in token preferences between the two gives a signal for which direction to improve the policy. A similar idea has appeared before in work on weak-to-strong supervision. What's new is that both models come from the same checkpoint.

How it works technically

The procedure has several steps:

• The base model generates solutions for unlabeled problems. In the experiments, that means 3,200 problems with four answers each.
• Two frozen models, the original and the weakened copy, score every token in those answers given the same context. A token gets a positive signal if the original considers it more likely than the weak copy does, and a negative signal if the reverse is true.
• The student starts from the original's weights and learns to shift its distribution toward tokens with a higher probability ratio. The trajectories and both scoring models stay fixed during training. At inference, only the student runs; there is no weak copy alongside it.

The copy is made in two ways. Neither requires calibration data or additional training:

• Low-bit quantization. Weights are split into groups, and within each group the values are rounded to a small number of levels using a symmetric scheme.
• Magnitude pruning. Within a group, weights below a relative threshold are zeroed, and then the norm of each output row is restored to its original value.

Only the attention projections and fully connected layers are corrupted, including the experts in Mixture-of-Experts models. Embeddings, output heads, normalization layers and routing are left untouched. The architecture and tokenizer stay the same, so the copy's predictions can be compared directly with the original's. If the two models were identical, there would be no signal. Corrupting the weights creates a difference that depends on the specific token.

What the experiments showed

The researchers tested the method on five models from different families and of different sizes, both dense and MoE: Qwen3-4B, OLMo-3-7B-Think, Qwen3.5-4B, Qwen3.6-27B and Qwen3.6-35B-A3B. There are five benchmarks: three in math, including HMMT 2025 and the 2026 problems from the Mathematical Association of America, and two in code, including LiveCodeBench v6. Math was scored as the average over 32 attempts per problem, and code over four.

• Quantization raised average accuracy over the base models by 4.0 points, and pruning by 5.5.
• Improvement was recorded in 21 of 25 model–benchmark pairs for quantization and in 23 of 25 for pruning.
• Qwen3.5-4B gained the most: its average across the five benchmarks rose by 11.3 points with quantization and by 13.4 with pruning.
• Pruning added an average of 5.2 points to Qwen3.6-27B.
• Quantization added 5.9 points to Qwen3.6-35B-A3B. On HMMT 2025 it rose from 68.5% to 74.6%, and on LiveCodeBench v6 from 61.8% to 69.5%.

The authors claim that on Qwen3-4B and OLMo-3-7B-Think, both variants are competitive with OPSD and SDPO, which need reference solutions or feedback. WISD's averages across the five benchmarks are higher than the published figures for those baseline methods. One important caveat: the comparison relies on already published results rather than a rerun of all the methods under identical conditions. This is a preprint, and there has been no independent verification yet.

What this means in practice

For teams that fine-tune open models on their own data, the low barrier to entry is the interesting part. All you need is a checkpoint and a set of problems without answers. No annotators, verification environment or strong external model are required. The weight operations are simple and need no calibration, and one of the weakening options, quantization, is already part of the standard deployment toolkit.

Questions remain. The results come from math and code, where correctness criteria are clear. It is unclear whether the method will transfer to open-ended tasks that lack such criteria. The experiments used models of up to 35 billion parameters. The authors have published the code on GitHub (jet-ai-projects/WISD), so you can already check the numbers on your own models.