Flipping one axis in Qwen's weights, with no training, cut the share of looping answers from 93% to 33%

Imagine your production model keeps getting stuck in a "wait, let me check again" loop, burns through its entire token limit, and never delivers an answer. The usual fix is fine-tuning. But researchers at MIT CSAIL showed a different route: the GPT-6 Astra model edited the weights of someone else's neural network directly, with no gradients and no training.

On the small Qwen3.5-0.8B, the share of answers that fell into endless repetition on olympiad problems dropped from 93% to 33.

1%. The edit changed 131 072 weights in two layers and boils down to reversing a single direction. This is the second part of a write-up on Habr, in which the author explains why the loop breaks and why this method cannot teach a model a new skill.

The practical takeaway for engineers: before you fix anything, work out what the model is actually missing.

Why the loop breaks

The author compared the base and the published checkpoint bit by bit. The down_proj matrices in layers 15 and 19 were changed, and the edit has rank 1: 99,999% of its energy lies along one direction. It was not subtracted but reversed: the component along this axis flipped sign.

The easiest way to picture the mechanism is as a vote. At each step, the model decides whether to keep going with the same piece of text or exit: close the reasoning block, write the answer, change the line of thought. Layers 15 and 19 used to cast a "keep going" vote. After the edit they vote "exit," and the balance tips toward exiting more often. The author acknowledges this is an assumption: later layers may partly compensate for the shift.

He tested the idea on two texts: an ordinary 220-token solution of an equation and an artificial 397-token "Wait, let me check again…" loop. On the loop, the base MLPs wrote more strongly along the relevant axis: 1.4 times more in layer 15 and 1.9 times more in layer 19. After the edit, the sign was the opposite. Two texts are an illustration, not statistics, and the author says so explicitly.

In a simple model, the drop from 93% to 33.1% corresponds to roughly a 15-fold increase in the per-token chance of exiting. This is a recalculation from the article's numbers, not a direct measurement. But it captures the essence: when the only way out of a loop is a tiny chance at each step, and there are tens of thousands of steps, the share of stuck answers is very sensitive to that chance. Astra's self-check also holds up: 576 of the 579 "rescued" problems were previously cut-offs, and accuracy on answers that already ran to completion did not change.

A switch does not teach

The main limitation: a projection adds nothing. The new representation is computed from the old one, it contains no more information about the correct answer, and its dimensionality does not grow. There is no strict prohibition here: an edit that reverses a direction is reversible, and a computation can make information more accessible. That is why the author states the strong conclusion at the level of the model. One axis with one scale can flip a "keep going" or "exit" mode, but it cannot write in a new algorithm, such as correct multi-digit arithmetic.

This matches the failure on Gemma. Qwen knew how to solve the problems and simply could not stop, so the edit removed the obstacle. Gemma has almost no such trap: its errors are spread across the computations themselves, and there is no "can't-do-arithmetic switch." On top of that, in Gemma 3 270M about 62.6% of parameters are embeddings, leaving roughly 100 million for the 18 transformer layers.

The toy task with a 171 664-weight network is quite different. There, one rule had to be added, something like "first reverse, then accelerate." It can be written in by hand as an "if–then" pair on ReLU units, which takes about 5 900 weights, or roughly 3.4%. The author checks this estimate against the 3.5% from the post: the order of magnitude is the same. What can be written in by hand is what can be described briefly.

Related work and what to take into practice

ROME and MEMIT also edit MLP matrices, but they add facts ("The Eiffel Tower is located in…" with a new answer) rather than remove an obstacle. The "refusal direction" method from Arditi and colleagues orthogonalizes all matrices that write into the residual stream. The Astra edit from 50 runs is more local: 2 matrices and 64 columns. The edit from 10 runs is closer to Arditi: 30 matrices.

For practice, the author suggests the following framing. If the model can solve the task but loops, a targeted weight edit may help. If the errors are in the computations themselves, it will not help. A risk remains: an edit touching less than one percent of the weights noticeably changes behavior, and a control edit with a random direction of the same norm is needed to tell the real effect from side interference. A rank-1 edit is easy to spot in a diff of two tensors, but one smeared across many directions is hard to find.

In the reference section he also covers how weights are laid out. In Llama 3.1 8B, about 70% of parameters sit in the MLPs. In MoE models such as Mixtral, Qwen3-30B-A3B and DeepSeek-V3, 95–97% of weights are in the experts, and any edit would have to account for the router.