Steering vectors in LLM fine-tuning improved SFT and distillation across three seeds

A steering vector is a direction in a transformer's residual stream (the flow of hidden states). If you add it to a hidden state during generation, the behavior of a frozen model shifts in a predictable way. Until now, such vectors have served only as an inference-time control tool.

An independent researcher (the work was not carried out as part of his job at Amazon) tested whether the same vector can also guide the parameter updates themselves during post-training. The paper compares plain SFT (supervised fine-tuning, training on reference answers) and distillation against variants where the vector is added to training, as well as RL with verifiable rewards based on GRPO. The improvement over plain SFT and distillation reproduced across three random seeds, meaning independent runs with different initialization, but the method works only under certain conditions.

How the vector gets into SFT and distillation

The paper targets the off-policy gap. In teacher-forced training, the model optimizes on reference trajectories from a dataset or a teacher, not on the ones its own policy produces. As a result, training and inference behave differently.

The author builds the vector directly from this gap. Before training, he takes the base model's greedy completion and the reference completion, runs both through the model, and averages the hidden states over the response tokens. The difference between these states at a chosen layer defines a direction "from where the model is to where the reference lies." The vector can be computed separately for each prompt or averaged across the dataset. It is computed once before training and never recomputed, so the direction does not chase the evolving policy.

The vector is added at a middle layer, since earlier work shows that the middle of the stack is best suited for interventions on behavior. Injection happens only on response tokens: that is where the loss is computed, and distorting the general encoding of the prompt would be pointless. The paper considers two operations: simple addition and Slerp (spherical interpolation), which rotates the hidden state toward the target direction along the shortest arc and does not change its norm.

Overall, adding the vector generally improved results relative to plain SFT and distillation. The gain grows with the off-policy gap and is largest when distilling from teachers in thinking mode. The available text does not give a specific percentage for the maximum gain. Per-prompt vectors match or beat global averages.

SV-Rescue: returning the gradient to GRPO

In the second mode, the vector is built into RL with verifiable rewards. GRPO has a known weakness: if all of a group's answers to a prompt are incorrect, the advantages are zero and the prompt yields no gradient. The author uses a "correctness contrast" direction, the difference between correct and incorrect answers, and applies it to repair the advantage on incorrect rollouts. The method is called SV-Rescue.

It restores the training signal even in "all incorrect" groups, and this signal transfers to held-out math benchmarks. Because the method shifts only the advantage of incorrect answers, it stacks with existing remedies for zero variance: dynamic sampling, entropy-based advantage shaping, and mixing in reference rollouts.

Where the method breaks

The author describes three limits of applicability. The first is code generation: the reference direction conflicts with the test-passing direction, and the vector stops helping. The second is layer transfer between model families: a layer choice that works for one family fails on another, so the layer has to be tuned again. The third is applying the vector at inference to a checkpoint that was already trained with it: this does not amplify the gain, it removes it.

The conclusion is cautious: a steering vector can be treated as a conditional but effective training signal for LLM post-training. It requires tuning the layer and the intervention mode for each specific model and task, and it is not yet suitable for code. The author calls this the first systematic study of such vectors as a persistent training signal rather than only an inference-time control tool.