Activation Steering: A tutorial on controlling a language model from within using PyTorch and nnsight

A Habr tutorial explains the Activation Steering technique — a method for controlling a language model without retraining. Three approaches, live Python code, and a demonstration: an intentional shift of the model toward toxic responses — to show how precise an intervention can be.

What is Activation Steering

Activation Steering enables control over the behavior of a language model without changing its weights or running fine-tuning. During inference, a researcher intercepts the internal activations of the neural network at the desired layer and adds a directed vector to them. The result — the model begins generating text with the specified property.

The method is based on one of the key discoveries in mechanistic interpretability: the activation space of an LLM turns out to be structured. Different concepts — anger, politeness, confidence, conversation topic, language membership — are encoded as relatively linear directions in this multidimensional space. Finding the right vector means obtaining a direct lever of control without changing weights.

A steering vector is obtained by the contrastive method: examples with the desired property and without it are taken, both sets are run through the model, the difference between mean activations is computed. The resulting vector is added to the activations of the desired layer with a scaling coefficient.

Three Approaches to Implementation

The tutorial examines three tools with increasing levels of abstraction:

• pytorch-hooks — `register_forward_hook` intercepts the activation tensor of the selected layer, the vector is added, the modified tensor is returned to the computation graph. Maximum control, minimum dependencies.

• nnsight — a library with declarative syntax. The intervention code reads almost like straightforward pseudocode — convenient for experiments in Jupyter notebooks.

• pyvene — a high-level framework for causal interpretability. Supports reproducible experiments and easy switching between transformer layers.

The choice of tool depends on the task: pytorch-hooks is suitable when full control is needed; nnsight — for readable research code; pyvene — for structured causal analysis.

Where Steering is Applied

The tutorial's demo — shifting the model toward hate-speech. The choice is intentionally uncomfortable: it clearly demonstrates that the intervention works. At the same time, the same tools are used for detecting and neutralizing undesirable behavior — steering works in both directions.

Practical directions of application:

• Alignment research: study which concepts are encoded in the neural network and how separable they are

• Safety red-teaming: check whether undesirable behavior can be activated without training data

• Interpretability: determine which transformer layers are responsible for specific semantic properties

• Fine-tuning-free editing: remove or amplify a pattern through targeted intervention

What This Means

Just a few years ago, Activation Steering was a tool of academic laboratories — it was used by researchers at Anthropic, DeepMind, and EleutherAI in mechanistic interpretability work. The emergence of nnsight and pyvene lowered the barrier to entry to the level of ordinary pytorch code. A Russian-language tutorial on Habr is a rare case when such a specialized topic receives quality explanation without a language barrier. For teams working on the safety and alignment of language models, mastering steering becomes a practical skill, not an academic exercise.