Neural Networks Are More Modular Than Their Neurons Suggest: Three Levels of Testing

Neural networks are widely considered hard to break into parts: a single neuron takes part in several different functions, and the conclusion seems obvious. Such networks have no modular organization. The authors of the preprint arXiv:2610.12842 disagree. They introduce a nested hierarchy of three levels of modularity and show, on vision models, language models and multitask recurrent networks, that networks without neuron-level modules often have clear functional modules at the subspace level.

Three levels instead of one question

The central idea of the paper is that the question "is the network modular?" is framed too crudely. If you require different computations to occupy non-overlapping neurons, most networks fail that test. But that is the strictest possible criterion. The authors build three criteria, each looser than the last:

• Unit-level. Different computations occupy non-overlapping sets of neurons. This is the classical notion of a module.
• Orthogonal. There is a rotation of the activation space after which functionally independent subspaces become visible.
• Linear. A constrained linear mapping splits the computations that matter for the task into independent subspaces.
• Verification. The authors developed a method for learning the decomposition at each level. They evaluate the result causally, through selective ablation: remove a subspace and see which function suffers.

The levels are nested. Unit-level modularity implies orthogonal modularity, and orthogonal implies linear. The converse does not hold, and the authors find their effect in exactly that gap.

What they found in practice

In vision models, the content and style subspaces separate under ablation, even though they use overlapping populations of neurons. In language models, the same happens with syntax and semantics: removing one subspace does not break the other.

The paper also compares the results with neuroscience. The learned subspaces align with brain data in different ways. Content subspaces predict activity in the higher areas of the visual cortex better, while style subspaces predict the lower areas better. Semantic subspaces predict activity in the language cortex more accurately than syntactic ones. This is indirect evidence that the discovered separation reflects the structure of the task itself, not an artifact of the method.

The third case is RNNs trained on several cognitive tasks at once. They were previously considered non-modular. The proposed decomposition revealed sharp double dissociations between tasks: removing one subspace breaks one task and leaves the other untouched, while removing the other subspace does the opposite. In neuroscience, a double dissociation is considered strong evidence that functions are separate.

What this means for interpretability

The practical takeaway for people working on mechanistic interpretability: there is no need to look for a neuron responsible for a specific property. Interventions can be made at the subspace level. Once a decomposition is found, a computation can be switched off or altered precisely, even when no individual neuron is specialized. For safety and for controlling model behavior, this is a more realistic point of intervention than hunting for "concept neurons".

There are limitations to keep in mind. This is a preprint without peer review, and the abstract names none of the models tested and gives no quantitative results, so the size of the effect cannot be assessed. It also remains an open question how stable the discovered subspaces are on large modern LLMs, and how much they depend on which functions the researchers define in advance. The method requires labeled computations, for example content versus style, and does not discover modules on its own.

For practitioners, the conclusion is this: when analyzing a model, check all three levels rather than stopping at the absence of neuron-level modularity. For implementation details, read the full text of the preprint itself.