Routed Sparse Autoencoders Separate Words From Intonation in Speech on WavLM and SPEAR

Self-supervised speech encoders store words, voice, emotion and intonation in a single shared feature space. A new arXiv paper proposes Routed Sparse Autoencoders: a sparse autoencoder with two routes, one linguistic and one paralinguistic. The method was tested on two frozen encoders, SPEAR and WavLM, and on two corpora, LibriSpeech and MSP-Podcast. Speaker identity, emotion and prosody end up in the paralinguistic route.

Why everything is mixed in one vector

Models like WavLM learn from huge volumes of unlabeled audio. They have to predict masked fragments of the signal, so they absorb everything that helps with that: phonetics, timbre, tempo, emotional coloring. All of it lands in a single hidden representation. For speech recognition, the speaker's voice is closer to noise. For speaker verification or emotion analysis, it is the whole point of the task. These tasks are usually solved on top of the same vector, and information of one type leaks into the other.

The authors want to split these factors into separate channels. That would make it possible to remove speaker identity data from the representation without breaking transcription, or to change the emotion without touching the meaning.

How the method works

The base is a TopK sparse autoencoder: at each step, only the K strongest features stay active. On top of it, the authors add two techniques. The first is route-specific supervision: each route gets its own training signal for its own type of information. The second is cross-factor adversaries: adversarial components keep the linguistic route from carrying paralinguistic factors, and vice versa.

Key parameters from the abstract:

• architecture: TopK sparse autoencoder with two routes, linguistic and paralinguistic;
• training: per-route supervision plus cross-factor adversaries;
• base encoders: SPEAR and WavLM, both frozen;
• paralinguistic factors: speaker identity, emotion, prosody;
• training on LibriSpeech, transfer to MSP-Podcast without retraining on the representation side;
• evaluation: independent probes and interventions in feature space.

What the evaluations showed

The main result: linguistic information is stronger in the linguistic route, while speaker identity, emotion and prosody are preserved in the paralinguistic route and substantially weakened in the linguistic one. The authors measured this with independent probes, meaning classifiers that took no part in training the autoencoder. That detail matters: if the separation holds up under probes the model has "not seen," it is unlikely to be tuned to one specific criterion.

The second evaluation tests transfer. LibriSpeech consists of books read aloud; MSP-Podcast consists of spontaneous speech with emotion labels. The route organization learned on the first corpus held up on the second. The third evaluation uses interventions: if you swap one route between examples, exactly the swapped factor transfers, while the information in the unchanged route is mostly preserved. The phrase "mostly" is an honest one here: the authors do not claim complete isolation.

The abstract gives no numerical metrics, so the size of the effect cannot be judged from it. The tables in the paper itself will show the real scale.

Where this could be useful

The most obvious applications:

• speech anonymization: remove identity features from the representation and keep the content;
• emotion and prosody recognition without leaking meaning, as well as analysis of what exactly the model encodes;
• controllable voice and style conversion: swap the paralinguistic route while the meaning stays unchanged.

Three things are worth checking before deployment. First, "substantially reduced" does not mean "removed": privacy requires testing with stronger attacks than the probes used in the paper. Second, both evaluations were run on English speech; how the method behaves on other languages and in noisy recordings is unknown. Third, the experiments use frozen encoders, and the question of end-to-end training remains open. For teams building speech pipelines, the work is useful as a recipe for interpretable decomposition of representations. It is worth reproducing on your own data first.