Researchers personalize three MoE LLMs and find the experts responsible for adaptation

Language model personalization is usually a black box: the model is fine-tuned on user data and starts answering "in the user's style." Where exactly in the network this knowledge lives remains unclear. A new paper featured on Hugging Face Daily Papers proposes changing the target: instead of dense LLMs, adapt mixture-of-experts (MoE) models. The authors took three open MoE models, fine-tuned them on a large personalization dataset, and showed that individual personalization features can be tied to specific experts.

Why dense models are hard to interpret

The main problem with dense networks is polysemanticity. A single neuron encodes several unrelated concepts at once, so finding the component responsible for something like "prefers short answers" or "writes in an informal tone" is extremely difficult. Without localization, you can neither understand the mechanism nor control it.

MoE works differently. Each token is routed to only a few specialized experts, and the authors hypothesize that this sparsity reduces polysemanticity. Experts become more meaningful units of analysis.

How the experiment was built

The setup has several steps. The authors adapted three open MoE LLMs on a large-scale personalization dataset. They then trained crosscoders on paired activations from the personalized and original models. Crosscoders are sparse feature dictionaries that map the internal states of two versions of the same network onto each other and yield interpretable personalized latents. Using these latents, the authors identified the experts that encode different personalization features.

The main results, as described in the abstract:

• personalized MoE models match the quality of dense LLMs while activating only a fraction of their parameters;
• at a comparable number of active parameters, MoE models show lower polysemanticity than dense models;
• within the same model family, the personalized latents of MoE models overlap less with the latents of the non-personalized version, meaning personalization sits in more isolated components;
• causal interventions confirm that the identified latents are sufficient to restore the behavior of the personalized model.

Causal verification

Localization alone proves nothing: you can find a neat correlation that has no effect on anything. So the authors tested causality. When they amplify the experts linked to the identified latents, the non-personalized model shifts toward the personalized one. This means the experts do not merely coincide with a feature; they take part in shaping the behavior. Different personalization features also follow different routes through the network.

The abstract contains no numerical metrics: no quality gains, no share of active parameters, no model names. The scale of the effect, and how it compares with existing methods such as LoRA, can only be judged from the full text of the paper. For now this is a proposed method, not a finished product.

Practical takeaway

Teams building personalized assistants get two signals. First, MoE architectures offer not only inference savings but also a more convenient point for interpretation. Second, if personalization is tied to specific experts, it can be controlled surgically, for example by strengthening or weakening individual features without retraining the whole model. This is interesting for tasks where control and auditing matter: privacy, predictable style, and rolling back unwanted user settings.

Before building this into a product, it is worth waiting for the code and full tables, checking reproducibility on your own MoE models, and weighing the cost of training crosscoders against the gain in transparency. Crosscoders themselves require compute and careful tuning.