BorisovAI tests MoE on an RTX 4090 and shows why perplexity breaks LLM evaluation

BorisovAI has published a breakdown of 22 experiments with an MoE architecture in which new experts plug in on top of a frozen language model like plugins. On a single RTX 4090, the setup looked almost perfect from an engineering standpoint: zero degradation of existing skills, precise routing and a noticeable drop in perplexity. But when the system was tested on a math benchmark, it turned out that an appealing metric can point in entirely the wrong direction.

How the setup works

The researcher froze the entire base model and added a small trainable expert to each MLP layer, with a router of roughly 37 thousand parameters on top. The logic is simple: leave the backbone alone, train the new skill separately, then fine-tune only the router so it sends the right tokens to the right expert. Training one new domain took about half an hour: roughly 15 minutes for the isolated expert and another 15 minutes to integrate it into the overall system.

At three scales the setup looked very convincing, with almost no trade-offs:

• GPT-2 124M with 4 domains cut perplexity by 33.4%
• Pythia-410M with 6 domains cut perplexity by 34.3%
• Pythia-1B with 8 domains cut perplexity by 31.2%
• Routing accuracy reached 96%, and degradation of existing skills stayed at 0.000%

The author also tested several popular tricks often recommended for MoE. Load-balancing penalties made results 11–27% worse, and joint training of experts and router led to a collapse in quality. Loss-free balancing worked best: it kept all experts "alive" without a separate additional loss. At this stage everything looked like a strong argument for modular LLMs, where new capabilities can be plugged in without full retraining.

Where the metric broke

Problems started when the architecture was moved to Qwen 2.5 3B and a math expert was added. By internal metrics everything was great again: perplexity on math texts fell by 23.9%, the cross-domain gap grew 64.9-fold, and the router picked the right expert almost without errors. But on GSM8K, which tests the ability to solve grade-school word problems, the model dropped from 74.4% to 65.8%.

"A model that speaks the language of mathematics has absolutely no

idea how to solve problems."

This is the key conclusion of the whole study. The expert, trained on textbooks and papers, did learn the statistics of mathematical language: where formulas usually appear, which terms sit next to each other and what "correct" text in this field looks like. But GSM8K demands chains of reasoning, not style recognition. So the drop in perplexity here measured not an actual improvement in reasoning but simply better modeling of domain text. The researcher separately re-checked alternatives (unfreezing the upper layers, joint training and a two-phase scheme), but all variants stayed at roughly the same minus 8.4–8.6 percentage points.

What worked better

The turning point came with self-distillation. Instead of raw math text, the expert was trained on step-by-step solutions that the base model itself had already solved correctly. For this, 750 GSM8K problems were used: the model solved 638 of them, and those solutions were assembled into a dataset of 119 thousand tokens. That is 33 times smaller than the corpus of 4 million tokens of textbooks and papers, but the format turned out to be much closer to real inference.

The result reversed expectations. After this training, GSM8K rose to 75.5%: 1.1 percentage points above the base model and 9.7 points better than the raw math text variant. Perplexity, meanwhile, got worse by 17.8%. It also turned out that even data packaging matters: the "question/answer" format gave another 2–3 points over the more academic "problem/solution" format. In other words, it is more useful to train the expert for the shape of its future use than for an abstractly "high-quality" corpus.

An attempt to turn this approach into a self-improvement loop did not take off. The first runs hinted at a rise from 75.5% to 76.0%, but after fixing the seed and enlarging the sample, the effect turned out to be statistical noise. With a cold start, the new expert quickly hit a plateau, while with a warm start quality even fell, because too many of the same problems repeated between cycles and the expert overfit. Label smoothing also failed: on math it cost another roughly 9 points.

What this means

For LLM developers, there are two practical takeaways. First, a modular architecture with pluggable experts really can add domain skills without catastrophic forgetting. Second, judging such systems by perplexity is risky: it can improve at exactly the moment the model starts reasoning worse. If the task involves logic, code or math, the main criterion should be behavioral benchmarks, not just attractive language metrics.