STAPLE adds billions of parameters to OLMo-2 and decodes twice as fast as MoE

A team of researchers has published STAPLE on arXiv, a method that increases the parameter count of an already trained language model without an expensive sparse pretraining run from scratch. On OLMo-2 1B, it delivers +4.27 points on HumanEval, +2.80 on MBPP, +2.27 on GSM8K and +1.65 on DROP relative to simply continuing dense training. According to the authors, generation slows by no more than 1.5% and training by no more than 3%, even though billions of parameters are added to the model late.

Why not MoE

Sparse computation has long been seen as a way to increase model capacity without inflating the active compute per token. The problem is cost: pretraining sparse architectures takes a lot of data and money. Existing upcycling methods, which convert a ready-made dense checkpoint into a sparse model, mostly target Mixture-of-Experts. The authors point to three bottlenecks there: expert homogeneity, low sample efficiency, and system overhead from routing and memory.

Static sparse memory works differently. Access to it is predictable: parameters are selected by token ID, not by a router's decision. Updates are also tied to a specific token. But, as the authors note, naively inserting such memory into a pretrained model yields weak returns.

How STAPLE works

The method builds on STEM, where the up-projections in FFN blocks are replaced with token-indexed embeddings. That works for training from scratch, but for an existing checkpoint the replacement breaks the computation the model has already accumulated. STAPLE does it differently: the up-projections stay in place, and the embeddings are added as a residual path. The pretrained part keeps working as before, while the new memory learns what the model knows less well about tokens: additional token-specific information.

One important engineering detail: the implementation offloads embeddings to the host. Only about 5% of the most frequent embeddings stay on the GPU, and the rest live in host memory. Because of how tokens are distributed, that is enough for throughput to barely drop.

The numbers

Main results from the preprint:

• OLMo-2 1B, gain over continued dense training: HumanEval +4.27, MBPP +2.80, GSM8K +2.27, DROP +1.65.
• The average score grows with static capacity, and the improvements hold on OLMo-2 7B.
• STAPLE outperforms static sparse baselines: STEM and an Engram-style model.
• Against MoE upcycling: higher average score, 2x faster decoding, less than half the GPU memory for the model, and less energy per token.
• Overhead: generation within 1.5% of the dense baseline, training within 3%.

One caveat matters: the speed and energy comparison with MoE comes from an evaluation oriented toward decoding throughput. Results may differ for scenarios with strict latency requirements on a single request. Also, this is still a preprint on 1B and 7B models, and scaling to tens or hundreds of billions of parameters has not been shown.

What this means in practice

For teams that fine-tune open models, STAPLE offers a cheap way to increase capacity: take an existing checkpoint, add static memory, and keep training with a small loss in speed. The gain is most visible on code and math, where exact knowledge of tokens and patterns is especially useful. The memory profile is interesting for inference: most of the parameters live on the host, and the GPU needs only a small cache. This lowers VRAM requirements compared with an MoE of comparable quality.

Before taking the approach to production, three things are worth checking: behavior on models larger than 7B, robustness to domain shift where the distribution of frequent tokens is different, and latency at small batch sizes. If those checks pass, upcycling through static embeddings could become a serious alternative to MoE for extending already trained models.