Nous Research speeds up LLM pretraining 2.5x without changing the architecture

Nous Research has developed Token Superposition Training (TST), an innovative two-phase pretraining method that cuts the training time of large language models by 2.5x at the same compute cost, without requiring any changes to the architecture, the tokenizer or inference behavior.

How Token Superposition Training works

The method rests on a simple but effective idea: in the first phase of pretraining, neighboring token embeddings are averaged into groups, or bags. Instead of predicting each token separately, the model works with aggregated representations of sequences. This lets it process information in larger blocks and significantly speeds up gradient computation during backpropagation. In effect, the first phase teaches the model to find patterns at a higher level of abstraction.

The second phase is somewhat shorter. The model switches to standard next-token prediction, as any LLM normally does. At this stage it quickly adapts to the final task and recovers any quality losses that may have appeared in the first phase. The transition between phases is smooth and natural for a neural network architecture, with no odd artifacts or incompatibilities.

The key advantage of TST is that it does not touch the model's internal architecture. The parameter count stays the same, and the surrounding tooling and ecosystem do not change: same number of weights, same tokenizer, same optimizer, whether Adam, SGD or any other. At inference, the model is fully compatible with existing deployment systems. This matters for industrial use, where changing the architecture can mean rewriting a lot of code.

Which models the technique was tested on

Nous Research tested TST on models of different scales and architectures to confirm that the approach generalizes:

• 270M parameters (small models for quick experiments)
• 600M parameters (a standard size for research projects)
• 3B parameters (dense architecture, dense models)
• 10B parameters with a MoE (Mixture of Experts) architecture

At all these scales, the method delivered a consistent 2.5x speedup at the same compute cost, measured in FLOP (floating point operations). The results are encouraging: this is not a lab trick that works only at one model size or architecture, but a general approach that scales well. That means it can be applied broadly.

Why this matters for the industry

LLM pretraining is the most resource-intensive and expensive stage of model development. Training a single large model takes thousands of hours on GPU clusters, and electricity and hardware costs run into millions of dollars. A 2.5x speedup is not just a 5-10% improvement but a serious, achievable cut in total costs that directly affects the economics of development.

For startups and small teams, this means the chance to train high-quality, competitive models on a smaller initial budget. For large labs such as Meta*, Mistral or OpenAI, it means the ability to experiment with far more architecture variants, hyperparameters and training strategies on the same infrastructure. That widens the room for experimentation, speeds up the pace of innovation and lets teams test fresh ideas faster.

What this means

Token Superposition Training shows that even in the well-studied area of pretraining, there are simple but powerful ways to save compute. It may inspire other researchers to look for similar optimizations at different stages of model training, from weight initialization to adaptive learning rate schedules. For the industry, it is a positive signal that the line between fundamental research and industrial application is increasingly blurred, and that good ideas quickly find their way into production.

*Meta is designated an extremist organization and banned in Russia.