Synthetic Pre-Pretraining for LLMs Doesn't Pay Off: At Equal FLOPs, It Loses to Plain Text

The idea sounds appealing: before feeding a language model terabytes of internet text, give it a "warm-up" on pure structure. Cellular automata, languages with properly nested brackets, procedurally generated sequences. The thinking goes that the model will learn useful patterns in advance and then pick up real language faster. This is called synthetic pre-pretraining, or PPT.

A new paper, Rethinking Pre-Pretraining (Alexander Stern, Tim Rocktäschel, Andrea Locatelli and Laura Ruis), tests this idea with no discounts. It shows that the elegant hypothesis does not survive honest bookkeeping.

What's the catch

Most earlier PPT papers compared models without regard for how much compute went into the synthetic stage. The authors fixed the total FLOPs budget: if part of it goes to synthetic data, less is left for natural text. Under that regime, PPT never once beat training on natural language alone.

An example with numbers. Take 100M FineWeb tokens and a budget of 1 × 10¹⁸ FLOPs. Standard training with repeated text gives 1.064 bits per byte (BPB) on validation, where lower is better. If you devote 1% to 10% of compute to PPT, the metric worsens to 1.068–1.076. The difference is small, but the direction is unambiguous: you paid in compute and got a loss.

The explanation is simple. When the model switches to natural language, gradients quickly overwrite the structures it learned on synthetic data. It is like learning chess openings and then playing checkers for six months.

Where earlier successes came from

The most interesting part of the paper is the analysis of why PPT seemed useful in the first place. The authors reproduced the cellular automata (NCA) experiment on a 400M-parameter LLaMA trained on OpenWebText. The perplexity gain from NCA PPT was 5.4%.

But if you change the embedding initialization from 𝒩(0, 1) to 𝒩(0, 0.02), the standard for such models, the baseline with no PPT at all improves by 9.0%. That fully cancels out the benefit of the synthetic data. It turns out PPT was largely fixing a poor initialization rather than adding knowledge to the model.

The authors also proposed a synthetic corpus for variable value lookup (DeclRef) and a tool called LLMTuna, an LLM-based hyperparameter optimizer. The code for both projects is open on GitHub.

Limitations

The conclusions should be read with caveats. All models are under 1B parameters, and it is unknown whether PPT behaves the same way at scales of tens of billions. The LLaMA initialization test and the data-shuffling experiments were run without re-tuning hyperparameters, so part of the result may shift with careful tuning.

It is worth noting separately that this is a negative result about a specific class of methods. It does not negate the value of synthetic data in general, for example for fine-tuning on code or math.

What to do in practice

If you train your own models, especially small ones, the takeaways are:

• Don't add a synthetic PPT stage to your pipeline "just in case." It costs compute and, judging by the data, does not pay off.
• Before believing an impressive gain from a new trick, check the basics: initialization, learning rate, scheduler. A weak baseline easily creates an illusion of progress.
• Compare methods at an equal FLOPs budget, not at an equal number of steps or natural-text tokens.

The main lesson is broader than the topic itself: many "improvements" in ML actually cure a poorly tuned baseline. If you don't have a strong control group with an honest budget, you are most likely measuring not the method but your own shortcomings.