Researchers Ran 120 CPT Configurations for Game Agents, From 0.8B to 27B

The research team posed a direct question: if you scale data, diversity, and model size, do agents get better in virtual worlds that have almost nothing in common? In the physical world, all experience follows the same laws, so generalization comes naturally. In games, every world has its own visual style, dynamics, controls, and goals. The authors looked for the answer across 120 continued pre-training (CPT) configurations: four model sizes from 0.8B to 27B, six data budgets from 50 to 1 600 hours, and five corpus compositions, from Minecraft alone to a set of different worlds.

Three Axes and Three Levels of Evaluation

The experimental design matters more than any single number. The authors varied three factors independently: data volume, diversity, and model size. They measured quality at three levels, using validation loss and game benchmarks. First came worlds from the training corpus itself, then related worlds, then worlds the model had never seen. This shows exactly where skill transfer stops.

Key metrics and parameters of the study:

• 120 training configurations;
• model sizes: from 0.8B to 27B parameters;
• data budgets: from 50 to 1 600 hours;
• five mixes: from Minecraft only to heterogeneous multi-world corpora;
• evaluation by loss and by gameplay benchmarks in three zones: familiar, related, and unfamiliar worlds.

What Each Axis Delivers

The picture turned out uneven. More data improves results within the corpus, as expected. Diversity helps transfer to related worlds, but the gain stays small in distant games. Model size improves data efficiency: a larger model extracts more value from the same hours of gameplay faster. But if you train it on a single world, it generalizes worse in distant worlds. A large model trained narrowly fits its own world more tightly.

This is a practical observation. Scale without diversity can reinforce specialization rather than generality. For teams building datasets for agents, it is an argument to count the budget not only in hours but also in the number of dissimilar worlds.

Two Training Phases

The most interesting part of the paper is the training dynamics. The authors identify an early "interface" phase, in which gains spread broadly across different worlds. The model learns the basics: how to read the screen, how observations and actions relate, and how the controls work. Then comes a "world" phase, in which gains narrow to worlds in the corpus and those related to them. The model starts memorizing how specific worlds work, and this barely transfers to distant ones.

Based on this, the authors fitted an empirical scaling law. It links validation loss to data volume, diversity, and model size. This makes it possible to estimate the effect before launching expensive training. However, the law describes loss, not success in the game, and the gap between the two remains a separate problem.

Comparison With Specialists and the Limits of Imitation

On the final benchmarks, the models are competitive with Minecraft specialists and with frontier models in 3D worlds. They perform best in real-time mode. Reaction speed matters here, and a compact model trained on game demonstrations does not spend time on long reasoning.

The authors name the limitations themselves. Training on demonstrations runs into an imitation ceiling: the model repeats what it has seen but does not discover new strategies. In distant worlds, the gap persists at every scale studied. The paper's conclusion is direct: universal embodied agents need richer training stages after CPT.

What to Take Into Practice

For agent developers, the takeaways are as follows. If the goal is a strong player in a specific game or a family of similar games, it is enough to grow the data and the model within that family. If you need transfer to new worlds, include heterogeneous games in the corpus and do not expect model size to solve the problem on its own. The early interface phase can serve as a cheap start, but after that, check the gains on unfamiliar worlds, not only by loss. For real generalization, the next step looks like reinforcement learning or other post-training stages that go beyond imitation.