SFT Fine-Tuning Narrows AI Behavior: arXiv Study on Diversity Collapse

In July 2026, researchers published the paper "When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play" on arXiv. It shows that standard fine-tuning (SFT) improves the accuracy of language models in games but also sharply narrows the diversity of their moves, more than the balance between accuracy and diversity requires.

What the study found

The paper (arXiv, category cs.CL, identifier 2607.19523) examined how supervised fine-tuning affects the behavioral diversity of models in sequential decision-making. The authors built a controlled set of deterministic board games based on tic-tac-toe variants. In these games, optimal moves can be computed exactly, so behavioral diversity can be measured directly, without contested estimates. The evaluation ran at three levels at once: individual game states, live games in an "arena" setting, and the training trajectories themselves. This made it possible to see how diversity degrades during fine-tuning, not just in the final model.

• Venue: arXiv, category cs.CL, identifier 2607.19523, publication type "new"
• Test environment: deterministic board games based on tic-tac-toe
• Optimal moves can be computed exactly, so diversity is measured directly
• Evaluation ran at three levels: individual states, arena games and training trajectories
• Mitigation method: action augmentation, training on all optimal moves for a state

Why does SFT narrow a model's choices?

Standard SFT causes premature diversity collapse: the model's policy shrinks to a narrow set of moves faster than the accuracy-versus-diversity tradeoff justifies. According to the study, step-by-step reasoning mode (reasoning-mode) often suppresses move diversity even further, without improving accuracy uniformly. One important nuance: reasoning mode, usually seen as a way to improve decisions, worked against diversity here. It narrowed the set of moves without a consistent gain in accuracy.

The cause, in the authors' view, is "narrow-support imitation": the model learns from a single demonstrated optimal move for each state and ignores the other equally good options. As a result, it loses not only the alternatives but also its inclination to explore the solution space.

How to mitigate it

Action augmentation partially solves the problem. Instead of a single demonstrated move, the model is trained on all optimal actions for each state. According to the abstract, this technique preserves "action support" and maintains the model's exploratory behavior without sacrificing accuracy. The method does not eliminate the problem entirely. The authors explicitly call the mitigation partial, but they show that simply training on a broad action support already changes the model's behavior for the better.

"Preserving action support during SFT is important for maintaining

exploratory behavior," the abstract of the arXiv study states.

What this means

The work points to a hidden cost of a popular fine-tuning method: by optimizing a model for accuracy, developers may quietly strip away its flexibility and its tendency to explore. For tasks where diverse solutions matter, such as game agents, planning and multi-step actions, the authors advise preserving action support already at the SFT stage.

Frequently asked questions

What is diversity collapse in language models?

Diversity collapse is a situation where, after fine-tuning, a model starts choosing the same narrow set of answers or moves and loses equally good alternatives. In the study, it appeared under standard SFT earlier than the balance between accuracy and diversity required.

How does action augmentation help preserve diversity?

Action augmentation trains the model on all optimal moves for each state rather than on a single demonstrated one. According to the paper, this partially mitigates diversity collapse and preserves the model's exploratory behavior.