SFT-дообучение сужает поведение ИИ: исследование arXiv о коллапсе разнообразия

In July 2026, researchers published a paper on arXiv titled "When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play," which showed that standard fine-tuning (SFT) improves the accuracy of language models in games, but at the same time sharply narrows the diversity of their moves — more than the accuracy-diversity trade-off would require.

What the study found

The paper on arXiv (category cs.CL, identifier 2607.19523) examined how supervised fine-tuning affects the behavioral diversity of models in sequential decision-making. The authors built a controlled set of deterministic board games based on tic-tac-toe variants: in these, optimal moves can be computed exactly, meaning behavioral diversity can be measured directly, without contested evaluations. Assessment was carried out simultaneously at three levels — individual game states, live matches in "arena" mode, and training trajectories themselves — which made it possible to see how diversity degrades during the fine-tuning process itself, not just in the final model.

• Venue — arXiv, category cs.CL, identifier 2607.19523, publication type "new"
• Test environment — deterministic board games based on tic-tac-toe variants
• Optimal moves are exactly computable, so diversity can be measured directly
• Evaluation was conducted at three levels: individual states, "arena" matches, and training trajectories
• Mitigation method — action augmentation, training on all optimal moves for a state

Why does SFT narrow the model's choices?

Standard SFT causes a premature diversity collapse: the model's policy collapses to a narrow set of moves faster than is justified by the accuracy-versus-diversity trade-off. According to the study, step-by-step reasoning mode (reasoning-mode) often suppresses move diversity even more strongly, without improving accuracy uniformly. An important nuance: reasoning mode, usually considered a way to improve decisions, here worked against diversity — narrowing the set of moves without a stable gain in accuracy.

The cause, according to the authors, is "narrow-support imitation": the model learns from a single demonstrated optimal move for each state and ignores the other equally valid options. As a result, it loses not only alternatives but also the tendency to explore the solution space.

How to mitigate this

Action augmentation partially solves the problem — training on all optimal actions for each state at once, instead of a single demonstrated move. According to the abstract, this technique preserves "action support" and maintains the model's exploratory behavior without destroying accuracy. However, the method does not eliminate the problem entirely — the authors explicitly call the mitigation partial, but show that the very framing of training on a broad action support already changes the model's behavior for the better.

"Preserving action support during SFT is important for maintaining

exploratory behavior," the abstract of the arXiv study states.

What this means

The paper points to a hidden cost of a popular fine-tuning method: by optimizing a model for accuracy, developers may unknowingly strip it of flexibility and the tendency to explore. For tasks where solution diversity matters — game agents, planning, multi-step actions — the authors recommend preserving action support already at the SFT stage.

Frequently asked questions

What is diversity collapse in language models?

Diversity collapse is a situation in which, after fine-tuning, a model starts choosing the same narrow set of responses or moves, losing equally valid alternatives. In the study, it occurred under standard SFT earlier than the accuracy-diversity balance would require.

How does action augmentation help preserve diversity?

Action augmentation trains the model on all optimal moves for each state, rather than on a single demonstrated one. According to the paper, this partially mitigates diversity collapse and preserves the model's exploratory behavior.