Могут ли трансформеры выбирать модель? Байесовский тест arXiv на 2,8M параметров

A transformer with 2.8 million parameters has learned Bayesian model selection — determining which class of hypotheses the data was generated from — with a discrepancy of just 0.01 bits from the theoretical optimum. This was shown by the paper "Bayesian Wind Tunnels for Model Selection," posted on arXiv in July 2026.

What are "Bayesian wind tunnels"

"Bayesian wind tunnels" are controlled environments where the exact posterior distribution over hypothesis classes is known in closed form, so the model's answer can be compared directly against the Bayesian optimum. It had already been shown that transformers perform exact Bayesian filtering within a single fixed hypothesis class. The authors' new question is whether the model can select the class itself — that is, recognize which structure generated the data.

The tasks are built on fixed-point-free involutions — functions with the property f(f(x))=x, which is purely relational. The transformer matches the optimum even when the symbols are opaque and their values change in every episode. Working with opaque symbols matters: it strips away cues the model could "latch onto," leaving only the structure of the task.

• Model — a transformer with 2.8 million parameters, tested across 3 seeds
• Entropy agreement with the Bayesian optimum — 0.01 bits
• Non-nested comparison (involutions vs. 3-cycles) — posterior MAE error below 0.001
• The boundary on arithmetic tasks holds across a 112x scale-up (2.8M → 316M parameters)
• Calibration gap versus current LLMs — about 55x

Where model selection breaks down

Model selection fails completely when the classes are distinguishable only through arithmetic — addition or multiplication modulo n: in that case the transformer handles integer tokens fine but fails on opaque symbols. According to the paper's abstract, this boundary holds across a 112x increase in model size — from 2.8 to 316 million parameters — so it isn't a matter of network size.

Subtask diagnostics localized the failure: it arises in the composition of header inversion with arithmetic, not in the header parsing itself.

It comes down to stable semantics

The reason for the failure isn't the numbers themselves, but the stability of the values. A stationarity control experiment showed: opaque tokens with a fixed relabeling produce an error of just 0.009 bits — performing no worse than integers.

"Stable semantics, not integer identity, is what allows a

computational scheme to be compiled," the paper's abstract on arXiv states.

Testing current frontier LLMs on the same tasks revealed qualitatively Bayesian behavior, but with a much larger calibration gap — about 55x; the authors note that this estimate was obtained through "lossy" probes and therefore indicates a direction rather than a precise figure.

What it means

The paper draws a concrete boundary around transformer capabilities: they can select a model, but only when the distinguishing feature has stable access to semantics. This is a reference point for mechanistic interpretability — it shows which computational schemes a network is able to "compile," and which remain out of reach regardless of parameter count.

Frequently asked questions

What is Bayesian model selection?

It's the ability to determine which class of hypotheses the data came from, rather than simply estimating parameters within a single class. In the paper, the transformer compares, for example, involutions against 3-cycles and produces a posterior with an error below 0.001.

Why does the model fail on arithmetic?

When the feature that distinguishes classes requires addition or multiplication modulo n, the transformer only handles integer tokens and fails completely on abstract symbols. The control experiment showed that what matters is the stability of token semantics, not their numeric nature.