Can Transformers Select a Model? An arXiv Bayesian Test at 2.8M Parameters

A 2.8M-parameter transformer learned Bayesian model selection, identifying which hypothesis class generated the data, with a divergence of just 0.01 bits from the theoretical optimum. That is the finding of "Bayesian Wind Tunnels for Model Selection," posted to arXiv in July 2026.

What are "Bayesian wind tunnels"?

"Bayesian wind tunnels" are controlled environments in which the exact posterior distribution over hypothesis classes is known in closed form, so the model's answer can be compared directly with the Bayesian optimum. Earlier work had already shown that transformers perform exact Bayesian filtering within a single fixed hypothesis class. The authors ask a new question: can a model select the class itself, that is, recognize which structure produced the data?

The tasks are built on fixed-point-free involutions, functions with the property f(f(x))=x, which is purely relational. The transformer matches the optimum even when the symbols are opaque and their meanings change in every episode. Working with opaque symbols matters: it removes any cues the model could latch onto and leaves only the structure of the task.

• The model is a 2.8M-parameter transformer, tested on 3 seeds
• Entropy agreement with the Bayesian optimum: 0.01 bits
• Non-nested comparison (involutions vs. 3-cycles): posterior MAE below 0.001
• The boundary on arithmetic tasks holds under 112x scaling (2.8M → 316M parameters)
• Calibration gap in modern LLMs: about 55×

Where model selection breaks down

Model selection fails completely when the classes can be distinguished only through arithmetic, addition or multiplication modulo n. In that case the transformer copes with integer tokens but fails on opaque symbols. According to the abstract, this boundary persists when the model grows 112-fold, from 2.8M to 316M parameters, so network size is not the cause.

Subtask diagnostics localized the failure: it occurs on composing header inversion with arithmetic, not on parsing the header itself.

It comes down to stable semantics

The cause of the failure is not the numbers as such but the stability of their meanings. A stationarity control experiment showed that opaque tokens with a fixed relabeling produce an error of only 0.009 bits, so they perform no worse than integers.

"Stable semantics, not integer identity, is what allows a

computational circuit to be compiled," the paper's arXiv abstract says.

Testing modern frontier LLMs on the same tasks revealed qualitatively Bayesian behavior, but with a large calibration gap of about 55×. The authors stress that this estimate comes from loss-based probes, so it indicates a direction rather than an exact magnitude.

What this means

The work draws a concrete boundary around what transformers can do: they can select a model, but only when the distinguishing feature has stable access to semantics. It is a reference point for mechanistic interpretability, showing which computational circuits a network is able to "compile" and which remain out of its reach regardless of parameter count.

FAQ

What is Bayesian model selection?

It is the ability to determine which hypothesis class the data came from, rather than simply estimating parameters within a single class. In the paper, the transformer compares, for example, involutions with 3-cycles and produces a posterior with an error below 0.001.

Why does the model fail on arithmetic?

When the feature that separates the classes requires addition or multiplication modulo n, the transformer copes only with integer tokens and fails completely on abstract symbols. The control experiment showed that what matters is the stability of the tokens' semantics, not their numeric nature.