FARO optimizer cuts GPT validation loss by 0.0136 nats on 6 of 6 seeds

Current recipes for training large language models split the work between optimizers by hand. Orthogonalized momentum such as Muon updates the attention and MLP matrices, Adam-style rules handle everything else, and a single global budget governs the update for the whole model. Researchers at the Hong Kong University of Science and Technology (Guangzhou) have proposed FARO, an optimizer that makes these decisions from measurements on the model being trained.

On a GPT-style model trained on FineWeb, the FARO-Adaptive variant (which only selects the update geometry) reached a loss of 3.2780 nats. The FARO-ERC add-on (which redistributes the budget by equal risk contribution, ERC) lowered loss on all 6 seeds, meaning runs with different random initializations, by an average of 0.

0136 nats. A nat is the natural unit for measuring cross-entropy loss. The code is published on GitHub.

Geometry chosen by outcome, not by layer type

The standard recipe rests on two assumptions. First, layer type determines the right update geometry: spectral norm for matrices, coordinate-wise adaptivity for vectors. Second, one scalar budget is enough for the whole network. Both assumptions come from human heuristics and small ablation grids, not from measurements on the model being trained. The newer matrix optimizers (NorMuon, Scion, Dion2, ARO, PolarGrad, Newton–Muon) change only one detail of the recipe and still fix a single geometry for all the matrices they touch.

FARO makes the choice itself the object of learning. It splits the parameters into 117 blocks. For each block, every candidate rule is applied to a held-out validation set, and the one that cuts loss the most is kept.

While the ranking remains informative, the routes can change; after that, they are frozen. This caps the cost of the search and removes route selection as a source of variance. FARO is a framework, not a new update rule: Muon, NorMuon and Adam-like rules enter it as candidates, not as competitors.

In the appendix, the authors show that the per-block decision agrees with a joint decision across all blocks under a mild separability assumption, and they estimate a bound on the coupling it ignores.

As a result, FARO-Adaptive reached 3.2780 nats on the full training schedule. The best hand-built recipe, NorMuonAndAdam, scored 3.2794. The system reproduced the human heuristic without any manual assignment of geometry to layers.

Risk parity from portfolio theory

The second part of the work concerns the budget. Once the routes are frozen, each block keeps its own multiplier that scales its update. Three scalar controllers (a fractional Kelly multiplier, its continuous relaxation, and a rule with a complexity estimate) scale one shared budget. All three were indistinguishable from the risk-neutral FARO-Adaptive, which fits the observation that widening the step when chasing risk is unproductive.

The authors give a structural explanation. A scalar multiplies all budgets by a common factor and moves only along the diagonal of the budget simplex, so it cannot decide which blocks need more updates and which need fewer. They therefore redistribute the budget across blocks while keeping the sum unchanged, using a rule from portfolio theory: equal risk contribution (ERC, also known as risk parity). In a portfolio, weights are chosen so that each asset contributes the same share of total risk. In FARO, a block plays the role of the asset, and the ratio of the update to the weight norm plays the role of volatility.

Raw risk differs across blocks by orders of magnitude. The method therefore clips it relative to the median across blocks (winsorization) and caps the multiplier from above. It then finds, by bisection, the scale that preserves the total budget to numerical precision. The redistribution uses only the previous step's risk, so the current gradient does not influence its own budget. Bisection gives the exact solution of a separable risk problem in which a single scalar price carries the budget coupling; equal risk contributions emerge from the solution rather than being imposed. This is why the comparison with FARO-Adaptive isolates the redistribution from a simple change in the effective learning rate.

A modest but consistent gain

FARO-ERC lowered loss on clean data on all 6 seeds, by an average of 0.0136 nats at the same compute. The sign test, the paired test and the confidence interval agree with one another, and the effect exceeds the measured run-to-run variation. Time and memory do not grow. The mechanism is that ERC trims the riskiest blocks rather than expanding the safest ones. The authors themselves call the gain modest but consistent.

The team also tested the method on a corrupted data stream. Mean loss still fell, and most seeds still favored FARO-ERC.

The practical takeaway for anyone tuning pretraining: the manual split of "matrices get Muon, the rest get Adam" can be replaced by measurement on a validation set, and redistributing the budget across blocks at a fixed total gives a small but reproducible improvement. The evaluation covers one GPT-style model and one dataset, so whether the result transfers to other scales and data remains an open question.