SAZO speeds up LLM fine-tuning without backprop: up to 6× fewer queries to reach the same loss

The authors of the arXiv:2610.12566 preprint present SAZO, Sample-Adaptive Zeroth-Order Optimization. It is a method for fine-tuning large language models without backpropagation, and it reaches the same training loss with up to 6× fewer queries than standard ZO approaches. It requires no extra model runs and adds only O(N) scalar values on top of MeZO, where N is the number of training examples. The experiments cover four model families, and average downstream results improved as well.

Where efficiency is lost in ZO training

Zeroth-order optimization replaces the exact gradient with an estimate. The model weights are perturbed slightly in a random direction, the loss difference is computed, and the update step is based on that difference. There is no need to store activations or gradients, so memory stays at inference level. For fine-tuning large models on limited hardware, that is a serious advantage. The price is noise: the estimate has high variance, and many steps are needed.

Existing methods, including MeZO, sample training examples with equal probability. The SAZO authors point to a weakness in this scheme. An example's importance for training is not constant: some examples have already been "absorbed" and barely move the weights, while others still produce a large gradient. Uniform sampling ignores this and builds avoidable variance into the estimate.

How SAZO learns to pick examples

The key difficulty is that true gradients are not available in ZO mode. So SAZO extracts information about gradient norms from the directional differences that the method computes at every step anyway. This requires no additional calls to the model.

The second problem is signal sparsity. A minibatch touches only a small part of the dataset, and example importance changes during training. For each example, SAZO keeps an exponential moving average that accumulates observations between updates. These estimates drive adaptive query allocation: important examples are sampled more often.

The theoretical part makes two claims. First, SAZO asymptotically matches the leading term of the convergence of the minimum-variance sampling oracle, an ideal scheme that cannot be realized in practice because it requires knowing the true norms. Second, the method shows a strictly faster convergence rate than standard approaches.

Key parameters

• Extra compute: zero additional model evaluations relative to MeZO.
• Extra memory: O(N) scalars for the moving-average state.
• Speedup: up to 6× in query count to reach the same training loss.
• Quality: improvement in average downstream results.
• Validation scope: four model families.
• Theory: matches the leading term of the minimum-variance oracle and has a strictly better convergence rate.

Practical takeaways

O(N) scalars is very little. For a dataset of 100 thousand examples in float32, that comes to about 400 KB, which is negligible next to the model weights. So the main advantage of ZO, low memory use, is almost fully preserved. For teams that fine-tune models on one or two GPUs and hit memory limits with backprop, this is potentially a cheaper path than before.

There are caveats. The speedup is measured in query count, not wall-clock time, and the abstract says nothing about real latency. ZO methods still need a large number of steps, and the publication does not explicitly claim a comparison with classic gradient-based fine-tuning. The text also names neither the models nor the tasks. Before building SAZO into a pipeline, it is worth waiting for the code and independent reproductions.

Still, the direction is clear. If example selection in ZO training can be optimized almost for free, other long-established assumptions, such as uniform sampling, may be worth revisiting too.