Phase-HDC trains a model with no optimizer history and stores 16–23x less than Adam

A team of researchers has published Phase-HDC on arXiv, a method for training a compact hyperdimensional classifier while keeping only the model itself in memory. Across eleven datasets (images, tables, text), it takes 16–23x less space than standard float32 Adam and 4–6x less than 8-bit Adam. The price is accuracy: about five points lower than float32 Adam on average.

Why the optimizer weighs more than the model

Small models have an awkward property: training needs far more memory than storing the finished result. The cause is the optimizer. For every parameter, Adam keeps a record of past gradients, namely the first and second moments. With low-bit parameters, this overhead becomes especially noticeable.

The paper deals with hyperdimensional (HDC) classifiers, whose trainable parameters are angles with a small number of bits. The authors call this set of parameters "phase memory." The optimizer state for it takes up several times more space than the model itself. That raises the question the paper opens with: can you train such a model while storing nothing but the model?

One step, against the gradient, only if it is large

Phase-HDC answers with a very simple rule. On each update, a stored angle moves by no more than one grid step. The direction is chosen against the sign of the current gradient. And a parameter moves only when that gradient is large enough. There is no history, just a threshold on the current value.

The authors claim the rule is not a heuristic but the exact solution of a first-order loss model in which every changed parameter pays a fixed cost. In other words, the threshold follows from the problem setup: a parameter is worth moving only when the expected gain in loss outweighs that cost.

The main results from the abstract:

• under otherwise equal conditions (only the update rule changes), Phase-HDC matches Adam with 6-bit moments in accuracy while storing three times less;
• the savings are 16–23x relative to float32 Adam and 4–6x relative to 8-bit Adam;
• the average accuracy loss relative to float32 Adam is about five points;
• on six of the eleven datasets, Phase-HDC is more accurate than 8-bit Adam;
• on byte-level text prediction, 8-bit Adam breaks down, while Phase-HDC does not.

What measurements inside training showed

The most interesting part of the paper is the explanation of why this works. The authors instrumented the training runs to see what Adam's moments actually do when parameters must sit on a discrete grid. They conclude that the moments mostly decide whether a parameter moves at all. That is a binary decision, and a threshold on the current gradient can make it without any memory.

Coarse quantization of the moments, by contrast, breaks exactly this decision. Inputs that appear rarely in the data suffer: their statistics in the moments are weak, and after rounding they can no longer distinguish "move" from "don't move." This explains the failure of 8-bit Adam on byte-level text, where rare symbols are plentiful.

Where the method is useful and where it is still too early

The practical takeaway is a cautious one. If you train or fine-tune compact HDC models on devices with a tight memory budget, such as embedded systems or on-site training scenarios, the optimizer cost can be removed almost entirely. For such tasks, five points of accuracy may be an acceptable price, and on rare or unevenly distributed data the method is also more robust than 8-bit Adam.

There are two limitations, and the authors name them themselves. First, the accuracy is below float32 Adam, so where every point matters, the method is not a replacement. Second, the savings are computed as logical state, not as measured hardware memory. The real gain on specific hardware depends on data packing and implementation, and it has yet to be measured. Finally, the evaluation was done on HDC classifiers, and the paper does not claim that the idea transfers to ordinary neural networks.

One observation is worth remembering regardless: in discrete training, gradient history is often needed only to decide "move or not," and a threshold on the current gradient is enough for that.