PACE compresses linear LLM state to 4 bits and saves a third of the memory

A team of researchers has proposed PACE, a post-training quantizer that stores the recurrent state of linear-attention models in four bits and requires no retraining. On four hybrid models ranging from 4B to 35B parameters, the variant with mixed 4-bit and 8-bit heads is comparable to 8-bit DAMP on the three larger models while using about two thirds of its memory. The version with all heads at four bits scores higher than Q-Mamba at the same memory budget.

Why the state becomes a bottleneck

In linear-attention models, context lives not in a growing KV-cache but in a fixed-size recurrent state. That sounds convenient. But an inference system has to keep a separate copy of this state for every sequence it decodes at the same time. The more copies fit into accelerator memory, the more requests are served in parallel, and the cheaper each token becomes.

Hence the interest in quantizing the state itself. Existing methods round it after every generated token. Rounding error accumulates step by step, and at four bits accuracy drops noticeably. Authors of earlier work reached the same conclusion: either raise the bit width or retrain the model for quantization. PACE shows there is a third path.

Two observations behind PACE

The first concerns rounding frequency. PACE rounds the state only once every few tokens and keeps the intermediate tokens in a small buffer. Chunked kernels already work in a similar way during prefill: they update the state once per chunk rather than on every token. Fewer rounding steps mean less accumulated error.

The second observation concerns the structure of the state in gated-delta layers. Large values there are not spread evenly but gather in columns. The reason is that some channels of the update written at every token stay large over many tokens. So PACE uses a shared quantization scale for an entire column. A single large entry no longer degrades the precision of all the other values that would otherwise have to share its scale.

The third detail is mixed precision. Some heads hold on to rounding error longer than others. PACE keeps exactly those heads at eight bits and converts the rest to four.

• Method: post-training, no model retraining
• Target format: four bits per state element, eight for the heads that are most "long-lived in error"
• Rounding: once every few tokens, with intermediate tokens kept in a buffer
• Quantization scale: shared per state column
• Evaluation: four hybrid models from 4B to 35B parameters

What the measurements showed

Mixed PACE with 4-bit and 8-bit heads came out on par with 8-bit DAMP on three of the four larger models. Its memory use was about two thirds of DAMP's. The purely four-bit variant beats Q-Mamba in the comparison at equal memory. The authors also note that rounding once every few tokens preserves the decoding speedup that the buffer provides: the memory savings do not come at the cost of speed.

That last point matters in practice. State quantization that adds a rounding operation at every step can eat up the gain from denser batching. Here rounding is moved out of the hot loop, which reduces that risk.

Where this helps on the infrastructure side

Hybrid architectures, in which linear layers alternate with standard attention, are increasingly common in open and commercial models. For those who serve them, PACE offers a clear lever: cut state memory by roughly a third relative to an 8-bit scheme, without retraining or fine-tuning. The freed space can go toward a larger batch or a longer queue of concurrent sessions.

There are limitations. The result is confirmed on four hybrid models, and on the smallest of them mixed PACE did not reach DAMP's level. Teams planning a deployment should check how the method behaves on their own model and workload before reworking production for four-bit state. Still, the core idea of rounding less often and fitting the scale to the real structure of the data looks portable and inexpensive to implement.