Conditional Program shrinks DeepSeek's n-gram memory from 203 to 51 GB at 2 bits

N-gram memory tables give a language model extra capacity with almost no increase in compute: only a few rows are read per token. But the cost shows up in storage. To answer any possible input, the whole table has to be available. A new paper from Hugging Face Daily Papers proposes a format called Conditional Program (CP), which stores rows at two bits per code and reconstructs only the rows an input requests. For DeepSeek's table, the size falls from 203 to 51 GB, and output KL divergence comes out 41–63% lower than with naive 2-bit rounding.

How CP works

The idea is to stop quantizing the table as one monolithic tensor and instead describe each row with a compact record. Every row has a short code and a local scale. The program that turns a code back into a vector is the same for all rows. It is stored once for the whole table and adds almost nothing per row.

The authors build this from three components. The first is a Gaussian trellis: a structured codebook in which a code selects a path through the trellis rather than an entry in an explicit list. The second is norm restoration: after decoding, the vector's length is returned to its original value so that quantization does not shift the scale of activations. The third is independently decodable row entries. This is the key requirement for conditional memory: reading one row should not require decompressing neighboring blocks.

That last property is what sets this problem apart from ordinary weight compression. There, decoding happens in batches, layer by layer. Here, access is random and sparse, and the format has to support arbitrary lookups.

What the measurements showed

The authors analyzed three families of trained n-gram memory. Their conclusion is twofold. The statistics of individual coordinates are close to Gaussian, which is why a Gaussian trellis fits. At the same time, learned relationships remain between rows, and they cannot be written off as noise. Even so, at the same storage budget the shared trellis beat both data-fitted codebooks and the low-rank residual variants the authors tested.

Key figures from the abstract:

• code: 2 bits per element plus a local scale per row;
• output KL is 41–63% lower relative to 2-bit round-to-nearest;
• DeepSeek table: 203 GB → 51 GB, roughly 4x;
• format: a shared reconstruction program, independent row entries;
• validated on three families of trained n-gram memory.

A note on the fourfold gain. If the original table is stored at 16 bits, a pure 2-bit code would give eightfold compression. The abstract does not say where the difference comes from. Overhead from the scales and row entries may explain part of it, but this is worth checking against the full text of the paper.

In practice: where the freed memory goes

The paper itself states the main practical benefit: when the table sits on a GPU, the freed gigabytes can go toward larger workloads. That means longer context, more concurrent requests, or a bigger batch. GPU memory almost always runs up against the KV cache, so the 150 GB saved on the table translates directly into headroom for it.

Teams deploying models with conditional memory have one more option. A 51 GB table is easier to keep entirely on the accelerator than a 203 GB one, which avoids moving it to host memory and paying a latency penalty on every lookup. That removes one of the main arguments against n-gram memory in production.

Caution is also warranted. The quality metric in the abstract is the KL divergence of the output distribution, not a set of task benchmarks. Before building CP into an inference stack, it is necessary to see how the 2-bit table behaves on real tasks and what row decoding costs on a GPU under heavy load. But the direction is clear: conditional memory now has a way to pay for capacity in tens of gigabytes rather than terabytes.