DeepSeek, Google and Meta: 10 Ways to Compress the LLM KV Cache and Cut Inference Memory

The KV cache has gone from a supporting detail to one of the main bottlenecks for production LLM inference. A new survey collects 10 techniques that reduce memory use without fully retraining the model and, in some cases, noticeably speed up generation.

Where LLMs hit the wall

The longer the context and the more simultaneous requests a model serves, the faster the KV cache balloons. It is the intermediate store of keys and values for the attention mechanism. The survey gives a telling example: for a 30-billion-parameter model with batch size 128 and a 1024-token input, the KV cache can take up to 180 GB of memory. Even for a 7B model, the weights occupy about 14 GB of GPU memory while the cache takes roughly 72 GB, so the generation mechanism itself starts to cost more than storing the parameters.

That is why KV cache optimization has become a field of its own rather than minor tuning. Compressing the cache lets you raise batch size, serve more users on the same GPU and avoid memory limits with long prompts. An important advantage is that a significant share of the methods work at the inference stage: the base model does not need to be retrained, and the effect shows up immediately in throughput and serving cost.

How the cache gets compressed

Researchers currently use several strategies, and they differ not only in quality but in where they sit in the pipeline. Some methods discard the least useful tokens, others reduce the precision of the cache representation, and still others change the attention architecture itself. A separate class redistributes memory across layers, because early layers need richer context while deep layers can work with a smaller set of keys and values. In effect, the question is no longer about percentage savings but about being able to run long context on the same hardware.

• Token eviction: H2O, StreamingLLM and SnapKV keep only part of the states. H2O keeps "heavy" tokens that contribute most to attention, StreamingLLM keeps the first tokens plus a recent window, and SnapKV picks important positions based on attention at the end of the prompt.
• Per-layer budget allocation: PyramidKV and PyramidInfer assume deep layers do not need context as rich as early layers do, so memory is allocated unevenly.
• Quantization: KIVI, KVQuant and TurboQuant reduce the precision of the KV cache while trying not to lose generation quality.
• Architectural changes: MQA, GQA and MLA shrink the cache within the attention design itself, rather than on top of an already finished model.
• Low-rank compression: Palu, LoRC and similar methods cut the hidden dimension of the KV tensors rather than the sequence length.

The easiest to operate are the methods that need no fine-tuning. H2O finds the tokens that gather most of the attention and drops the weak positions. StreamingLLM keeps the first tokens as "attention anchors" plus a recent window, which suits endless conversations but risks losing an important fragment from the middle. SnapKV works at the prefill stage and selects important positions separately for each attention head, so it usually beats cruder schemes at the same cache limit.

Attention weight distributions often resemble a power law, so removing

low-contribution tokens does not always hurt quality much.

Who delivers the biggest gain

In quantization, KIVI, KVQuant and TurboQuant stand out. KIVI converts the KV cache to a 2-bit representation without fine-tuning and, according to the survey, delivers up to 2.6 times lower peak memory use for weights plus cache combined, while allowing batches up to four times larger. KVQuant goes further: it uses calibration, mixed precision and separate handling of outliers to hold quality even at extremely long contexts.

The most aggressive result in the survey is attributed to TurboQuant from Google Research. The method first evens out the distribution of values with a random orthogonal rotation, then corrects the quantization error so that the estimate of dot products stays unbiased. On an H100 it shows at least a 6x memory reduction and up to 8 times faster attention at 3-bit precision. For infrastructure teams this is no longer a local optimization but a bid for a new serving standard.

A separate line of work changes the model architecture itself. GQA has become the de facto norm for modern open-weight LLMs: Llama 2 used it only in the 70B version, while Llama 3 extended it to both 8B and 70B. DeepSeek's MLA goes further: instead of full-size keys and values for every token, it stores a compressed latent representation. The survey says DeepSeek-V2 used MLA to cut the KV cache by 93.3% compared with the previous dense 67B model.

What this means

The LLM market is increasingly constrained not just by weight size but by the price of memory at long context. For teams building inference services, the takeaway is simple: no single magic technique delivers the gain now. What works is a well-judged choice among eviction, quantization and architecture for a specific workload, SLA and GPU budget.