How Much Memory a Model Will Eat: How GQA, Sliding Windows, and MLA Change the KV-Cache

A familiar situation: you spin up an off-the-shelf model, the weights fit comfortably in GPU memory, and the server crashes with an out-of-memory error at the tenth concurrent request. The model size is not to blame. The KV-cache is. Its size depends on how the attention layer is built, and you can no longer change that: the decision was made at training time.

The checkpoint fixes the architecture

The config and weights define the number of query heads and key-value heads, the head size, the local attention pattern, and the dimensions of the latent representation. Swapping MHA for GQA after training means changing the model, and you will most likely have to recover quality with fine-tuning. Replacing full attention with a sliding window means throwing away context the model is used to seeing.

So the practical routine is this: open the config (in Hugging Face, that is AutoConfig) and look at the number of layers, num_attention_heads, num_key_value_heads, head_dim, and the sliding-window fields. One caveat: the window field may be present in the config but disabled by a neighboring flag, so check both the value and the flag. Some models alternate local and global layers, and the ground truth is always in the implementation code.

How to calculate the cache on a napkin

Persistent memory lives between decoding steps, and it determines how many requests you can serve at once. Temporary buffers, such as tiles of attention scores, can be freed after the layer finishes. For a standard cache the formula is: 2 × bytes per element × layers × tokens × number of KV heads × head size. The 2 stands for keys plus values. For a batch, multiply by the batch size.

Take a hypothetical model: 32 layers, 32 heads, head_dim 128, fp16, a context of 8192 tokens. With classic MHA, one request takes 4 GiB. If the model uses GQA with 8 KV heads, that drops to 1 GiB. With MQA and a single KV head, it is just 128 MiB. That is the difference between fitting 4 requests and fitting 16, with no change in parameter count.

Calculate mixed schemes layer by layer. Say the same 32-layer GQA model has 24 local layers with a 4096 window and 8 full layers, and a context of 32 768 tokens. Full attention everywhere would give 4 GiB, while the mixed scheme gives about 1.4 GiB. But the total cache is still not bounded: every new token adds memory in the full layers. It grows more slowly, but it does not stop growing.

Which attention variant is it?

• MHA: each query head has its own key-value pair. This is the most expensive cache, and during decoding it also means the heaviest memory reads.
• MQA and GQA: query heads share KV heads. The cache and the K/V projection parameters shrink by exactly the factor by which the number of KV heads shrinks.
• Sliding window: the layer remembers only the last W tokens, so the cache can sit in a fixed-size ring buffer. Positions are computed from the token's logical index, not from the buffer slot.
• MLA: instead of full keys and values, it stores a compressed latent vector. The cache is smaller, but decoding requires more computation. This usually pays off, because decoding is bound by memory bandwidth, not arithmetic.

Do not plug MLA into the MHA formula via num_key_value_heads: it caches different tensors. Calculate from what the implementation actually stores, and use specialized kernels.

What the runtime can change

It helps to separate four questions: who attends to whom (full or windowed attention), how many KV representations were learned (MHA/GQA/MQA), what gets cached (KV tensors or a latent), and how it is executed. The checkpoint decides the first three; the engine decides the fourth. FlashAttention computes exact attention in tiles and does not write huge score matrices to memory: temporary buffers and memory traffic shrink, while the logical cache and the number of KV heads stay the same.

Paged attention stores the cache in blocks that do not have to be contiguous, which improves allocation and sharing, but it does not reduce the cache size by itself. The allocator must also respect the model's eviction rule: you cannot drop old entries from a full layer just because neighboring layers use windows.

Practical takeaway

Before choosing a checkpoint, read the config and estimate the cache for your target context and batch. If two candidates are close in quality, pick the one with cheaper KV heads or windows: that translates directly into the number of concurrent requests per card. And do not forget about padding: with dense batches you may pay for empty positions, while a ragged implementation avoids this.

The limitation of this method is simple: the calculation estimates persistent memory, but it does not account for weights, activations, or allocator overhead. The numbers above are illustrative; for a real model, plug in the values from its config.