Freezing the Decoder in KV-Cache Compression Saves 3x Memory but Doesn't Improve Accuracy

The researchers behind arXiv preprint 2610.10552 show that a popular conclusion about KV-cache compression, that only the encoder needs fine-tuning and the decoder can stay frozen, rested on a methodological error. With a shared learning rate, the encoder-only variant beat the alternatives with apparent statistical significance across several seeds. After each variant got its own tuned learning rate, the gap vanished. What remains is what was actually measured: three times fewer trainable parameters and three times less memory for optimizer state at the same quality.

How a phantom advantage appears

Some context. Post-processing a pretrained model for a compressed cache works like this: the key and value matrices are factored with SVD into two multipliers. The first, the down-projection, the authors call the encoder. It compresses the representation into a low-rank latent space. The second, the up-projection, they call the decoder. It expands the latent back out. The result is a cache in the spirit of multi-head latent attention, only without training from scratch. Truncating the rank hurts accuracy, so short fine-tuning follows, known as "healing."

Here is the flaw in the comparison. The healing variants differ sharply in the number of trainable parameters: encoder only, decoder only, or both multipliers. If every variant gets the same learning rate, it happens to suit the smallest variant and misses for the larger ones. According to the authors, the larger variants end up with a depressed mean and inflated variance. The result is a large, seemingly reproducible advantage for the smallest parameter set. There is no real effect behind it.

What was tested and what came out

The main experiment ran on the multimodal model Qwen2.5-VL-3B-Instruct. Each variant had its learning rate tuned separately and was run with three seeds per configuration. The result was reproduced on a text-only setup with two different base models.

• Method: SVD factorization of the key/value weights into an encoder and a decoder, followed by short fine-tuning.
• Variants compared: healing the encoder only, the decoder only, and both multipliers.
• Protocol: individually tuned learning rate, three seeds per configuration.
• Quality result: the encoder-only variant reaches parity with the alternatives.
• Resource result: 3 times fewer trainable parameters and 3 times less memory for optimizer state.
• Limitation: the VLM check covers a single compression ratio.

The caveat about the compression ratio appears in the abstract itself and should not be skipped. Parity is confirmed at one point, and extending it to more aggressive compression remains a hypothesis. Replication on text models adds confidence, but the models there are small, and the paper offers no data on larger scales.

What to change in practice

For teams converting already-trained models to a compressed cache, the takeaway is practical. Healing only the encoder makes sense, but for reasons of savings: a frozen decoder takes two-thirds of the parameter and memory load off the optimizer. That matters when fine-tuning runs on a limited number of GPUs and the model already fills nearly all the memory. Do not expect an accuracy gain, though.

The second recommendation is broader and applies to any comparison of fine-tuning recipes, including LoRA-like schemes. If the variants have different numbers of trainable parameters, a shared learning rate cannot be treated as a neutral condition. It quietly tilts the result in favor of one of the contenders. The minimum hygiene: a separate learning-rate sweep for each variant and several seeds. Then the smaller variant won't earn undeserved credit, and the team won't spend weeks reproducing an effect that doesn't exist.

If you plan to apply the method at a different compression ratio, recheck the parity on your own data. An extra run with a tuned learning rate costs less than picking the wrong recipe in production.