Google unveils TurboQuant: 6x LLM KV cache compression with no accuracy loss

Google Research has unveiled TurboQuant, a KV cache compression algorithm for large language models that aims to remove one of the main constraints on long context. According to the company, the method cuts memory consumption at least sixfold and, in some configurations, speeds up attention computation by up to eight times, with no quality loss on test tasks.

Why the KV cache is a bottleneck

When an LLM works with a long context, it stores intermediate keys and values in a KV cache so it does not have to recompute them for every token. This saves computation, but it quickly runs into memory limits: the larger the model and the longer the conversation or document, the faster the cache grows. As a result, the bottleneck is not only the GPU itself but also data transfer between fast SRAM and HBM memory. For inference this is especially painful, because long requests become noticeably more expensive in both latency and hardware.

Google compares the KV cache to a "high-speed digital cheat sheet"

that the model uses instead of repeating computations.

Conventional quantization partly solves the problem, but it has a side effect: extra quantization constants must be stored alongside the compressed data. These overhead bits eat into the gains, especially with billions of values in a long context. This is where Google builds TurboQuant: the idea is not simply to compress vectors harder, but to remove the extra overhead that prevents real memory savings.

How TurboQuant works

TurboQuant has two stages. First comes PolarQuant: the algorithm randomly rotates the vectors, converts them into a more convenient representation, and then quantizes each coordinate separately. This preserves the main structure of the original data without complex tuning for each block.

The second layer is Quantized Johnson-Lindenstrauss, or QJL. It takes the residual error left after the first stage and encodes it with one additional bit, removing the systematic bias when computing the inner product and attention score.

In practice this matters for two reasons. First, TurboQuant remains data-oblivious: it needs no calibration datasets, extra training or fine-tuning for a specific model. Second, the method suits online scenarios, where the cache must be compressed during inference itself rather than through a separate offline pipeline.

Google stresses that the scheme is useful not only for LLMs but also for vector search, which likewise needs to store and compare large arrays of embeddings quickly and cheaply. The company is preparing to present TurboQuant itself at ICLR 2026.

What results Google reports

Google tested TurboQuant on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER and L-Eval, using the open models Gemma and Mistral. According to the company, TurboQuant preserves quality on long-context tasks while noticeably shrinking the KV cache. Google's blog emphasizes 3-bit quantization with no quality loss on the tested benchmarks, while the abstract of the arXiv paper separately notes full quality preservation at 3.5 bits per channel and only slight degradation at 2.5 bits.

• at least 6x reduction in KV cache memory
• up to 8x speedup in computing attention logits on an H100 in a 4-bit configuration, compared with unquantized 32-bit keys
• no need for retraining, fine-tuning or calibration datasets
• strong results in vector search too: TurboQuant beat the PQ and RabbiQ baselines on recall on the GloVe dataset

Google also bets on applying the method to search. TurboQuant, PolarQuant and QJL reduce not only memory use but also index build time, while preserving nearest-neighbor search accuracy. That makes the technology interesting not only for generative models but for any infrastructure that handles huge vector collections: from semantic search to recommender systems and the retrieval layer of AI products.

What this means

TurboQuant suggests that the next big gain for LLMs may come not from new model parameters but from smarter memory handling. If Google's approach is confirmed in production and lands in popular inference stacks, long context will become cheaper, faster and more accessible even without a hardware upgrade. For developers, it is a chance to fit longer sessions and RAG scenarios into the same GPU budget, and for users, to get more stable answers on large documents and long conversations.