Google unveils TurboQuant, a 3-bit KV cache for LLMs, but the memory market panicked too early

Google Research has introduced TurboQuant, an algorithm that promises to compress the KV cache of language models to 3 bits without fine-tuning and with almost no loss of quality. These claims quickly spooked the memory market, even though a real revolution in data centers is still far off.

Why the market jumped

On March 24, 2026, Google Research published a write-up on TurboQuant, and two days later investors began selling shares of memory-related companies. SK Hynix lost 6.23%, Samsung 4.8%, and Micron and SanDisk in the US fell by roughly 5% and 8%. The market's logic looked straightforward: if large models can get by with several times less memory for inference, demand for HBM and DRAM in data centers should fall as well. But that reading was too crude and ignored where exactly TurboQuant delivers its gains.

The problem the algorithm targets concerns not all of a model's memory, only the KV cache. These are the auxiliary token representations a transformer stores during text generation so it does not have to recompute them at every step. With short context the KV cache hardly matters, but with long context it becomes the main consumer of memory. For large models with windows of tens and hundreds of thousands of tokens, the cache can take up tens of gigabytes and become a bottleneck in high-volume inference.

What TurboQuant does

The industry learned to quantize model weights long ago, with GPTQ, AWQ and other approaches. The KV cache is harder, because it is created in real time and is unique to each request. You cannot prepare the data once, calibrate a scheme and then simply apply it everywhere. You need a method that can compress any new vector on the fly without wrecking answer quality on long contexts. That is the task TurboQuant tries to solve.

Google's scheme has two stages. First, the PolarQuant stage rotates the vector with a random orthogonal matrix so that the distribution of values becomes more even and predictable. After that, a precomputed optimal quantizer can be applied without calibration data. Then the QJL stage kicks in, encoding the sign of the residual error in a single bit and reducing systematic bias in dot products. As a result, error accumulates less noticeably over a long token sequence, and the model better preserves answer quality.

• 3-bit representation of the KV cache without retraining the model
• Up to 8x faster computation of attention logits on H100, according to the authors
• At least 6x less VRAM for the KV cache itself
• No mandatory offline calibration for a specific model

Where the method hits its limits

The most important caveat is that the technology does not yet look like an industrial standard. The community has already noticed that on small models of up to 3B parameters, aggressive 3-bit compression can noticeably hurt quality, cause repetition and degrade coherence. For many practical scenarios, 4-bit mode remains the safer option.

In addition, Google has so far published only a blog post and a preprint. There is no official implementation yet, and as of April 29, 2026 the algorithm is not built into vLLM, llama.cpp or SGLang.

There is also a scientific dispute over priority. Jianyang Gao, one of the authors of the earlier RaBitQ algorithm, said TurboQuant is too close to their approach while describing its predecessor incorrectly. His complaints include downplaying the methodological similarity, disputed criticism of RaBitQ's theory, and an unequal comparison: TurboQuant was tested on an A100 GPU, while RaBitQ in one benchmark ran on single-threaded Python. A complaint has already been filed with the ICLR ethics committee, and Google has not yet given a public response.

What this means

TurboQuant looks not like a crash for the memory market but like a strong improvement at one specific bottleneck of LLM inference. If Google releases code and the method makes it into standard stacks, long contexts will get cheaper and running large models on more modest hardware will become more realistic. But right now it is an important research result rather than a ready-made industry upheaval.