Galahad-kv stores KV cache on NVMe and loads a block 2.8–4.3x faster than recomputing it

Researchers have tested a memory layer for LLMs that does not recompute the prompt from scratch but retrieves a ready-made KV state from disk. They ran the galahad-kv package on 50M tokens of real public text through vLLM on a single NVIDIA H100 with Gemma 4 12B and Gemma 4 31B. All 100 of 100 tested blocks came back from storage without recomputation. Loading was 2.8–4.3x faster, and the GPU used 8.8–12.3x less energy.

How the memory layer works

A model uses only the text that fits in its context window. On top of that, every time a prompt is sent, it recomputes the internal KV state (key-value) from scratch, even if the text has not changed. On long contexts, this recomputation costs time and energy.

Galahad-kv splits the stream into blocks of roughly 16,000 tokens. The KV state of each block is written to an encrypted local NVMe drive and returned byte for byte on access. By a rough estimate, 50M tokens yield about 3,100 blocks. Storage is measured in terabytes.

The authors state up front that this is reuse of saved state. The attention window stays the same, only one block is loaded at a time, and answer quality depends on the model itself.

What the measurements showed

The test ran on a single H100 under vLLM, with tested block depths reaching 50M tokens. Key metrics:

• 100 of 100 tested blocks were loaded from encrypted storage without recomputation, on both models, at depths from 0 to 50M tokens;
• loading a block was 2.8–4.3x faster than recomputing it;
• GPU energy consumption was 8.8–12.3x lower;
• GPU memory stayed flat across the entire 50M-token stream;
• Gemma 4 12B answered 82 of 100 questions correctly, Gemma 4 31B 98 of 100;
• neither model made up an answer.

The last point matters as much as the speed. On questions about facts planted millions of tokens before the query, a model most often fails by answering confidently and off target. Here, errors came down to a missing correct answer, not to hallucinations. The gap between 82 and 98 percent shows that the memory layer delivers the state, but the model itself has to read it correctly. The larger model does so noticeably better.

Where the limits are and who benefits

The approach has three limitations. Writing the memory is a one-time cost: each block has to be computed and saved once. Storage needs terabytes of local NVMe. And only one block is loaded at a time, so questions that require combining facts from many places in the corpus at once are not covered by this test. This is a first-version preprint, and no independent reproductions are visible yet. But the authors describe a protocol designed to guard against the typical ways of gaming long-context benchmarks, and they provide a reproduction on a single GPU with public software and a free license for the package.

The practical takeaway for engineers is this. The approach makes sense where the same large body of text is read again and again: code repositories, documentation, archives, an agent's long history. There, prefix recomputation eats most of the GPU time. Before adopting it, it is worth working out three things: how many terabytes of NVMe your corpus will need, whether the one-time write pays off at your access frequency, and whether your model is good enough to read a loaded block accurately. One H100 and the public stack from the paper are enough to check. A run on your own data will quickly show whether the figures of 2.8–4.3x for speed and 8.8–12.3x for energy hold up.