PrefixSim: open-source KV cache simulator compares 40+ eviction policies

If you run LLMs in production, you have almost certainly heard of prefix caching. The idea is simple: many requests share the same beginning—a system prompt, tool descriptions or conversation history. There is no need to recompute the KV cache for that prefix every time; it can be stored and reused. But GPU memory is finite, and sooner or later the cache has to evict something. What you discard determines how much GPU time goes to waste.

That is the problem PrefixSim aims to address. Released as an open-source project by Juncheng Yang (@1a1a11a), it is a fast KV block cache simulator for LLM serving. The authors also released two datasets of workload traces. One caveat: the original post on X cuts off mid-sentence, so the account below includes only what it explicitly states. Details such as trace formats and licensing should be checked in the repository.

The problem it addresses

Traditional cache policies such as LRU were designed for web pages and disk blocks. KV blocks have different characteristics. They form a prefix tree: a block at the beginning of a chain is needed by every block that follows it. Evicting an “old” root block means rendering an entire branch unusable. Different blocks also carry different computational costs: recomputing a long prefix is much more expensive than recomputing a short one.

That makes the familiar hit rate metric potentially misleading here. The PrefixSim authors explicitly identify compute savings as the main criterion: how much prefill work was avoided. Two policies with the same hit rate can deliver very different GPU savings.

How it works in practice

Imagine a chat assistant with a long system prompt and tens of thousands of sessions a day. You take a real request trace and replay it through the simulator using different eviction policies. According to the author, there are more than 40. The results show which policy saves the most computation for the amount of memory you have allocated to the cache.

The key word is “real.” Synthetic workloads often produce impressive numbers that do not hold up in production. Releasing two open datasets of serving traces is therefore useful in its own right: it gives the community a common basis for comparison instead of a collection of incompatible internal benchmarks.

Another reason the simulator is valuable is the cost of experimentation. Testing an eviction policy on a live cluster is slow and risky: you have to modify the engine, deploy the changes and wait for representative traffic. Simulating a recorded trace takes minutes and does not break anything.

What to do with it

If you are building your own inference stack or tuning vLLM, SGLang or similar engines, PrefixSim offers a way to check whether you are leaving money on the table. Record a trace of your workload, run it through the simulator and compare your current policy with the alternatives. If the difference in prefill savings is substantial, that makes a case for improving the cache. If it is not, you will at least know that the standard option meets your needs.

For product managers and founders, the useful takeaway is that cache efficiency directly affects time to first token and cost per request. For services with long shared prefixes—agents, RAG assistants and coding assistants—this can be a significant expense.

Limitations

A simulator is still a model. It does not account for everything that happens in a real engine: memory fragmentation, scheduling or queue overload. Results from traces collected at particular companies will not necessarily carry over to your workload. We have also yet to see independent validation: at publication, this was an announcement on X with around 7,500 views. Still, as both a tool and a collection of open datasets for a contentious and underappreciated area of LLM serving, the project looks worth an evening of exploration.