vLLM Teaches Decode Nodes to Send KV Cache Back, Cutting TTFT by Up to 3.3x

The vLLM team has added bidirectional KV cache transfer to disaggregated inference. Prefill nodes can now pull the already computed cache of earlier conversation turns from decode nodes instead of recomputing the entire context on every turn. On long multi-turn prompts, time to first token (TTFT) drops by up to 3.3x.

Where the time was lost

In a disaggregated serving setup, the prefill and decode phases run on separate GPU pools. Prefill processes the prompt and builds the KV cache, then hands it to a decode node, which generates the response token by token. The split makes sense: prefill is compute-bound, decode is bound by memory and bandwidth, and each pool can be tuned for its own workload.

Cache transfer, however, was one-way, from prefill to decode. That created redundant work on the second and later turns. The client sends the full conversation history plus the new message. The cache for that history already sits on the decode node, but the prefill node doesn't know about it and recomputes everything.

For an ordinary chat, this is tolerable. For agents, it is not. An agent loop consists of dozens of steps: a tool call, a response, reasoning, the next call. The context grows to tens of thousands of tokens, while the new portion at each step is small. We pay the full prefill price for something that was computed a minute ago.

How the return channel works

The idea is simple. After generating a response, the decode node keeps the KV cache for both the prompt and the generated tokens. When the next turn arrives, the prefill node can request the matching prefix from it, receive it, and compute only the new tail. From there, everything works as before: the full cache goes to the decode node.

Instead of recomputing the entire history, the system transfers already computed tensors over the network. The gain appears wherever transfer over the interconnect is cheaper than recomputation. The longer the shared prefix, the bigger that gap.

Key figures from the announcement:

• TTFT speedup: up to 3.3x;
• scenario: long multi-turn prompts;
• what is saved: repeated prefill compute over already processed history;
• transfer direction: decode → prefill in addition to prefill → decode;
• platform: vLLM in a configuration with separate prefill and decode nodes.

The 3.3x figure is presented as an upper bound. It is the best case on long conversations, not a guaranteed average result. In short exchanges with a small prefix, the difference will be noticeably more modest.

What the announcement leaves out

The short version of the announcement lacks the details that determine the real-world effect: which transport and connector are used, what hardware and model the measurements were taken on, how network load changes, and how the scheme affects overall throughput. They are most likely covered in the full blog post, and it is worth reading it before adopting the feature.

There is also an architectural question. For a decode node to return the cache, it has to retain it, and GPU HBM is limited. The longer sessions live, the greater the memory pressure and the more important the eviction policy becomes. If the cache has already been evicted, there is no gain, and prefill falls back to full recomputation.

Who will benefit in practice

Primarily teams that already run vLLM in a disaggregated configuration and serve agents, coding assistants, or other long-running sessions. For them, TTFT at the later steps of the loop determines how fast the product feels.

Before adopting it, check the following against your own traffic:

• what share of the prefix repeats between turns, and how that share grows over the course of a session;
• whether the interconnect bandwidth between prefill and decode nodes is sufficient;
• how long decode nodes hold the cache, and how often it is evicted before the next turn;
• how p95/p99 TTFT changes, not just the best result.

If prefixes are long and sessions fit within the cache retention window, the effect may come close to the claimed figure. If dialogues are short or disjointed, the benefit will be smaller, and the added complexity of the scheme may not pay off.