SageMaker Now Routes LLM Requests to GPUs With a Warm Cache: First Token 97% Faster
When chat or RAG requests land on a random GPU, it recomputes a long prompt from scratch even though a neighboring GPU already holds the cache. The new KV_AWARE routing in SageMaker AI sends each request to the instance that holds the longest match. On Llama 3.1 70B, time to first token fell 97% and throughput rose 234%. If you run long conversations or RAG, it is worth testing on your own traffic.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Imagine a support bot built on an LLM: every conversation starts with the same 6 000-token chunk of a knowledge base. The model computes it once and keeps the result in GPU memory. This is called the KV-cache. If the next turn of the conversation lands on a different server that has no such cache, its GPU recomputes all 6 000 tokens, and the user waits for the first token of the reply. Amazon SageMaker AI has added a KV_AWARE routing strategy for real-time endpoints, which sends a request to the instance where the needed cache actually resides. On Llama 3.1 70B, this cut median time to first token (TTFT) by 97% and raised throughput by 234%.
What was wrong with prefix routing
SageMaker already had a PREFIX_AWARE strategy. It hashes the beginning of a request, a set number of bytes or characters (PrefixLength), and sends similar requests to the same instance. The router cannot see what is actually in the cache, so it gets things wrong in two cases.
First: GPU memory fills up, the instance evicts old cache entries, and the router keeps sending requests there, assuming the cache is still warm. Second: in a multi-turn chat, the history grows with every turn. While the request is shorter than PrefixLength, the hashed text changes on every turn, so the next turn may go to a different instance. A conversation only "sticks" to one server once the request grows longer than PrefixLength.
How KV_AWARE works
The model container reports what it holds in its cache: it publishes an event every time it stores or removes entries. The SM_VLLM_KV_EVENTS_CONFIG variable turns these events on. SageMaker collects the events into a managed index. The index contains no prompts, token IDs, request bodies or KV tensors, only irreversible truncated hashes and pointers to the instance or inference component copy where the entries reside.
When a request arrives, the endpoint tokenizes the prompt and looks up its tokens in the index. It then weighs the match length against the instance's load. If the best candidate is busy, the request goes to the next free one. If there are no useful matches, or all instances are busy, the endpoint falls back to ordinary load-based balancing, as in LEAST_OUTSTANDING_REQUESTS.
An example from AWS: the fourth turn of a conversation is 6 200 tokens long, and instance A holds the first 6 000, so A only has to compute the remaining 200. In a second support-bot scenario under load, A has evicted almost all of the knowledge-base chunk, but instance B still remembers its first 4 000 tokens. A prefix router would send the request to A, while a KV-aware router sends it to B. Under low load there is no cache eviction, and both strategies pick A. Matching is done by tokens, so there is no concurrency threshold to tune.
Numbers and limitations
AWS ran Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances. The tests lasted 30 minutes each, with 20 or 100 virtual users, multi-turn chats and document Q&A, and prompts up to 64 000 tokens. Compared with random routing, with long prompts under load, KV_AWARE cut p50 TTFT by 97%, raised throughput by 234%, and lifted the cache hit rate from 3–48% to 80–93%. Long prompts also benefited at lower load. On short prompts the gain is minimal: there is almost nothing to recompute.
What to keep in mind:
- The strategy only works with two or more instances or two or more copies of an inference component. With a single instance, it changes nothing.
- You need a supported container (the AWS vLLM Deep Learning Container) with prefix caching and KV-cache events enabled.
- The router adds a few milliseconds of latency on a short prompt and more on a long one, mostly due to tokenization. For the prefix strategy, this overhead is 1.3–1.9 ms.
- After scaling or an update, new instances start with an empty cache, and latency is higher for a while.
- Fallbacks to load-based balancing are not visible in the console or metrics at launch, so you will have to verify the effect yourself.
What to do in practice
Enabling it takes two steps: add the environment variable to the model, and set RoutingStrategy to KV_AWARE in the RoutingConfig of the endpoint variant. Clients call the endpoint with the same APIs.
If you have a stable shared system prompt, short requests and moderate load, PREFIX_AWARE is enough: it works with any container. KV_AWARE is for cases with long context, high load and not enough GPU memory for the cache: long conversations, RAG with large documents, and assistants with heavy prompts. The numbers from the blog depend on load, so run your own traffic through two endpoints and compare TTFT, throughput and cache hit rate.
- Nov 19, 2026AI Models Added a School to the Target List in 70% of Vals AI Tests
- Nov 19, 2026Q.ANT and AMD Run a 1B-Parameter LLM With an Optical Chip in Place of Attention
- Nov 19, 2026The Word "Video Game" Raised GPT-6 Astra's Willingness to Pick Targets From 6% to 92%
- Nov 19, 2026AWS adds a leaderboard of 70+ models to SageMaker: quality, speed and price in one table
Key Questions Answered
What was wrong with prefix routing?
SageMaker already had a PREFIX_AWARE strategy. It hashes the beginning of a request, a set number of bytes or characters (PrefixLength), and sends similar requests to the same instance. The router cannot see what is actually in the cache, so it gets things wrong in two cases. First: GPU memory fills up, the instance evicts old cache entries, and the router keeps sending requests there, assuming the cache is still warm. Second: in a multi-turn chat, the history grows with every turn.
How KV_AWARE works?
The model container reports what it holds in its cache: it publishes an event every time it stores or removes entries. The SM_VLLM_KV_EVENTS_CONFIG variable turns these events on. SageMaker collects the events into a managed index. The index contains no prompts, token IDs, request bodies or KV tensors, only irreversible truncated hashes and pointers to the instance or inference component copy where the entries reside. When a request arrives, the endpoint tokenizes the prompt and looks up its tokens in the index.
What are the key details on Numbers and limitations?
AWS ran Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances. The tests lasted 30 minutes each, with 20 or 100 virtual users, multi-turn chats and document Q&A, and prompts up to 64 000 tokens. Compared with random routing, with long prompts under load, KV_AWARE cut p50 TTFT by 97%, raised throughput by 234%, and lifted the cache hit rate from 3–48% to 80–93%. Long prompts also benefited at lower load. On short prompts the gain is minimal: there is almost nothing to recompute.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.