Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureAPI / Product · ActionableAWS Machine Learning Blog→ original

AWS gives agents a shared LLM cache skill: up to 40% cheaper, first token 80% faster

If you host LLMs on GPUs, the server recalculates the entire conversation history (the KV cache) after a pause, and a lawyer waits 13 seconds for the first token. AWS has released a skill for Claude Code, Kiro and Codex that evaluates the benefit of a shared cache on ElastiCache and deploys it itself. The company claims up to 40% savings on inference and up to 80% less time to first token (TTFT). Estimate the benefit on your own traffic first.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
AWS gives agents a shared LLM cache skill: up to 40% cheaper, first token 80% faster
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1Works in Kiro, Claude Code, OpenAI Codex and other agents with Agent Skills
Key metric #2KVTrace calculates the benefit for your traffic for free before you launch
Key metric #3The agent creates and deletes resources only after confirmation

Picture a legal AI assistant running on your own GPUs. A single conversation has accumulated hundreds of thousands of tokens of case history. The lawyer returns to it after lunch, but the request lands on a server whose memory no longer holds that history. The GPU has to recompute the KV cache from scratch: the model's internal "notes" about every token of text it has already read. This stage is called prefill, and while it runs, the user sees nothing. In AWS's example, with a context of 64,000 tokens, the first token arrived after 13 seconds. This delay is commonly abbreviated as TTFT, or time to first token.

AWS proposes moving the cache out of individual servers' memory into a shared layer next to the GPU fleet, built on Amazon ElastiCache. Any server can then pick up the ready state, even if the request went to a different machine or the original server failed. According to AWS, such a distributed KV cache cuts inference costs by up to 40% and TTFT by up to 80%. These are the upper figures from the company's statement, not a guarantee for any workload. The news is that AWS has released a ready-made skill for setting this up, part of the Agent Toolkit for AWS. It works in Kiro, Claude Code, OpenAI Codex and other agents that support Agent Skills.

Four scenarios of how the skill works

AWS shows four dialogues with the agent.

In the first, the agent diagnoses a slow first token. It first looks for solutions that don't require new infrastructure. If you have disabled prefix caching, simply turn it back on. If the local GPU cache hit rate is already above 90% with sessions pinned to a server, a shared cache may not be needed. The agent also warns about a multi-tenancy risk: based on response timing, one client could guess that another client sent the same text. For that reason, cases that must be isolated are kept in separate caches, and when a shared cache is used, the tenant ID is added to the hash.

In the second scenario, the agent estimates the savings. For a fleet of 8 p5 instances handling requests of 64,000 tokens each, it took prices from the AWS price list and recommended two cache.i8ge.48xlarge nodes. It does not invent the cache hit rate; instead, it suggests measuring it with KVTrace.

KVTrace, setup and debugging

KVTrace records KV cache events from vLLM or SGLang and then replays them offline against shared caches of different sizes. The records contain block hashes and token counts, not prompt texts. The hashes are built from content, so for sensitive data AWS recommends the --anonymize-key-file flag. It is best to record on all servers over a representative period, such as a working day. The report recommends the smallest capacity that delivers 90% of the maximum savings. For short prompts, reading from the cache can take longer than recomputing, so prompt length and repetition frequency are what decide the outcome.

In the third scenario, the agent deploys the cache from scratch. After your confirmation, it creates a security group with EFA permissions and a Valkey 9.1 cluster with eviction policies.

For LMCache, it sets up a separate user, with the password stored in Secrets Manager. The agent then connects the LMCache connector to the existing vLLM or SGLang, including variants for llm-d and Ray Serve. It assigns all servers a single KV ID mechanism and an explicit data type, otherwise each server writes state the others cannot find.

Before production, it clears the local tiers and checks that two runs loading from the cache produce identical answers. If the data layout does not match, hit counters look good while the answers come out wrong.

In the fourth scenario, the agent analyzes a failure. The connection to ElastiCache is there, but data transfer fails and the rdma_write_errors counter grows. The likely cause is a security group blocking EFA traffic between its own members. The agent provides commands to verify and fix it.

Limitations and the practical takeaway

The requirements are strict: instances that support EFA RDMA write (for example, p5), LMCache 0.5.8rc1 or newer, vLLM 0.11.1+ or SGLang 0.5.13+, and ElastiCache for Valkey 9.1+ on cache.i8ge.48xlarge nodes. Freed-up GPU time turns into money only if you reduce the number of servers or serve more traffic on the same ones.

If you run long multi-turn conversations, RAG, or resumable agent sessions on your own GPUs, first capture a KVTrace log over a working day. Along the way, check the cheaper options: prefix caching and session pinning. A shared cache is justified when the calculation shows that the savings cover the cost of the nodes. The skill itself only automates the routine work and asks permission before each significant action.

Story

Key Questions Answered

What are the key details on Four scenarios of how the skill works?

AWS shows four dialogues with the agent. In the first, the agent diagnoses a slow first token. It first looks for solutions that don't require new infrastructure. If you have disabled prefix caching, simply turn it back on. If the local GPU cache hit rate is already above 90% with sessions pinned to a server, a shared cache may not be needed. The agent also warns about a multi-tenancy risk: based on response timing, one client could guess that another client sent the same text.

What are the key details on KVTrace, setup and debugging?

KVTrace records KV cache events from vLLM or SGLang and then replays them offline against shared caches of different sizes. The records contain block hashes and token counts, not prompt texts. The hashes are built from content, so for sensitive data AWS recommends the --anonymize-key-file flag. It is best to record on all servers over a representative period, such as a working day. The report recommends the smallest capacity that delivers 90% of the maximum savings.

What are the key details on Limitations and the practical takeaway?

The requirements are strict: instances that support EFA RDMA write (for example, p5), LMCache 0.5.8rc1 or newer, vLLM 0.11.1+ or SGLang 0.5.13+, and ElastiCache for Valkey 9.1+ on cache.i8ge.48xlarge nodes. Freed-up GPU time turns into money only if you reduce the number of servers or serve more traffic on the same ones. If you run long multi-turn conversations, RAG, or resumable agent sessions on your own GPUs, first capture a KVTrace log over a working day.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Hands-On AI Engineering · Hamidun Academy

Build & Ship Production AI Agents Without Boilerplate Code

Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.

Start 72h Free Trial →Vibe-Coding Intensive
What do you think?
Loading comments…