AWS launches managed NVIDIA Dynamo for models that don't fit on a single server
If a model's weights don't fit into the memory of any single GPU server, teams have to build an inference cluster by hand. AWS is taking that job on: NVIDIA Dynamo now runs in SageMaker HyperPod as a managed engine. One copy of a model stretches across several nodes, prompt processing and token generation run on separate GPU pools, and a failed node is replaced automatically. For teams running large reasoning models, this removes the most tedious part of operations.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Large reasoning models run into simple physics: the weights don't fit into GPU memory on any single server. Add long prompts with retrieval context, which slow down generation for other users, and you get a familiar headache for production teams. In November 2026, AWS announced the launch of managed NVIDIA Dynamo on SageMaker HyperPod Inference. It is part of a broader review of SageMaker Inference launches in 2026, but this release addresses the hardest engineering problem: serving models that are bigger than one machine.
What Dynamo is and why it was awkward to use
NVIDIA Dynamo is an open-source inference framework. It does three things: it spreads one copy of a model across several nodes (sharding), it separates prompt processing (prefill) from token generation (decode), and it routes each request to the place where the needed KV-cache already lives.
You could run Dynamo on your own Kubernetes before, but you had to build everything around it yourself. You had to place all nodes of a replica close to each other. You had to configure the Elastic Fabric Adapter (EFA) network so that prefill could pass the KV-cache to decode at hardware speed. You had to maintain server images and deal with the situation where a single failed node takes down an entire multi-node replica.
How the work is split now
Dynamo makes the serving decisions: how to shard the model, how to split prefill and decode, and where to send each request. HyperPod provides everything those decisions depend on:
- GPU capacity and the EFA network are provisioned and connected automatically;
- node health is monitored, faulty nodes are replaced, and multi-node workers are rescheduled on their own;
- autoscaling, endpoint registration, TLS and observability come out of the box;
- the Inference Operator add-on installs and updates Dynamo, the server images, and the Grove and KAI schedulers as a single package.
In practice it looks like this. A team with a reasoning model of hundreds of billions of parameters runs one replica across several nodes. Prefill and decode run on separate GPU pools, and you can pick a suitable instance type for each. A long prompt no longer slows down responses for other users. And if a node fails, HyperPod replaces it and restarts the replica, so no on-call engineer gets woken up.
KV-aware routing sends a request to the worker that already holds the needed cache. This reduces time to first token, especially in chats and RAG with repeating prefixes.
How to get started and what about compatibility
AWS describes three paths:
- If you already run Dynamo on your own Kubernetes, you can move your existing DynamoGraphDeployment (DGD) manifests over almost unchanged.
- If you are already on HyperPod Inference, it is enough to update the add-on and add a short dynamoSpec block to your existing InferenceEndpointConfig. HyperPod converts it into a DGD, and existing endpoints keep working.
- If you are new to both, you can create a cluster with the add-on and Dynamo enabled directly from the console.
Clients call a standard OpenAI-compatible endpoint, so client code does not need to change. The vLLM, SGLang and TensorRT-LLM engines are supported, and availability matches the regions where HyperPod is available.
Limitations and takeaway
The announcement includes no performance figures: AWS cites neither speedups nor cost savings. You will have to measure the effect of separate pools and KV routing on your own traffic. In addition, this is a path for HyperPod, with its clusters and Kubernetes approach. For simple models, AWS still recommends managed SageMaker endpoints.
The practical takeaway is simple. If you were planning to deploy Dynamo or multi-node vLLM yourself, first estimate how much engineering time you would spend on node placement, networking and failure recovery. That is exactly the part AWS is now selling as a service. In a pilot, check time to first token and cost at your actual mix of long and short requests. For teams planning to move to a large reasoning model, this is the fastest way to avoid becoming a GPU cluster operator.
- Nov 20, 2026Anthropic raised $104.8 billion in its IPO, but its infrastructure commitments total $1 trillion
- Nov 20, 2026Kimi Code CLI now works with Amazon Bedrock without a separate API key
- Nov 20, 2026Cisco's Agent Tournament: 1200 Engineers, and the Winner Wrote 1220 Tokens Instead of 1430
- Nov 20, 2026Amazon SageMaker Now Flags Idle GPUs and Rising Latency on LLM Endpoints on Its Own
Key Questions Answered
What Dynamo is and why it was awkward to use?
NVIDIA Dynamo is an open-source inference framework. It does three things: it spreads one copy of a model across several nodes (sharding), it separates prompt processing (prefill) from token generation (decode), and it routes each request to the place where the needed KV-cache already lives. You could run Dynamo on your own Kubernetes before, but you had to build everything around it yourself. You had to place all nodes of a replica close to each other.
How the work is split now?
Dynamo makes the serving decisions: how to shard the model, how to split prefill and decode, and where to send each request. HyperPod provides everything those decisions depend on: GPU capacity and the EFA network are provisioned and connected automatically; node health is monitored, faulty nodes are replaced, and multi-node workers are rescheduled on their own; autoscaling, endpoint registration, TLS and observability come out of the box; the Inference Operator add-on installs and updates…
How to get started and what about compatibility?
AWS describes three paths: If you already run Dynamo on your own Kubernetes, you can move your existing DynamoGraphDeployment (DGD) manifests over almost unchanged. If you are already on HyperPod Inference, it is enough to update the add-on and add a short dynamoSpec block to your existing InferenceEndpointConfig. HyperPod converts it into a DGD, and existing endpoints keep working. If you are new to both, you can create a cluster with the add-on and Dynamo enabled directly from the console.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.