Starts Nov 23Live Cohort 3 of Vibe Coding & AI Agents: 3 weeks, 9 live hands-on sessionsCohort 3 (Nov 23) →
InfrastructureInsight · On RadarAWS Machine Learning Blog→ original

AWS adds Dynamo to HyperPod: Qwen3-235B delivers 207 tokens/s per GPU instead of 101

A large model doesn't fit on a single server, and one customer's long prompt slows down everyone else's responses. AWS has added support for NVIDIA Dynamo to SageMaker HyperPod. Dynamo assigns prompt reading and response generation to separate groups of GPUs. On Qwen3-235B with 16 H200 GPUs, the result was 207 tokens per second per GPU instead of 101, a 2.05x gain. If you run your own LLM on AWS, test disaggregated mode on real traffic.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
AWS adds Dynamo to HyperPod: Qwen3-235B delivers 207 tokens/s per GPU instead of 101
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Two ways to deployyour own DynamoGraphDeployment or an IEC with dynamoSpec
Key metric #2Qwen3.5-397B on two nodes retained 99.1% of single-node speed
Engine choicesvLLM (default), SGLang, TensorRT-LLM

Picture a chatbot built on a large language model. One user pastes a hundred-page document into a prompt, and while the GPUs "read" it, every other customer waits for the next words of their own answers. There is a second problem: the model weights don't fit into the memory of a single server at all.

Amazon has added support for NVIDIA Dynamo, an open-source distributed inference framework, to SageMaker HyperPod Inference, the AWS-managed platform for running models on GPU clusters. Dynamo splits prompt reading and response generation across different groups of GPUs. In a test on Qwen3-235B, this produced 207 output tokens per second per GPU versus 101 with the conventional setup, a 2.

05x gain under the same latency constraints.

For scale: the FP8 weights of GLM-5.3, which AWS uses as its example, do not fit into the 640 GB of memory on a single ml.p5.48xlarge instance. The model also needs room for the KV-cache (the model's memory of the text it has already read) and for runtime overhead.

What Dynamo actually does

First, it splits the work into two phases. In prefill, the model "reads" the prompt; in decode, it generates the response tokens one at a time. The two phases load the hardware differently, and when they share the same GPUs, a long prompt slows down generation for other users. Dynamo allocates a separate worker pool to each phase, and the pools can be scaled independently. The KV-cache moves between pools over the fast EFA network.

Second, it supports multi-node deployment. A single worker can span several instances if the model doesn't fit on one. This works in disaggregated mode and in regular (aggregated) mode, where both phases run on the same worker.

Third, it offers KV-aware routing. The router sees which worker already holds the cache for the shared beginning of a prompt, and it also takes that worker's load into account. For example, if all requests share the same long system prompt, it doesn't need to be recomputed each time. Time to first token drops, and overloaded workers don't get extra requests.

HyperPod adds what teams usually have to assemble by hand: health monitoring for nodes, GPUs and EFA, replacement of faulty nodes, and coordinated pod placement. For multi-node workers, the Kubernetes AI scheduler (KAI) uses gang scheduling to start all the required pods at once, so a worker doesn't hang half-launched.

Two ways to deploy

If you already work with Dynamo, you can bring your own DynamoGraphDeployment (DGD). The documentation and recipes stay the same, but you configure storage, networking, pod placement and autoscaling yourself. If you use HyperPod Inference, an InferenceEndpointConfig (IEC) with a dynamoSpec section is enough. The operator pulls the weights from S3 or FSx, brings up an endpoint with TLS, and connects autoscaling through KEDA.

The configuration is short. For two instances (16 GPUs), you set nodesPerReplica: 2 and tensorParallelism: 16, and routingMode: kv-aware turns on smart routing. In disaggregated mode, the aggregation block is replaced by prefill and decode, each with its own replica count. The AWS example uses one prefill replica and two decode replicas of two instances each, six instances in total. Requests to the endpoint go through SageMaker Runtime in an OpenAI-compatible format.

What the benchmarks showed

The first test ran Qwen3-235B-A22B-FP8 on TensorRT-LLM across two p5en.48xlarge instances (16 H200 GPUs), with requests of 4000 input and 200 output tokens. Only configurations where P90 time to first token was under one second and median generation speed was at least 50 tokens per second per user were counted. The best disaggregated result was 207 tokens per second per GPU at concurrency 80; the best aggregated result was 101 at concurrency 32.

The second test checked whether inter-node communication carries a penalty. Qwen3.5-397B-A17B-FP8 on vLLM with TP4 × PP2 was run on one instance with eight GPUs and on two instances with four GPUs each. On an agentic workload with frequent prefix repetition at concurrency 32, the two-node setup delivered 126.4 tokens per second per GPU, or 99.1% of the single-node result. This model fits on one instance, so in practice you shouldn't split it. The test served as a sanity check: crossing the node boundary costs almost nothing.

What to do with this

The figures come from AWS and NVIDIA themselves, on two models and specific traffic profiles, so you shouldn't assume 2.05× will carry over to your own system. Disaggregated mode makes the most sense to try with long prompts and strict latency limits.

The authors' recommendations are practical. Take the Dynamo recipe for your model, engine and GPU. Before measuring, confirm that the KV-cache moves between workers over EFA. Run AIPerf on real traffic at different concurrency levels and keep the data points that fit your SLAs. Start tuning with topology and worker distribution, and only then move on to engine parameters, changing one variable at a time.

Story

Key Questions Answered

What Dynamo actually does?

First, it splits the work into two phases. In prefill, the model "reads" the prompt; in decode, it generates the response tokens one at a time. The two phases load the hardware differently, and when they share the same GPUs, a long prompt slows down generation for other users. Dynamo allocates a separate worker pool to each phase, and the pools can be scaled independently. The KV-cache moves between pools over the fast EFA network. Second, it supports multi-node deployment.

What are the key details on Two ways to deploy?

If you already work with Dynamo, you can bring your own DynamoGraphDeployment (DGD). The documentation and recipes stay the same, but you configure storage, networking, pod placement and autoscaling yourself. If you use HyperPod Inference, an InferenceEndpointConfig (IEC) with a dynamoSpec section is enough. The operator pulls the weights from S3 or FSx, brings up an endpoint with TLS, and connects autoscaling through KEDA. The configuration is short.

What the benchmarks showed?

The first test ran Qwen3-235B-A22B-FP8 on TensorRT-LLM across two p5en.48xlarge instances (16 H200 GPUs), with requests of 4000 input and 200 output tokens. Only configurations where P90 time to first token was under one second and median generation speed was at least 50 tokens per second per user were counted. The best disaggregated result was 207 tokens per second per GPU at concurrency 80; the best aggregated result was 101 at concurrency 32.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…