AWS shows how to run a single LLM across two GPU nodes on EKS with Ray LLM DLC

AWS has published a step-by-step guide to running a single large language model on two Amazon EKS GPU nodes. One node performs prefill, the other performs decode, and from the outside it all looks like a single OpenAI-compatible endpoint. The setup builds on multi-node Ray LLM Deep Learning Containers. These are ready-made images that already package the entire inference stack. Teams no longer have to assemble the components; what remains is the deployment configuration.

What exactly is being deployed

The layout is simple. There is one model, two GPU nodes in an Amazon EKS cluster, and two distinct stages of request processing split across separate machines. The client application knows nothing about the topology. It calls a single address using the familiar OpenAI protocol, and existing SDKs and tooling keep working without changes.

The point of the release is not the idea itself but the packaging. Previously, building multi-node inference meant picking compatible versions of the engine, orchestrator, drivers and networking libraries yourself. Now AWS ships all of this as a prebuilt container from its Deep Learning Containers line. Ray support in such an image makes sense: Ray has long served as an orchestration layer for distributed workloads, and its LLM extension is designed for serving models through a single API.

Why separate prefill and decode

Text generation has two phases with different load profiles. Prefill processes the entire input prompt. This is dense computation, and it is bound by GPU compute performance. Decode emits tokens one at a time and is more often bound by memory bandwidth and KV-cache size.

If both phases run on the same accelerators, they interfere with each other. One user's long prompt delays token output for everyone else. Splitting them across nodes lets you tune each side separately: parallelism, batch size and memory allocation can be matched to its own task. The cost of this approach is well known. The KV-cache has to be transferred between nodes, so the network and correct connector configuration matter. The container takes care of exactly this part.

Key parameters of the setup

• Model: one large LLM, distributed across nodes.
• Topology: two GPU nodes, one dedicated to prefill and the other to decode.
• Platform: Amazon EKS, i.e. standard Kubernetes in managed form.
• Interface: a single OpenAI-compatible endpoint for all clients.
• Delivery: a prebuilt multi-node Ray LLM Deep Learning Container with the full inference stack.
• Configuration approach: deployment configuration without manual component assembly.

In practice: what to check before adopting it

This architecture is not justified everywhere. For a small model that fits on a single node, disaggregation will only add network hops and points of failure. It makes sense where the model does not fit into the memory of one node, or where the traffic profile is sharply uneven. Examples include long input contexts with short outputs, or the reverse.

Before a pilot, it is worth checking a few things:

• Connectivity between nodes. KV-cache transfer is sensitive to latency and bandwidth, so place the nodes in the same availability zone and check the AWS documentation for supported instance types and network adapters.
• Per-phase metrics. Track time to first token, which depends on prefill, and token generation speed, which depends on decode. A single average latency figure will hide an imbalance.
• Node balance. One node per phase gives you a starting point. If a queue builds up on one side, you will need to scale that side independently.
• Client compatibility. An OpenAI-compatible API simplifies migration, but it is better to test the behavior of rarely used request parameters against your own scenarios.

For teams already running on EKS and looking for a way to serve large models without building their own inference platform, this is a quick path to a working prototype. Each team must measure cost and real performance on its own traffic: the guide describes the deployment and does not guarantee a specific gain.