Two pods share one A10G GPU on Amazon EKS: NVIDIA Run:ai gave each 8 GiB
Normally Kubernetes hands a GPU to a single pod in its entirety, so a colleague waits while a notebook uses 6 GiB of 23. AWS showed how NVIDIA Run:ai on Amazon EKS splits one A10G card between two pods with an 8 GiB limit each. Both pods used 6 GiB, and a 9 GiB request was rejected. For notebooks and small inference jobs this shortens queues, but the authors did not measure any speedup.
AI-processed from AWS Machine Learning Blog; edited by Hamidun News
Picture a cluster with a single GPU. An analyst has a notebook open that needs only 6 GiB of memory, while the card has 23 GiB of video memory. A colleague wants to run a short model evaluation, but Kubernetes gave the whole GPU to the first pod, so the second one has to wait even though there is plenty of free memory.
AWS has published a walkthrough of how to fix this with NVIDIA Run:ai GPU Fractioning on Amazon EKS (the managed Kubernetes service on AWS). In the tested example, one NVIDIA A10G GPU in a g5.xlarge instance is shared between two pods: each gets a memory "contract" of 8 GiB (GiB is a gibibyte), and each actually uses 6 GiB.
For teams whose expensive GPUs sit idle under notebooks and small services, this is a way to shorten queues without buying more cards.
What is the problem, and what does Run:ai offer
By default, the NVIDIA device plugin exposes a GPU to Kubernetes as a single nvidia.com/gpu resource. Two pods that each request nvidia.com/gpu: 1 will not run on the same node at the same time, even if their combined memory fits on the card. In the authors' test setup, that is exactly what happened without fractioning: the second pod waits.
There are other ways to share a GPU. MIG carves the card into hardware partitions, but it works only on supported cards. Time-slicing in the device plugin simply alternates processes and provides neither memory isolation nor fault isolation. MPS lets several CUDA processes run at the same time, while fault protection depends on configuration. Run:ai builds a Kubernetes layer on top of MPS: a scheduler, a per-pod memory contract, enforcement of the limit, and per-pod metrics. NVIDIA open-sourced Run:ai on 13 October 2026; the example uses two of the four modules, KAI Scheduler and GPU Fractioning Operator.
The approach makes sense when you need hard GPU memory limits without MIG and are ready to work with a fixed-size node group. Typical workloads are interactive notebooks with long pauses, short model evaluations, compatibility checks, and small inference services.
What happened in practice
The test setup was built in the us-east-2 region on Kubernetes 1.36. A system node, an m7i.large, runs the operator controllers and KAI Scheduler, and the only GPU node is a g5.xlarge with an A10G of 23 028 MiB. Each of the two pods got annotations with a request and a limit of 8 GiB and was assigned the kai-scheduler scheduler. Neither pod requests a whole GPU.
Both pods started on the same card, and each sees exactly 8 192 MiB. The CUDA probe in each pod allocated 6 144 MiB without problems. The memory metric showed about 6.24 GiB per pod, 250 MiB more than the probe allocated. The authors attribute this to the CUDA context and overhead, but they did not measure the breakdown separately.
The negative test passed too: a pod with an 8 GiB limit asked for 9 GiB, and cudaMalloc returned an out-of-memory error. The pod's status was Completed, not OOMKilled, because the probe handled the error itself. This proves that the memory allocation is refused, but it does not prove fault isolation.
Pitfalls
The most annoying one is the driver version. GPU Fractioning Operator 0.1.7 requires a driver from the R615 branch or newer, while the EKS-optimized AL2023 image had only R580 and R595 at the time of testing. So the authors built their own image with Packer, using driver 615.71.10, NVIDIA Container Toolkit 1.20.1, and containerd 2.3.5 with NRI. For production, they advise pinning package versions and the base image. Once a ready-made EKS image gets the required driver, you can switch to it, but the example's scripts will need adjusting.
It is better to read the limitations before you start:
- only the G5/A10G combination was tested; other GPUs need their own compatibility check;
- the memory limit does not reserve a proportional share of compute;
- the test measures neither performance nor fault isolation, so it is not a security guarantee;
- more pods per card does not mean higher throughput or lower costs;
- the setup creates paid resources: the EKS control plane, a NAT gateway, instances, and an AMI snapshot, so you need to clean up after testing.
What a team should do
If your GPUs sit idle under notebooks and small services, you can reproduce the setup from the aws-samples repository on a throwaway cluster. Choose the per-pod limit based on measured peak consumption plus headroom for overhead, then check latency under concurrent load. For large-scale training and latency-sensitive services, fractioning is probably not a fit. For everything else, it is a low-cost way to clear the queue for a GPU.
- Nov 19, 2026Anthropic goes public and raises $81.3 billion, beating SpaceX's $75 billion record
- Nov 19, 2026Amazon and Google's Anthropic Stakes Valued at $308 Billion in IPO at $83 a Share
- Nov 19, 2026AWS AgentCore Agents Can Now Reach a Private Keycloak Behind a Corporate CA
- Nov 19, 2026Nvidia posts record $115.8 billion revenue, yet trades cheaper than at any point since 2012
Key Questions Answered
What is the problem, and what does Run:ai offer?
By default, the NVIDIA device plugin exposes a GPU to Kubernetes as a single nvidia.com/gpu resource. Two pods that each request nvidia.com/gpu: 1 will not run on the same node at the same time, even if their combined memory fits on the card. In the authors' test setup, that is exactly what happened without fractioning: the second pod waits. There are other ways to share a GPU. MIG carves the card into hardware partitions, but it works only on supported cards.
What happened in practice?
The test setup was built in the us-east-2 region on Kubernetes 1.36. A system node, an m7i.large, runs the operator controllers and KAI Scheduler, and the only GPU node is a g5.xlarge with an A10G of 23 028 MiB. Each of the two pods got annotations with a request and a limit of 8 GiB and was assigned the kai-scheduler scheduler. Neither pod requests a whole GPU. Both pods started on the same card, and each sees exactly 8 192 MiB. The CUDA probe in each pod allocated 6 144 MiB without problems.
What are the key details on Pitfalls?
The most annoying one is the driver version. GPU Fractioning Operator 0.1.7 requires a driver from the R615 branch or newer, while the EKS-optimized AL2023 image had only R580 and R595 at the time of testing. So the authors built their own image with Packer, using driver 615.71.10, NVIDIA Container Toolkit 1.20.1, and containerd 2.3.5 with NRI. For production, they advise pinning package versions and the base image.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.