SageMaker can now split one GPU into 7 parts: more models on the same hardware

A familiar situation: you have a classifier, a reranker, an embedder, and a couple of fine-tuned models. Each needs 10–20 GB, but SageMaker allocates an entire GPU to each one. The remaining memory sits idle while the bill keeps growing. AWS has finally fixed this.

What changed

SageMaker Inference now supports GPU partitioning based on NVIDIA Multi-Instance GPU (MIG). A physical GPU is divided into independent partitions, each with its own share of compute blocks, cache, memory, and memory bandwidth. The isolation is hardware-based, not software-based.

Previously, anyone who wanted MIG had to run their own Kubernetes stack: drivers, a device plugin, partition configuration, and scheduling. Multi-model endpoints did not solve the problem: models share the device without isolation, and the initial load also incurs a cold start. Now you select a partition profile and SageMaker handles the rest.

What it looks like in practice

Two optional fields have been added. AcceleratorPartitionLayouts is specified in the production variant when creating the endpoint configuration and describes how each GPU is partitioned. For example, on a p4d.24xlarge, the mig-4g.20gb + mig-2g.10gb + mig-1g.5gb layout uses all seven A100 slices. Across eight GPUs, that produces 24 partitions: eight of each type.

AcceleratorPartitionType is specified in the inference component and determines which partition hosts the model. There are no new API operations or IAM actions, and InvokeEndpoint remains unchanged. By increasing CopyCount, you can place additional copies of a small model on available partitions. Creating the partition layout added about 17 seconds to endpoint deployment.

Available profiles depend on the GPU. A100 and H100 GPUs are divided into sevenths by compute capacity and eighths by memory. G7 (RTX PRO 4500 Blackwell) is split into halves, while G7e (RTX PRO 6000 Blackwell) is split into quarters. The 3g profile is unusual: it takes 3 of 7 compute blocks but 4 of 8 memory blocks, so two such partitions fill the entire GPU.

How to choose a profile

Compute blocks matter for prefill, meaning prompt processing: the longer the input and the more concurrent requests, the more important they become. Memory matters for decoding and capacity: model weights, the KV cache, and 1–2 GB of overhead for the CUDA context. An 8B model in FP16 weighs about 16 GB, and on a 40 GB partition it can comfortably support an 8,000-token context. If you account only for the weights, the model may load but fail under load or silently reduce its context window.

Where it pays off

• A collection of small and midsize models: fine-tuned variants, embedders, and rerankers.
• Multitenancy: a hardware boundary separates customers, but the host, network, and routing remain shared, so this is not complete isolation.
• Pipelines on one instance: transcription, diarization, and an LLM without network hops.
• A guardrail model with predictable latency.
• Canary deployments: a candidate on a small partition alongside the production model.

Limitations

The layout must be selected when the endpoint is created and cannot be changed on the fly. It works only with inference components: standalone, serverless, and heterogeneous endpoints are not supported. A model that needs more than one GPU still requires whole devices. On P4d, the mig-1g.10gb profile requires driver R525 or later, while the default image ships with version 470, so you must specify InferenceAmiVersion al2023-ami-sagemaker-inference-gpu-4-1. Otherwise, partitioning will fail without a clear error message.

What to do

If you have several small models running on separate GPUs in SageMaker, calculate their memory requirements using the formula “weights + KV cache + overhead” and try consolidating them onto one GPU. Run load tests as well: a partition's performance does not depend on its neighbors, so the results can be carried over to production.