Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
InfrastructureInsight · On RadarAWS Machine Learning Blog→ original

SageMaker starts jobs in seconds: a 35.4 GB image loads in 3–11 s instead of 7 minutes

Until now, SageMaker downloaded the entire container image before running your code, and you paid for minutes of idle GPU time. With a SOCI index, it fetches only the files needed to start, and pulls the rest in the background, much like a video stream. A 35.4 GB image that used to take 7 minutes 17 seconds now starts in 3–11 seconds, and GPU idle time during image loading drops by up to 90%. For short experiments and hyperparameter sweeps, you only need to change the image tag.

AI-processed from AWS Machine Learning Blog; edited by Hamidun News
SageMaker starts jobs in seconds: a 35.4 GB image loads in 3–11 s instead of 7 minutes
Source: AWS Machine Learning Blog. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceAWS Machine Learning Blog ↗
Editor-in-Chief
Key metric #1GPU idle time during image loading dropped by up to 90% in AWS measurements
Key metric #2The first epoch takes 3–16 s longer, but the job finishes 1–6 minutes sooner overall
Key metric #3Private Amazon ECR only; images up to about 4 GB gain little

Every minute a deep learning container spends downloading is a minute you pay for and a minute your team waits. PyTorch or TensorFlow images easily exceed 10 GB. Before running your code, SageMaker downloads the entire image, unpacks it, and writes every layer to disk. For a large image, that means several minutes of billed time during which the model makes no progress at all. AWS has announced that Seekable OCI (SOCI) indexing now works for Training and Processing jobs in SageMaker AI. It removes the full image download from the critical startup path.

How it works

SOCI is an open-source technology for lazy container loading. Think of streaming a movie: the player starts with the beginning and fetches the rest as you watch, without making you wait for the full download. It works the same way here. Using the index, the runtime fetches only the files needed to start the container, and pulls everything else in the background when the code accesses it.

The index is metadata containing a list of all files in the image. It is stored in Amazon ECR as a separate artifact alongside the source image, so the image digest and existing signatures remain valid. You do not need to change the job definition, entry point, or data channels.

SageMaker checks on its own whether the image has an index. If it does, lazy loading is enabled. If it does not, the job downloads the image as usual. If the index cannot be read or lazy loading fails, the job also falls back to a regular download and keeps running. Adding an index cannot break an existing workflow.

What the measurements showed

AWS compared regular downloads and SOCI on eight images ranging from 1.8 to 35.4 GB. The tests used GPU and CPU instances from ml.g4dn to ml.p5 and ml.m7i in the us-west-2 and us-east-1 regions. Time was measured from the start of the image download until the container was ready. All figures are medians of repeated runs:

  • 1.8 GB image: 48 seconds with a regular download versus 3–11 seconds with SOCI;
  • 13.2 GB image: 3 minutes 42 seconds versus about 6 seconds;
  • 35.4 GB image: 7 minutes 17 seconds versus 3–11 seconds.

Lazy loading stays at 3–11 seconds regardless of image size, so the larger the image, the bigger the gain. For the 13.2 GB image, AWS tested 13 combinations of instance types and sizes. Regular downloads took between 3 minutes 16 seconds and 4 minutes 20 seconds, while lazy loading took 6–8 seconds, roughly 34 times faster in every configuration. You do not need to change instances to get the benefit.

AWS also checked whether training itself suffers. It ran ResNet-50 image classification on Imagenette (9,469 images) and full fine-tuning of Qwen2.5-0.5B on alpaca-cleaned (51,760 examples). From the second epoch onward, the time per epoch was the same. The first epoch on an indexed image is 3–16 seconds longer, because the framework accesses files for the first time and they have to be fetched. Overall, each job finished 1–6 minutes earlier. AWS says GPU idle time during image loading dropped by up to 90%.

Limitations and how to enable it

The gain depends on your code. If the entry point reads most of the image at the start (scanning it, computing checksums, or preloading contents), the first epoch waits for roughly the same number of bytes as with a full download, and the net savings will be small. Even so, heavy startups stayed positive in the tests. torch.compile and JIT builds of CUDA extensions added 7–15 seconds to the first epoch, yet the net savings were between 3 minutes 25 seconds and 3 minutes 56 seconds. A 6 GB model inside the image loaded in 2–3.5 minutes during startup, but the job still finished 2–2.5 minutes earlier.

Keep in mind:

  • it works only with private Amazon ECR repositories; images from other registries and the TrainingRepositoryAccessMode=Vpc setting are not supported;
  • for images under about 4 GB, the improvement is modest;
  • AWS recommends checking scripts for Docker dependencies before switching to the new loading method.

There are two ways to enable it. The first is to use a ready-made AWS Deep Learning Container with the -soci suffix in the tag. Such variants exist for PyTorch Training, TensorFlow Training, vLLM, and SGLang.

The second is to index your own image with the command nerdctl image convert --soci on a Linux host running containerd and the SOCI snapshotter, then push it to ECR. If your image is built on top of a DLC, the base image's index does not carry over, so you must create a new index for the final image. Index creation can be automated in CI/CD with the CFN ECR SOCI Index Builder: it reacts to ECR push events through EventBridge and does not require changes to your build pipeline.

For teams running hundreds of short jobs during hyperparameter tuning, this is nearly free savings: only the image tag changes. First, check in your own pipeline how much of the image your code reads at startup. That determines how much you will actually save.

Story

Key Questions Answered

How it works?

SOCI is an open-source technology for lazy container loading. Think of streaming a movie: the player starts with the beginning and fetches the rest as you watch, without making you wait for the full download. It works the same way here. Using the index, the runtime fetches only the files needed to start the container, and pulls everything else in the background when the code accesses it. The index is metadata containing a list of all files in the image.

What the measurements showed?

AWS compared regular downloads and SOCI on eight images ranging from 1.8 to 35.4 GB. The tests used GPU and CPU instances from ml.g4dn to ml.p5 and ml.m7i in the us-west-2 and us-east-1 regions. Time was measured from the start of the image download until the container was ready. All figures are medians of repeated runs: 1.8 GB image: 48 seconds with a regular download versus 3–11 seconds with SOCI; 13.2 GB image: 3 minutes 42 seconds versus about 6 seconds; 35.4 GB image: 7 minutes 17 seconds versus 3–11 seconds.

What are the key details on Limitations and how to enable it?

The gain depends on your code. If the entry point reads most of the image at the start (scanning it, computing checksums, or preloading contents), the first epoch waits for roughly the same number of bytes as with a full download, and the net savings will be small. Even so, heavy startups stayed positive in the tests. torch.compile and JIT builds of CUDA extensions added 7–15 seconds to the first epoch, yet the net savings were between 3 minutes 25 seconds and 3 minutes 56 seconds.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…