SageMaker jobs can now start in seconds, if your image is SOCI-indexed

Amazon has announced Seekable OCI (SOCI) indexing for Amazon SageMaker AI training and processing jobs. Jobs can now start from a partly downloaded container image, and in AWS’s benchmarks a 35.4 GB image went from 7 minutes 17 seconds to 3-11 seconds.

Lazy loading replaces the full image pull

Until now, SageMaker AI downloaded an entire container image from Amazon ECR, then decompressed and wrote every layer to disk before running any user code. Amazon notes that deep learning images often exceed 10 GB, so this adds minutes to each job, billed as job time.

With SOCI, the service fetches only the files needed to launch the container and streams the rest in the background as the code touches them. The index is metadata stored in ECR as a separate artifact linked to the original image, so the image digest and existing signatures stay valid. SageMaker AI checks for an index automatically. If none exists, or lazy loading fails, the job falls back to a standard download. Amazon says no change to the job definition, entry point or data channels is needed.

There are two ways in. AWS publishes pre-indexed variants of some Deep Learning Containers, marked by a -soci suffix on the tag, for PyTorch Training, TensorFlow Training, vLLM and SGLang, including some arm64 builds. Or teams can index their own image on a Linux host with containerd, the SOCI snapshotter and Finch or nerdctl, using nerdctl image convert --soci. A CloudFormation project, the CFN ECR SOCI Index Builder, can generate indexes on every ECR push through an EventBridge rule. An image built FROM a DLC base needs its own index, because the base image’s index does not carry over.

What AWS measured, and what it cost in the first epoch

Amazon’s tests used eight images of 1.8 to 35.4 GB compressed, on instances from ml.g4dn to ml.p5 plus ml.m7i, in us-west-2 and us-east-1. Figures are medians of repeated runs.

• A 1.8 GB image: 48 seconds with a standard pull, 3-11 seconds lazy-loaded.
• A 13.2 GB image: 3 minutes 42 seconds against about 6 seconds.
• The 13.2 GB image across 13 instance type and size combinations: 3 min 16 s to 4 min 20 s standard, 6-8 s lazy-loaded.

The saving is not free. In full training runs, ResNet-50 on Imagenette and fine-tuning Qwen2.5-0.5B, the first epoch took 3-16 seconds longer because files were fetched on first use. From the second epoch, epoch times matched. Net, each job finished 1-6 minutes sooner, according to Amazon. Amazon also reports that GPU idle time during image pulls fell by up to 90 percent.

Who benefits, and where the gain shrinks

The angle that matters most is limits. Amazon is open that results depend on what a job reads at startup. If an entry point scans or preloads most of the image, the first epoch waits for roughly the bytes a full pull would have fetched, and the net gain is small. Even a 6 GB model baked into the image, which added 2-3.5 minutes at load time, still left the job 2-2.5 minutes ahead in Amazon’s test.

Several eligibility limits apply. Only private Amazon ECR repositories qualify; images from other registries, or pulled with TrainingRepositoryAccessMode set to Vpc, are excluded. Images under roughly 4 GB see smaller gains. SOCI-indexed images run on a containerd-based runtime, so scripts that call the Docker CLI, read /var/run/docker.sock or derive identifiers from cgroup paths need review. Amazon says the feature is available in all AWS Regions, and it does not announce a separate price.

Outside context supports both the promise and the caveats. AWS’s own 2023 write-up on SOCI for Fargate said the benefit is largest for images over 250 MB and that smaller images can start slower, and that workloads needing most of the image soon after launch gain less. The SOCI snapshotter project is open source under the Apache 2.0 license and cites a 2016 FAST paper finding that image download accounts for 76 percent of container startup time, while on average only 6.4 percent of fetched data is needed to start a container. That is research on containers in general, not a SageMaker measurement.

All performance numbers come from Amazon’s own benchmarks on two workloads, a small ResNet-50 run and a 0.5B-parameter fine-tune. Neither is a multi-node run on a very large model, and the post gives no independent verification. Teams running hyperparameter sweeps of hundreds of short jobs, where startup dominates, are the clearest winners. Those with heavy startup preloading should test before switching tags.

The release fits a wider pattern of AWS trimming overhead around AI infrastructure, as in our coverage of a Hugging Face Space that shows which Hub models run on AWS Trainium.