SageMaker can now split one GPU seven ways and route inference across Regions
Amazon's part 2 review covers 13 SageMaker inference launches since July, from seven-way GPU partitions and managed Dynamo to cross-Region routing and default-on drift monitoring.

Amazon Web Services has published part 2 of its SageMaker inference review, covering 13 launches from July to November 2026. They include managed NVIDIA Dynamo on HyperPod, GPU partitioning into up to seven slices, and cross-Region routing for endpoints.
The post is a roundup, not a single product release. AWS organizes it around one hypothetical support team that grows from a single 70B-parameter assistant to a multi-Region deployment. Each stage maps to a set of launches, numbered 14 to 26 as a continuation of part 1.

Thirteen launches in five stages
The first stage is choosing a configuration. Inference recommendations, which AWS launched in April as an API, now run in SageMaker AI Studio (July) and in the SageMaker Python SDK v3, version 3.17.0 and later (August). In September they gained support for LoRA adapters, passed in as Model Package ARNs or S3 URIs, and a search mode called max-concurrency-under-sla. That mode looks for the highest concurrency that still meets targets for time to first token, time per output token, end-to-end latency and error rate. In October AWS added an aws-ai-ml agent skill for coding agents that support the Model Context Protocol, which AWS says takes about 10 minutes to set up.
The second stage is packing more models onto each GPU. Inference Components, which host several models on one endpoint, can now be managed in the SageMaker AI console as well as the SDK and CLI, including scaling a component down to zero copies. AWS also added GPU partitioning with NVIDIA Multi-Instance GPU, which splits one physical GPU into as many as seven isolated partitions, depending on the accelerator. AWS is explicit that pricing does not change: customers pay the standard rate for the whole instance-hour, so any saving comes from fitting more models per instance. Host CPU, memory, networking and routing are still shared between partitions.

One GPU can be divided into as many as seven isolated partitions, but the instance-hour is still billed at the full rate.
Managed Dynamo and routing for larger deployments
For models too big for one instance, AWS now runs NVIDIA Dynamo as a managed serving engine on SageMaker HyperPod Inference (November). Dynamo handles multi-node sharded serving, separate prefill and decode pools, and KV-aware routing. HyperPod supplies GPU capacity, Elastic Fabric Adapter networking, node replacement when a node fails, autoscaling and TLS. It supports vLLM, SGLang and TensorRT-LLM, and is available wherever HyperPod is. Teams with existing Dynamo manifests can bring them over largely unchanged, according to AWS, and clients keep calling an OpenAI-compatible endpoint.

Routing is split into four levels. On endpoints, KV-cache-aware routing (November) sends each request to the instance holding the longest matching cached prefix. It relies on cache events reported by the container, so the first release requires an AWS-verified container. The older prefix-aware routing from part 1 works with any container. On HyperPod, the Inference Gateway (September) gives each cluster one OpenAI-compatible endpoint and scores pods on KV cache use, queue depth, prefix cache hits and LoRA adapter residency. AWS says it cut P99 time to first token by up to 97% on mixed GPU generations and up to 98% under bursty traffic, compared with round-robin. Those figures are AWS’s own and the post does not describe the test setup.
Cross-Region Inference Routing lets an application call one routing target that falls back to other registered endpoints, in the same or different accounts, over AWS PrivateLink. AWS says it adds no capacity and deploys no endpoints. Owners can cap overflow traffic and restrict which Regions may serve a request, which matters for data residency.
Monitoring on by default, and in-place updates
Continuous performance monitoring is on by default for SageMaker inference endpoints, with opt-out available. According to AWS, it reads only latency, GPU utilization and serving configuration, not requests or responses. It reports latency drift, configuration gaps present since deployment, and idle GPUs, and nothing deploys until the customer applies a fix. In-place rolling updates need no customer action. For a 20-instance fleet with a batch size of 2, AWS says an update acquires 2 new instances instead of 20. AWS plans scale-in-first updates in early 2027 to remove the need for extra instances entirely.
A steady build-out of SageMaker as a serving platform
This roundup follows a string of narrower SageMaker items. Earlier this month, Amazon put a 75 GB NVFP4 build of Nemotron 3.5 Super VL on JumpStart for G7 instances, as we reported. It also added per-use billing for streaming, so Marketplace models can charge by audio processed, covered here. On November 19, HyperPod gained a way to flag the single bad GPU in a training job. Read together, the pattern is Amazon moving the operational work around inference, such as placement, failover and tuning, from customer scripts into managed features.
The review states its own thesis: “Part 1 argued that advantage in AI inference comes from operating an efficient inference stack.” It also ends with an offer of AWS’s Experience-Based Acceleration program, so it reads partly as a sales document.
Several questions are left open. The post gives no pricing for the new routing, monitoring or managed Dynamo features, no regional availability for cross-Region routing or KV-cache-aware routing, and no independent benchmarks. The team story is hypothetical, so no customer results are reported. Teams that want to try the features should start from the SageMaker AI Developer Guide, which AWS says has documentation for each capability.
Sources
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Alex CarterAI News & Big Tech CorrespondentAlex Carter writes for Aiholics, keeping readers updated on the fast-paced world of AI and Big Tech.
You may also like

Nvidia’s interim CEO promises early raises as directors split over Huang

CUDA and ROCm: The GPU software layer that decides which chips AI can run on

Fal’s valuation doubles to $16B in seven months as inference demand grows
