SageMaker HyperPod now flags the one bad GPU in a training job
Amazon's new HyperPod observability compares each GPU in a training job with its peers and flags hardware, network, throttling and overheating faults, on both EKS and Slurm clusters.

Amazon has added enhanced observability to Amazon SageMaker HyperPod. It compares every GPU in a training job with its peers and flags the ones that drift, on clusters orchestrated by either Amazon EKS or Slurm.
AWS frames the problem plainly: “a single unhealthy accelerator can stall an entire distributed job”. The hard part, it says, is rarely noticing a slowdown. It is working out whether a throttled GPU, a saturated network link or a stalled data pipeline caused it.

How the outlier detection and fault rules work
The diagnostic summary combines two methods. The first uses Amazon CloudWatch’s DetectGroupOutliers API to compare GPUs inside one job on utilization, memory, temperature, power draw and SM clock frequency. A GPU that deviates significantly from the group shows up as an outlier card. AWS says this works for both EKS and Slurm jobs.
The second method is a set of static rules for known failure patterns, sorted into four types:

- Hardware: ECC memory errors, XID faults and accelerator failures.
- Network: lower EFA throughput, packet drops and retransmissions.
- Throttling: clocks held back by heat, board or reliability limits.
- Overheating: temperatures above configured limits.
Each detected issue names the job, cluster, node, accelerator and rank, according to AWS.
Metrics, stores and what they cost
Telemetry comes in two tiers. Default metrics are on automatically and cover cluster health, CPU, disk and OS usage, GPU utilization and temperature, and per-job resources. They are collected every 30 seconds. AWS puts the overhead at roughly 1-2% CPU and 250-300 MB of memory.

Advanced metrics are opt-in. They add the full node exporter suite, all NVIDIA DCGM and AWS Neuron metrics, and Elastic Fabric Adapter metrics for throughput, drops, retransmissions and errors. AWS notes they raise the volume of telemetry and therefore the bill. Its view is that small jobs on small clusters can usually skip them, while long runs on large clusters benefit.
Amazon CloudWatch is now the default metric store, with views through CloudWatch SageMaker Insights. Amazon Managed Service for Prometheus, previously the only option, remains available for teams that use Prometheus and Grafana. AWS says CloudWatch costs less than the Prometheus service, though it publishes no figures in the post.
On EKS the feature ships as an add-on, named amazon-sagemaker-hyperpod-observability in the CLI examples. On Slurm it is a cluster extension, installed with aws sagemaker create-cluster-extension. Access is controlled through IAM. In Quick setup it is enabled by default and cannot be unchecked, while Custom setup lets users choose categories. It is available in every AWS Region where HyperPod is offered. The announcement points to the HyperPod pricing page for costs and supported instance types.
Default metrics are collected every 30 seconds with about 1-2% CPU overhead, according to AWS.
Why peer comparison is the technical bet
Synchronous distributed training moves at the pace of its slowest participant, so one throttling GPU out of hundreds slows all of them. A fixed threshold on temperature or utilization is hard to set, because healthy values change with the model, batch size and phase of the run. Comparing each GPU with the others running the same job sidesteps that: the baseline is the job itself. The static rules cover the cases where a cause is already known, such as an XID error.
The same design shows in the new console views. AWS describes a Hardware tab with a fleet topology map of device states, a Workload tab that flags jobs with abnormal network usage, and a Network tab that tallies healthy and degraded nodes by instance type. Prebuilt Amazon Managed Grafana dashboards are offered for teams that prefer them.
The release fits a week of HyperPod operations updates from AWS. A separate announcement on November 17 added the ListClusterOperations and DescribeClusterOperation APIs for tracking scaling and on-demand Deep Health Check requests, at no extra charge and for clusters using continuous provisioning. Together they cover what happened to a cluster as well as why a job slowed. AWS also published a post on November 18 about workload recovery and expanded infrastructure checks.
What the announcement leaves open
The post gives no accuracy figures for the outlier detection, no false-positive rate and no measured reduction in troubleshooting time. The claim that it “reduces troubleshooting time” is AWS’s own and has not been independently verified. It also does not say how many GPUs a job needs before the peer comparison becomes reliable, how long metrics are retained, or what CloudWatch and Prometheus cost per cluster. The numbers on the example Observability tab, such as 128 nodes and 1,024 accelerators, are illustrations from a screenshot, not benchmarks.
For teams already running HyperPod, the practical change is that fault diagnosis moves from stitched-together dashboards into the SageMaker console, with a choice of CloudWatch or Prometheus. Those who enable advanced metrics should budget for the extra telemetry volume.
Sources
Sign Up for the Daily AI Pulse
One email a day. All the stories that matter.
By signing up, you agree to our Terms of Use and acknowledge the data practices in our Privacy Policy. You may unsubscribe at any time.
Alex CarterAI News & Big Tech CorrespondentAlex Carter writes for Aiholics, keeping readers updated on the fast-paced world of AI and Big Tech.



