Slurm — схвалення — InstanceStress
Slurm — схвалення — InstanceStress
Slurm — запуск — HRES
Together, these capabilities help you detect failures earlier, shorten investigations, and raise goodput by reducing the GPU time spent on workloads that can't make useful progress.
The agent follows HANG_DETECTED with a record that says Hang notification sent, no automatic action taken .
A HyperPod Slurm AMI released on or after November 10, 2026, which includes hung job detection.
Today, we're launching hung job detection for SageMaker HyperPod Slurm clusters.
Capability What it catches When it runs Available on
MultiNodeTrainingValidation deep health check NaN or infinite loss, or training that doesn't finish in time, on GPUs that pass hardware checks When instances are created or updated, or on demand EKS clusters, on supported EFA-enabled NVIDIA GPU instances
HyperPod Kubernetes Resource Monitoring Agent Node condition changes, pod and DaemonSet failures, and failed training jobs, with the pods and nodes involved.
It writes an event to CloudWatch and can cancel the job so that Slurm requeues it.
Slurm passes them to the plugin without interpreting them, which removes the need for a second command-line flag.
- No device awareness: Slurm cannot tell on its own when a QPU is recalibrating or offline, so it keeps starting jobs that cannot use the device until it is back online.
It then explains the solution: how HRES models a QPU, and what HRES plugins add in Slurm 26.11, including the API callbacks and capabilities that middleware teams build on.
Orchestrating Resources in an Open Quantum-GPU Platform The Slurm expansion adds a community-developed orchestration layer to NVIDIA's open platform for quantum-GPU supercomputing, which centers around the CUDA-Q open platform for hybrid quantum-classical computing.