Build status · Apache-2.0 license
NVIDIA Recovery detects crashes, hangs, and stragglers in distributed AI training and coordinates automated recovery. This saves operators and researchers from having to watch and debug jobs manually around the clock.
In distributed training, one failed process, stalled collective, or slow worker can interrupt progress across an entire GPU allocation. As clusters and runs grow, so does exposure to failures.
When this happens, recovery becomes an investigation: is the problem in the training code, the framework, communication libraries, or hardware? Is it safe to retry? Should the next attempt avoid a node? Overnight, that investigation may not even start until someone sees the stalled run.
Because of the potential scale of these failure scenarios, recovery must be both timely and correct. Lengthy manual investigations can delay recovery.
NVIDIA Recovery automates this process by:
- Detecting drops in forward progress and job failures.
- Using NVIDIA Attribution to diagnose anomalies and guide the recovery response needed to restore forward progress.
- Integrating with the job system and scheduler to restart the workload and exclude nodes if needed.
This automated workload-recovery loop runs on infrastructure you operate and requires only minor modifications to job submission, allowing researchers to continue using their scheduler and checkpoint-capable training scripts.
- Crashes: failed executions and recognized application errors, including PyTorch/NCCL, CUDA, and GPU interconnect failure signatures.
- Hangs: jobs that stop emitting expected progress or report recognized communication or application hang symptoms.
- Stragglers: sustained drops in reported training throughput relative to the recent baseline—not just jobs that have exited.
Log-based detection requires configured patterns and progress signals; the failed scheduler state also provides a crash-detection fallback. Straggler-triggered recovery is opt-in. NVIDIA Attribution adds deeper diagnosis across application, system software, and hardware. The recovery loop works without it; recognizing a symptom is not the same as proving its root cause.
flowchart LR
detect["Detect"] --> decide["Decide"]
decide --> act["Recover"]
act --> verify["Verify"]
NVIDIA Recovery records a failed attempt and decides whether policy allows another try. It persists that decision as Advice: "restart this attempt, with these placement constraints." The scheduler integration performs the necessary actions and checks their results. The new execution then loads its checkpoint using the application's own restart logic.
This separation makes the decision inspectable and lets the controller resume unfinished recovery work from the current scheduler state. Restarting an allocation, restoring training progress, and completing training are distinct milestones. Refer to the architecture and full job lifecycle for more details.
| Your goal | Guide |
|---|---|
| See recovery on a disposable local stack; no physical GPU needed | Get started locally |
| Submit and follow jobs on an installed service | Run jobs |
| Build images, install, validate, and operate your deployment | Deploy and operate |
| Set up the repository and make a change | Develop and contribute |
| Understand components and recovery boundaries | Architecture |
Find more detailed guides and references in the documentation map, or start with the FAQ.
NVIDIA Recovery is experimental and not ready for production use. APIs, configuration, and behavior may change. It supports NVIDIA GPU clusters from Ampere through Hopper to Blackwell, with Slurm or Slurm-on-Kubernetes through Slinky. Kubernetes-native workload recovery is not currently implemented.
The project distributes source and Helm charts; operators build their own images. NVIDIA Attribution is enabled by default; its source is not yet publicly fetchable. See Attribution prerequisites and degraded OFF mode. The local demo explicitly selects OFF and requires no private provider credentials. NVIDIA Recovery manages workloads, not hardware repair or checkpoint storage. Refer to the Support and Releases sections for the current boundaries.
Start with the developer guide and focused testing guide. External contributions are welcome; Contributing explains review, DCO sign-off, and CI approval.
For help, see Support. Report vulnerabilities privately through Security. See Governance, Maintainers, and the Code of Conduct for how the project works.
NVIDIA Recovery is licensed under Apache-2.0, except where a file or component carries its own license. Upstream license grants, copyright notices, and attribution remain in effect; the repository license does not replace them. The Fern stylesheet is NVIDIA-owned source released under Apache-2.0.
The repository contains the following 10 nested license files:
| Directory or component | License | License file |
|---|---|---|
| Vendored code-simplifier skill | Apache-2.0 | LICENSE |
| Vendored Context7 skill | MIT | LICENSE |
| Vendored feature-dev skill | Apache-2.0 | LICENSE |
| Slurm Advisor Controller chart | Apache-2.0 | LICENSE |
| NVIDIA Recovery umbrella chart | Apache-2.0 | LICENSE |
| Alloy license material | Apache-2.0 | alloy.LICENSE |
| Bitnami charts license material | Apache-2.0 | bitnami-charts.LICENSE |
| Grafana charts license material | Apache-2.0 | grafana-charts.LICENSE |
| MinIO license material | GNU AGPL v3 | minio.LICENSE |
| OpenTelemetry charts license material | Apache-2.0 | opentelemetry-charts.LICENSE |
The two chart-level LICENSE files are copies of the root Apache-2.0 license
so each chart carries the license when packaged or distributed independently.
The umbrella chart also retains upstream license material in its
licenses directory, including the
MinIO notice. Chart licenses do not
relicense the applications or container images deployed by those charts;
those components retain their own licenses and notices.
Thanks to everyone who contributes to NVIDIA Recovery.