Skip to content

fix(asr): catch RuntimeError when checking CUDA graphs conditional-node support#15957

Open
mittalpk wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
mittalpk:fix/cuda-graphs-runtimeerror-fallback
Open

fix(asr): catch RuntimeError when checking CUDA graphs conditional-node support#15957
mittalpk wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
mittalpk:fix/cuda-graphs-runtimeerror-fallback

Conversation

@mittalpk

Copy link
Copy Markdown

Fixes #15905

What does this PR do ?

maybe_enable_cuda_graphs() crashes instead of falling back gracefully when cuda-python is importable but there's no real NVIDIA driver to talk to (e.g. ROCm/AMD builds of torch).

Collection: asr

Changelog

  • nemo/collections/asr/parts/submodules/transducer_decoding/label_looping_base.py: added RuntimeError to the exception types caught around check_cuda_python_cuda_graphs_conditional_nodes_supported() in maybe_enable_cuda_graphs().

Root cause

On systems where torch.cuda.is_available() returns True (e.g. ROCm/AMD builds of torch, which shim the CUDA API) but no real NVIDIA driver/libcuda.so is present, check_cuda_python_cuda_graphs_conditional_nodes_supported()'s call to cuda.cuDriverGetVersion() raises a raw RuntimeError ("Failed to dlopen libcuda.so.1") from inside the cuda-python native extension — before the function's own EnvironmentError/ImportError checks even run. maybe_enable_cuda_graphs() only caught (ImportError, ModuleNotFoundError, EnvironmentError), so this RuntimeError propagated uncaught and crashed decoding instead of falling back to CudaGraphsMode.NO_WHILE_LOOPS as intended.

Testing

Added tests/collections/asr/decoding/test_label_looping_cuda_graphs_fallback.py (3 tests, no real GPU/ROCm hardware needed — mocks check_cuda_python_cuda_graphs_conditional_nodes_supported directly):

  • falls back to NO_WHILE_LOOPS when the check raises RuntimeError (the new case)
  • still falls back on the previously-handled error types (EnvironmentError)
  • still enables FULL_GRAPH mode when the check succeeds

Verified the new RuntimeError test fails against the pre-fix code and passes against the fix. Ran the full tests/collections/asr/decoding/ + tests/collections/common/test_optional_cuda_graphs.py suites (283 passed). One unrelated pre-existing failure (test_ctc_decoding.py::test_batched_decoding_labels) reproduces identically with and without this change — confirmed by running it both in isolation and as part of the full suite on unmodified main, so it's not something this PR introduces.

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation? (not applicable — no user-facing behavior/API changed, only an internal fallback's exception handling)
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc) — no

@copy-pr-bot

copy-pr-bot Bot commented Jul 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…de support

maybe_enable_cuda_graphs() only caught (ImportError, ModuleNotFoundError,
EnvironmentError) around check_cuda_python_cuda_graphs_conditional_nodes_supported().
On systems where torch.cuda.is_available() is True (e.g. ROCm/AMD builds of
torch, which shim the CUDA API) but no real NVIDIA driver/libcuda.so is
present, cuda.cuDriverGetVersion() raises a raw RuntimeError from the
cuda-python native extension instead. This crashed decoding instead of
falling back to CudaGraphsMode.NO_WHILE_LOOPS as intended.

Adds RuntimeError to the caught exception types. Regression test added in
tests/collections/asr/decoding/test_label_looping_cuda_graphs_fallback.py,
covering the new RuntimeError case, the previously-handled error types,
and the success path.

Signed-off-by: Praveen Mittal <pkmittal28@gmail.com>
@mittalpk
mittalpk force-pushed the fix/cuda-graphs-runtimeerror-fallback branch from 4727a8f to 4a9a4eb Compare July 24, 2026 07:58
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Jul 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ASR community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TDT/label-looping decoder crashes instead of falling back when libcuda is absent (e.g. ROCm builds of torch)

2 participants