fix(asr): catch RuntimeError when checking CUDA graphs conditional-node support#15957
Open
mittalpk wants to merge 1 commit into
Open
fix(asr): catch RuntimeError when checking CUDA graphs conditional-node support#15957mittalpk wants to merge 1 commit into
mittalpk wants to merge 1 commit into
Conversation
…de support maybe_enable_cuda_graphs() only caught (ImportError, ModuleNotFoundError, EnvironmentError) around check_cuda_python_cuda_graphs_conditional_nodes_supported(). On systems where torch.cuda.is_available() is True (e.g. ROCm/AMD builds of torch, which shim the CUDA API) but no real NVIDIA driver/libcuda.so is present, cuda.cuDriverGetVersion() raises a raw RuntimeError from the cuda-python native extension instead. This crashed decoding instead of falling back to CudaGraphsMode.NO_WHILE_LOOPS as intended. Adds RuntimeError to the caught exception types. Regression test added in tests/collections/asr/decoding/test_label_looping_cuda_graphs_fallback.py, covering the new RuntimeError case, the previously-handled error types, and the success path. Signed-off-by: Praveen Mittal <pkmittal28@gmail.com>
mittalpk
force-pushed
the
fix/cuda-graphs-runtimeerror-fallback
branch
from
July 24, 2026 07:58
4727a8f to
4a9a4eb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #15905
What does this PR do ?
maybe_enable_cuda_graphs()crashes instead of falling back gracefully whencuda-pythonis importable but there's no real NVIDIA driver to talk to (e.g. ROCm/AMD builds of torch).Collection: asr
Changelog
nemo/collections/asr/parts/submodules/transducer_decoding/label_looping_base.py: addedRuntimeErrorto the exception types caught aroundcheck_cuda_python_cuda_graphs_conditional_nodes_supported()inmaybe_enable_cuda_graphs().Root cause
On systems where
torch.cuda.is_available()returnsTrue(e.g. ROCm/AMD builds of torch, which shim the CUDA API) but no real NVIDIA driver/libcuda.sois present,check_cuda_python_cuda_graphs_conditional_nodes_supported()'s call tocuda.cuDriverGetVersion()raises a rawRuntimeError("Failed to dlopen libcuda.so.1") from inside thecuda-pythonnative extension — before the function's ownEnvironmentError/ImportErrorchecks even run.maybe_enable_cuda_graphs()only caught(ImportError, ModuleNotFoundError, EnvironmentError), so thisRuntimeErrorpropagated uncaught and crashed decoding instead of falling back toCudaGraphsMode.NO_WHILE_LOOPSas intended.Testing
Added
tests/collections/asr/decoding/test_label_looping_cuda_graphs_fallback.py(3 tests, no real GPU/ROCm hardware needed — mockscheck_cuda_python_cuda_graphs_conditional_nodes_supporteddirectly):NO_WHILE_LOOPSwhen the check raisesRuntimeError(the new case)EnvironmentError)FULL_GRAPHmode when the check succeedsVerified the new
RuntimeErrortest fails against the pre-fix code and passes against the fix. Ran the fulltests/collections/asr/decoding/+tests/collections/common/test_optional_cuda_graphs.pysuites (283 passed). One unrelated pre-existing failure (test_ctc_decoding.py::test_batched_decoding_labels) reproduces identically with and without this change — confirmed by running it both in isolation and as part of the full suite on unmodifiedmain, so it's not something this PR introduces.Before your PR is "Ready for review"
Pre checks: