RuntimeError: CUDA error: device-side assert triggered

[JiangSYKY]

torch是2.3.0
K2: Version: 1.24.4.dev20241029+cuda12.1.torch2.3.0
lhotse：Version: 1.30.0.dev0+git.13c7616.clean
在使用mdcc粤语数据集，执行prepare.sh的第十二步训练rnn_lm语言模型时，出现错误
Traceback (most recent call last):
File "/root/autodl-tmp/.autodl/icefall/egs/mdcc/ASR/../../../icefall/rnn_lm/train.py", line 689, in
main()
File "/root/autodl-tmp/.autodl/icefall/egs/mdcc/ASR/../../../icefall/rnn_lm/train.py", line 682, in main
run(rank=0, world_size=1, args=args)
File "/root/autodl-tmp/.autodl/icefall/egs/mdcc/ASR/../../../icefall/rnn_lm/train.py", line 647, in run
train_one_epoch(
File "/root/autodl-tmp/.autodl/icefall/egs/mdcc/ASR/../../../icefall/rnn_lm/train.py", line 474, in train_one_epoch
loss, loss_info = compute_loss(
^^^^^^^^^^^^^
File "/root/autodl-tmp/.autodl/icefall/egs/mdcc/ASR/../../../icefall/rnn_lm/train.py", line 376, in compute_loss
nll = model(x, y, sentence_lengths)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/.autodl/icefall/icefall/rnn_lm/model.py", line 151, in forward
rnn_out, _ = self.rnn(embedding)
^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.12/site-packages/torch/nn/modules/rnn.py", line 911, in forward
result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[csukuangfj]

File "/root/autodl-tmp/.autodl/icefall/egs/mdcc/ASR/../../../icefall/rnn_lm/train.py", line 376, in compute_loss
nll = model(x, y, sentence_lengths)

你把 x, y, sentence_lengths 的值打印出来？

为方便调试，请使用 batch size == 1 做测试。