多GPU使用报错

[Lemonnn7]

Caught signal 11 (Segmentation fault: address not mapped to object at address 0x7f207c0ab470) ==== backtrace (tid: 100548) ==== 0 /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2a4) [0x7f2106e83824] 1 /opt/hpcx/ucx/lib/libucs.so.0(+0x2b9ff) [0x7f2106e839ff] 2 /opt/hpcx/ucx/lib/libucs.so.0(+0x2bd34) [0x7f2106e83d34] 3 /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f26c5cff3c0] 4 /lib/x86_64-linux-gnu/libibverbs.so.1(+0x141c8) [0x7f21043c51c8] 5 /lib/x86_64-linux-gnu/libibverbs.so.1(+0x14ba4) [0x7f21043c5ba4] 6 /lib/x86_64-linux-gnu/libibverbs.so.1(ibv_get_device_list+0x55) [0x7f21043c2635] 7 /U_PZL2022ZZ0003/tools/miniforge3/envs/llm/lib/python3.10/site-packages/torch/lib/../../../.././libnccl.so.2(+0x84825) [0x7f21c54eb825] 8 /U_PZL2022ZZ0003/tools/miniforge3/envs/llm/lib/python3.10/site-packages/torch/lib/../../../.././libnccl.so.2(+0x9f27c) [0x7f21c550627c] 9 /U_PZL2022ZZ0003/tools/miniforge3/envs/llm/lib/python3.10/site-packages/torch/lib/../../../.././libnccl.so.2(+0x508ae) [0x7f21c54b78ae] 10 /U_PZL2022ZZ0003/tools/miniforge3/envs/llm/lib/python3.10/site-packages/torch/lib/../../../.././libnccl.so.2(+0x402c9) [0x7f21c54a72c9] 11 /U_PZL2022ZZ0003/tools/miniforge3/envs/llm/lib/python3.10/site-packages/torch/lib/../../../.././libnccl.so.2(+0x49f57) [0x7f21c54b0f57] 12 /U_PZL2022ZZ0003/tools/miniforge3/envs/llm/lib/python3.10/site-packages/torch/lib/../../../.././libnccl.so.2(+0x3bbe7) [0x7f21c54a2be7] 13 /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f26c5cf3609] 14 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f26c5ab3293] =================================

偶尔会报错如上，有概率不出报错

[545999961]

可以提供下更具体的报错代码么，也可以尝试目前更新的BGE模型推理方法