Support for newer CUDA capability (e.g. sm_86)

[Leengit]

I successfully build a docker image with monai-deploy package that runs on the computer on which I built it. However when I try to run the same docker image on a computer with a significantly newer / more powerful GPU, it fails. It appears that the underlying docker image nvcr.io/nvidia/pytorch:21.07-py3 uses a version of CUDA 11.3 and torch that do not support sm_86. Upgrading to torch==1.12.1 within the docker image that I create (and committing the change to a new image that I then use) does not help. Despite my attempts with apt-get, I have been unable to install a newer version of CUDA within the created docker image.

Your help with getting support for an NVIDIA RTX A5000 would be much appreciated! The error from running the docker image that I created with monai deploy includes

NVIDIA RTX A5000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA RTX A5000 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
...
  File "~/venv/lung/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 453, in _conv_forward
                            weight, bias, self.stride,
                            _pair(0), self.dilation, self.groups)
        return F.conv2d(input, weight, bias, self.stride,
               ~~~~~~~~ <--- HERE   
                        self.padding, self.dilation, self.groups)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[MMelQin]

Thanks @Leengit for the issue. The default base image for build MONAI App Package is of 2021 vintage, and as you pointed out, out of date. I previously tried out a base container of 22.xx vintage (with newer version of PyTorch), and found some incompatibility issues with the Python packages.

It is important for the App SDK to keep up with the new releases of the dependencies, so I'll convert this to an issue so that we can schedule and track it.

[MMelQin]

Thanks for looking into the specifics and testing it out.

Since we base the MONAI App Package on NVIDAI PyTorch container image, this published framework compatibility matrix is often time useful on finding out the details.

[Leengit]

Even with

monai-deploy package \
    deploy_app.py \
    -b nvcr.io/nvidia/pytorch:22.09-py3 \
    --tag ghcr.io/my_organization/my_repository/my_image_name:22.09-py3 \
    --model model.zip

... and then updating Python packages within the produced docker image as described above, I am getting

NVIDIA RTX A5000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA RTX A5000 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

That base docker image should be good enough, so I don't understand what is going wrong here.

Although I am running on a computer with the RTX A5000, I am displaying on a weaker machine via SSH X11 forwarding; could that have something to do with it?

[vikashg]

@MMelQin Correct me if I am wrong. But, i would suggest this.

1. Check your cuda version by running nvidia-smi
2. Create a clean conda environment and install the correct pytorch version based on the cuda version. The correct version can be found on pytorch.org.
3. Run the map in the environment following one of the tutorials.
   Only after step 3, try packaging it. This will help a lot.

[Leengit]

It does work if I skip my customization step of pip install within the docker container!

[MMelQin]

The PR #381 has been created to fully address the issue here as well as other few minor issues in the Packager.