cufile.log file in the current directory

[wyli]

Describe the bug
running

import monai

will create a cufile.log in the current directory.

to replicate:

docker run --gpus all -ti --rm  projectmonai/monai:latest bash -c "python -c 'import monai'; ls cufile.log"

expected behaviour:

No such file or directory

docker run --gpus all -ti --rm  projectmonai/monai:latest bash -c "ls cufile.log"

[wyli]

may I assign this to you @mingxin-zheng? this will affect all the containers I think.

[mingxin-zheng]

Sure. Just took a stab on it. The file appeared in PyTorch 22.11 but not 22.09.

[mingxin-zheng]

Hi @wyli , is there any component in MONAI using GPUDirect Storage?

[wyli]

looks like it's from cucim... rapidsai/cucim#514

[mingxin-zheng]

Yes. If we use pip uninstall cucim in the container, the cufile log will be gone.

[mingxin-zheng]

It seems we need to set logging:dir in the /etc/cufile.json

 "logging": {
                // log directory, if not enabled will create log file under current working directory
                //"dir": "/home/<xxxx>",

                // NOTICE|ERROR|WARN|INFO|DEBUG|TRACE (in decreasing order of severity)
                "level": "ERROR"
},

[wyli]

so for the pip install monai users, how to address this issue? I think in this case we should delay 'import cucim' so that we don't create this log file.

[mingxin-zheng]

I just checked that pip install would cause the same issue on a freshly-created conda environment on my linux.

I cannot think of another way to get rid of this file for pip install users. But I am quite confused now because the cucim version wasn't changed since 1.0.0 release. I would first test it on more systems.

[wyli]

Hi @mingxin-zheng, I've created a workaround to remove unnecessary cucim imports... #6106 I feel it makes sense for now that the log file is only created when the user triggers some cucim functions.

[mingxin-zheng]

Hi @wyli , I test this on another system with virtualvenv on another machine with pip install. And this time the cuFile.log did not appear. So I am still confused how this is triggered.

I think delaying the imports seems a reasonable solution for MONAI 1.2 and later versions. But the fix wouldn't affect the previous versions. Should we specify the location of this file in existing container building process (e.g. MONAI 1.0.1 + PyTorch 23.02 in Toolkit), or just leave it in the working directory? cc @Nic-Ma .