Implementing a Metric and including a nn.Module doesn't work correctly in parallel

[import-antigravity]

I implemented the FID metric, which involves using a pre-trained Inception network. I have the following code to move it to CUDA:

def __init__(self, ...):
    ...
    self._model = InceptionV3(...)
    if cuda.is_available():
        self._model.to('cuda')

When I train using more than one GPU in DDP, this causes an exception RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable. I'm not sure what's causing this. I know that this stuff is supposed to be taken care of automatically but for some reason it's not working for me.

[import-antigravity]

Bump

[jhschwartz]

Did you ever resolve this? Having the same problem.

[import-antigravity]

Sidenote: using self.to() causes the same error

[SkafteNicki]

Are you using a shared cluster/machine by any chance? That error can be due to another user using the gpu resources (and the gpus set to exclusive mode)

[SkafteNicki]

Normally, in lightning you should not have to move modules to devices as we do this automatically for you. What happens if you remove the

if cuda.is_available():
        self._model.to('cuda')

part of your code?

[import-antigravity]

If I don’t move the module manually, the module does not get moved by the trainer and I get an error. The trainer really doesn’t have any way of controlling the Metric because it’s part of a callback so I’m not sure how that would work. Also I know that Metric extends nn.Module, so in the current version of my code that line is self.to('cuda').

[SkafteNicki]

alright, but since you are running in multi-gpu setting, calling self.to('cuda') is not correct as it instead should be self.to(cuda:0), self.to(cuda:1) ect depending on what gpu currently are being used

[import-antigravity]

Ah interesting. Do you have a documentation link for that so I can learn more?

Alternatively, is there a way I can somehow register the Metric instance with the Trainer so the trainer can actually handle the device management? Thanks!

[SkafteNicki]

what I would do in your place was to equip my model with the inception net necessary to calculate fid so it gets moved to the correct device during training. Then in my callback I can access it through the pl_module argument which most hooks have.