DDP - Synchronization on DGX - Use CPUs or GPU-to-GPU interconnect

[KevinCrp]

Hi,

I'm training a model on several GPUs with the DDP strategy, I'm using an Nvidia DGX where GPUs are interconnected with NVLinks.

I want to know when the gradients are synchronized, does the communication use the NVLinks or GPU-to-GPU or use the CPUs to communicate?

[akihironitta]

This thread in PyTorch forum may be what you find useful:
https://discuss.pytorch.org/t/simple-code-example-with-nvlink-support/125304