Multi-GPUs DDP - How the dataset is distributed accross the GPUs

[KevinCrp]

Hi,

I'm using some GPUs and the Distributed-Data-Parallel strategy and I want to know how the global dataset is split across all GPUs.

Is it split iteratively:

• The N first graphs go-to GPU 0
• The N next go-to GPU 1
• ...
• Until the end ?

Is it randomly distributed?

Is it another way?

I looked at the DistributedSampler class, but I didn't find the answer.

[akihironitta]

I believe this line in PyTorch code explains it all:

indices = indices[self.rank:self.total_size:self.num_replicas]