Fix: #3833 Feedback about Multi GPU training with DDP #6153

Job	Run time
build / pytorch_tutorial_build_worker (4, 15, linux.g5.4xlarge.nvidia.gpu)	35m 3s
build / pytorch_tutorial_build_worker (3, 15, linux.g5.4xlarge.nvidia.gpu)	25m 9s
build / pytorch_tutorial_build_worker (1, 15, linux.g5.12xlarge.nvidia.gpu)	17m 11s
build / pytorch_tutorial_build_worker (10, 15, linux.g5.4xlarge.nvidia.gpu)	31m 53s
build / pytorch_tutorial_build_worker (6, 15, linux.g5.4xlarge.nvidia.gpu)	18m 54s
build / pytorch_tutorial_build_worker (8, 15, linux.g5.4xlarge.nvidia.gpu)	31m 16s
build / pytorch_tutorial_build_worker (7, 15, linux.g5.4xlarge.nvidia.gpu)	15m 49s
build / pytorch_tutorial_build_worker (12, 15, linux.g5.4xlarge.nvidia.gpu)	21m 50s
build / pytorch_tutorial_build_worker (5, 15, linux.g5.4xlarge.nvidia.gpu)	24m 0s
build / pytorch_tutorial_build_worker (14, 15, linux.g5.4xlarge.nvidia.gpu)	16m 49s
build / pytorch_tutorial_build_worker (13, 15, linux.g5.4xlarge.nvidia.gpu)	16m 5s
build / pytorch_tutorial_build_worker (2, 15, linux.g5.4xlarge.nvidia.gpu)	43m 19s
build / pytorch_tutorial_build_worker (11, 15, linux.g5.4xlarge.nvidia.gpu)	42m 36s
build / pytorch_tutorial_build_worker (9, 15, linux.g5.4xlarge.nvidia.gpu)	17m 43s
build / pytorch_tutorial_build_worker (15, 15, linux.g5.4xlarge.nvidia.gpu)	20m 32s
build / pytorch_tutorial_build_manager	22m 16s
	6h 40m 25s

Provide feedback