Correct approach to calculate metrics in DDP setting

[digital-idiot]

In the case of DDP:

• The metrics should be calculated in validation_step or the metrics should be calculated at validation_step_end after gathering output tensors returned by validation_step?
  • If the metrics are calculated in validation_step, would be it correct to take the mean of the corresponding metrics in validation_step_end? Considering batch partitions for each device can be uneven?
  • Does calling all_gather on the output tensors inside validation_step_end adds an extra dimension before the batch dimension? For example, if my original batch tensor is of the shape N x C x H x W and 2 GPUs are in use then after all_gather the tensor will be of the shape 2 x M x C x H x W (where 2M = N)? What happens if the batch size (N) is an odd number?

[rohitgr7]

the tensor will be of the shape 2 x M x C x H x W (where 2M = N)

if your batch size is N, with 2 GPUs and DDP, the effective batch_size will be 2N
https://pytorch-lightning.readthedocs.io/en/latest/accelerators/gpu.html#batch-size

The metrics should be calculated in validation_step or the metrics should be calculated at validation_step_end

validation_stepis enough. To sync the metrics across devices, use self.log(..., sync_dist=True)
ref: https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html#automatic-logging

Does calling all_gather on the output tensors inside validation_step_end adds an extra dimension before the batch dimension?

it will be (2, N, ...).
ref: https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.html#all-gather

[digital-idiot]

So in case of DDP validation_step_end will not be called at all?

so if you want to accumulate metrics across all the processes you need to set sync_dist=True

Accumulate here means taking sum?

If I move the metrics logging part to validation_step without dist_sync_on_step=True then logging will happen independently on each GPU? In that case will I be able to see two instances of each metric on tensorboard?

If I just add dist_sync_on_step=True in the current implementation, it will work uniformly on all three strategies (DP, DDP, DDP2)?

P.S. - Sorry to be so bothersome, all this is a bit tricky to get my head around.

[rohitgr7]

So in case of DDP validation_step_end will not be called at all?

it will be called.

Accumulate here means taking sum?

mean (by default)

If I just add dist_sync_on_step=True in the current implementation, it will work uniformly on all three strategies (DP, DDP, DDP2)?

for DP and DDP2, it won't have any effect. You should set dist_sync_on_step=True only if you want to sync across multiple devices. Note that it will slow down the training since all_gather takes some time internally. Also if you are training on a single node then DDP2 = DP.

P.S. - Sorry to be so bothersome, all this is a bit tricky to get my head around.

np!

[sarafrr]

Hi @rohitgr7,
if I use the self.log_dict with both torchmetrics and 'not torchmetrics', what should I do?

1. instead of using one single logging, must I separate the loggings of torchmetrics, having set in the definition dist_sync_on_step=True and another log for what is not a torchmetric with dist_sync=True?
2. is it enough to put all the metrics in one single self.log_dict with dist_sync=True and without setting in the definition of the torchmetrics dist_sync_on_step=True?

What is the best/correct solution?
In the end, is it functioning dist_sync=True parameter also for torchmetrics objects (I suppose yes, so it should be enough the second solution)?

Here the 2 examples:

1. separate logs

self.char_err_rate = torchmetrics.CharErrorRate(dist_sync_on_step=True)
def validation_step(self, batch, batch_idx):
...
val_logs_torchmetrics = {
            "val_cer": val_cer,
            "val_wer": val_wer
            }
        # torchmetrics are already synchronized between processes
        self.log_dict(
            val_logs_torchmetrics,
            on_epoch=True,
            prog_bar=True
            )
        # other metrics have to be synchronized between processes
        self.log(
            'val_loss',
            val_loss,
            on_epoch=True,
            prog_bar=True,
            sync_dist=True
            )
...

3. unique log:

self.char_err_rate = torchmetrics.CharErrorRate()

def validation_step(self, batch, batch_idx):
...
val_logs = {
            "val_loss":val_loss,
            "val_cer": val_cer,
            "val_wer": val_wer
            }
self.log_dict(
            val_logs,
            on_epoch=True,
            prog_bar=True,
            dist_sync=True
            )

[rohitgr7]

sync_dist isn't applied to torchmetrics instances so should be fine keeping all in one

[sarafrr]

I have another related question,
I am saving the checkpoints using the appropriate class ModelCheckpoint.
Here is the checkpoint:

checkpoint_callback_val_cer = ModelCheckpoint(
                save_top_k=1,
                monitor="val_cer",
                filename='{epoch}-{val_cer:.4f}-{val_wer:.4f}-{val_loss:.4f}'
            )

To see if the metrics reported in the checkpoint name are correct, I tested train.test() the saved model loading the validation set, the torchmetrics (val_cer and val_wer) are correct (the values logged by the code and the values in the checkpoint's name are the same), instead val_loss is different (more or less half what is reported in the checkpoint name).

I am using 2 gpus.

Thanks for the help,
S