Trainer.validate() get different result from trainer.fit

[minwang-ai]

hi I just run trainer.validate and got bad results, however, the model is the best saved automatically after trainer.fit()

My datamodule’s setup is like this:

def setup(self, stage: Optional[str] = None):
        """Split the train and valid dataset."""
        if stage in (None, "fit", "val"):
            # init datasets
            dataset_train_orignal = dataset_desc.Dataset('train')

            # split original training dataset -> train and val
            self.train_val_split = 0.89  # train:val ~= 8:1

            train_size = int(self.train_val_split * len(dataset_train_orignal))
            val_size = len(dataset_train_orignal) - train_size
            self.dataset_train, self.dataset_val = random_split(dataset_train_orignal, [train_size, val_size])

        if stage in(None, "test"):
            self.dataset_test = dataset_desc.Dataset('test')

And my val_step():

def val_dataloader(self):
        """orignal val set uses a subset of the training set for validation."""
        loader = DataLoader(
            self.dataset_val,
            batch_size=self.batch_size_val,
            shuffle=False,
            num_workers=self.num_workers_val,
            drop_last=True,
            pin_memory=True,
        )
        return loader

[rohitgr7]

can you share the code for trainer.validate call?

[rohitgr7]

its very hard for me to tell what's causing the issue without a reproducible example.
Can you reproduce the issue with this: https://colab.research.google.com/drive/1HvWVVTK8j2Nj52qU4Q4YCyzOm0_aLQF3?usp=sharing

[minwang-ai]

Let me have a try.
Could you tell me how to avoid these error (the iou calculation call _bincount_cuda which is non-deterministic after update pytorchlightning )?

I updated pytorchlightning from 1.4.9 to 1.5.1 and then I cannot run trainer.validate at all for some models’ checkpoints while for some other models , trainer.validate() works

Traceback (most recent call last):
  File "/fs/applications/pycharm-ce/2019.1.1/helpers/pydev/pydevd.py", line 1741, in <module>
    main()
  File "/fs/applications/pycharm-ce/2019.1.1/helpers/pydev/pydevd.py", line 1735, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/fs/applications/pycharm-ce/2019.1.1/helpers/pydev/pydevd.py", line 1135, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/fs/applications/pycharm-ce/2019.1.1/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/mwa7rng/FFB6D/ffb6d/main.py", line 144, in <module>
    main()
  File "/home/mwa7rng/FFB6D/ffb6d/main.py", line 135, in main
    trainer.validate(model, datamodule=dm, ckpt_path=args.checkpoint)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 814, in validate
    return self._call_and_handle_interrupt(self._validate_impl, model, dataloaders, ckpt_path, verbose, datamodule)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 857, in _validate_impl
    results = self._run(model, ckpt_path=self.validated_ckpt_path)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1193, in _run
    self._dispatch()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1268, in _dispatch
    self.training_type_plugin.start_evaluating(self)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 206, in start_evaluating
    self._results = trainer.run_stage()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1279, in run_stage
    return self._run_evaluate()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1327, in _run_evaluate
    eval_loop_results = self._evaluation_loop.run()
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 109, in advance
    dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/loops/base.py", line 145, in run
    self.advance(*args, **kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 123, in advance
    output = self._evaluation_step(batch, batch_idx, dataloader_idx)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 215, in _evaluation_step
    output = self.trainer.accelerator.validation_step(step_kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/accelerators/accelerator.py", line 236, in validation_step
    return self.training_type_plugin.validation_step(*step_kwargs.values())
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 447, in validation_step
    return self.lightning_module.validation_step(*args, **kwargs)
  File "/home/mwa7rng/FFB6D/ffb6d/models/ffb6d_lightning.py", line 619, in validation_step
    self.log("val_miou", self.val_miou(preds['pred_rgbd_segs'], target), sync_dist=True, prog_bar=True, logger=True)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/torchmetrics/metric.py", line 197, in forward
    self.update(*args, **kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/torchmetrics/metric.py", line 255, in wrapped_func
    return update(*args, **kwargs)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/torchmetrics/classification/confusion_matrix.py", line 131, in update
    confmat = _confusion_matrix_update(preds, target, self.num_classes, self.threshold, self.multilabel)
  File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/torchmetrics/functional/classification/confusion_matrix.py", line 48, in _confusion_matrix_update
    bins = torch.bincount(unique_mapping, minlength=minlength)
RuntimeError: _bincount_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.

Process finished with exit code 1

Another questions is about iou calculation.
Should I use iou metric like the following .update() and .compute() to accumulate pred and target or like the validation_step()?

metric = torchmetrics.IoU(...)
for image in images:
    pred = model(image)
    metric.**update**(pred, image)
miou = metric.**compute**()

def validation_step(self, batch, batch_idx):
        target = batch['labels']
        # 1. forward pass
        preds = self(batch)
        # 2. compute loss and accuracy
        if self.task == 'ss':
            loss = self.loss_segmentation(
                    preds['pred_rgbd_segs'], target.view(-1)
                    ).sum()
            # metrics
            self.log("val_loss", loss, sync_dist=True, prog_bar=True, logger=True)
            self.log("val_acc", self.val_acc(preds['pred_rgbd_segs'], target), sync_dist=True, prog_bar=True, logger=True)
            self.log("val_miou", self.val_miou(preds['pred_rgbd_segs'], target), sync_dist=True, prog_bar=True, logger=True)
        # use epoch as x axis
        self.log('step', self.trainer.current_epoch, on_step=False, on_epoch=True)

[rohitgr7]

looks like its coming from pytorch directly, suggesting to turn off deterministic using Trainer(deterministic=False).
You can open this issue on pytorch to support this.

[minwang-ai]

File "/home/mwa7rng/FFB6D/env/lib/python3.6/site-packages/torchmetrics/functional/classification/confusion_matrix.py", line 48, in _confusion_matrix_update

I think it is from torchmetric, because there is no error before updating.

Could you have a look at it and following questions ?

Can I simply passing checkpoints without unit the model like this? if args.trainer == "test": trainer.test(datamodule=dm, ckpt_path=args.checkpoint) (Yes)

I just noticed that for trainer.predict(), I also need to setup the stage.

if stage in(None, "test"):
self.dataset_test = dataset_desc.Dataset('test')

Should I change datamodule.setup() to If stage in (None,”test”, “predict”) ?

Another questions is about iou calculation. Should I use iou metric like the following .update() and .compute() to accumulate pred and target or like the validation_step()?

metric = torchmetrics.IoU(...)
for image in images:
    pred = model(image)
    metric.**update**(pred, image)
miou = metric.**compute**()

def validation_step(self, batch, batch_idx):
        target = batch['labels']
        # 1. forward pass
        preds = self(batch)
        # 2. compute loss and accuracy
        if self.task == 'ss':
            loss = self.loss_segmentation(
                    preds['pred_rgbd_segs'], target.view(-1)
                    ).sum()
            # metrics
            self.log("val_loss", loss, sync_dist=True, prog_bar=True, logger=True)
            self.log("val_acc", self.val_acc(preds['pred_rgbd_segs'], target), sync_dist=True, prog_bar=True, logger=True)
            self.log("val_miou", self.val_miou(preds['pred_rgbd_segs'], target), sync_dist=True, prog_bar=True, logger=True)
        # use epoch as x axis
        self.log('step', self.trainer.current_epoch, on_step=False, on_epoch=True)

[minwang-ai]

Thank you for your reply!

looks like its coming from pytorch directly, suggesting to turn off deterministic using Trainer(deterministic=False).

If I want to make sure the result of each run of training is the same, I need to set deterministic-True.

How about validation and test? Can I simply set it to False or comment it out?

I still have a question for seed_everything.
I have a structure like this:

• Main
  ——lightning model
  ——lightning datamodule
  ————original Dataset class

For the np.random inside dataset class, will it also be controlled by this seed?