Multiple test datasets, metric name depend of dataset

[cdancette]

Hi,
What is the correct way to have multiple test sets and have the metric names (for ex accuracy) that depend on the test dataset name ?

For example:

trainer.fit(module, datamodule)
trainer.test(module, test1_loader)
trainer.test(module, test2_loader)

and in in my module:

def test_step(batch, batch_idx):
    loss, acc = self._shared_eval_step(batch, batch_idx)
    metrics = {"test_acc": acc, "test_loss": loss}
    self.log_dict(metrics)
    return metrics

Here, both datasets will use the same "test_acc" metric. I'd like to have two different names, for example test_acc_1 and test_acc_2

[rohitgr7]

you can pass in multiple test dataloader together and additionally you will get a an additional argument dataloader_idx inside which you can use to update the name

class LitModel(pl.LightningModule):
    ...

    def test_step(self, batch, batch_idx, dataloader_idx):
        ...
        metrics = {f"test_acc_{dataloader_idx + 1}": acc, ...}  # + 1 because they 0-indexed
        ...

trainer.test(module, dataloaders=[test1_dataloader, test2_dataloader])

[fmocking]

Hello thank you for the suggestion! After applying the code above this is what I'm getting 'test_acc_1/dataloader_idx_0'. I'm the following code for it: self.log(f'test_acc_{dataloader_idx+1}/', loss). I understand why this suffix is added but in this case it is unnecessary but I couldn't find a way to remove it.

Also I have a suggestion to support dictionaries. So we can have custom keys for data loaders. So we would get keys instead of position in the array for idxes. It would make it easier to make sense of the results. The dirty fix would be save the ordered dictionary to model object. Then when testing use the values of the dictionary as test dataloader. On test_step, map it back using the ordered dictionary. I believe updating internally to support dictionary would be easier than this process.