Multiple train/validation dataloaders/multiple evaluation metrics

implementation help
1

Hi
I am implementing a model which has multiple validation dataloader, so I am considering multiple tasks and each of them needs to be evaluated with a different metric, then I have one dataloader for training them.
Could you assist me with providing me with examples, how I can implement multiple validation dataloaders and mutliple evaluation metrics with pytorch lightening?
thanks a lot

2

validation_step can have different behavior for each dataloader using the dataloader_idx argument:

class MyModule(pl.LightningModule):
    def __init__(self):
         ...
         # Lets say we want one metric each of our three validation loaders
         self.metrics = nn.ModuleList([
               pl.metrics.Accuracy(),
               pl.metrics.FBeta(),
               pl.metrics.Recall()
         ])

    def validation_step(self, batch, batch_idx, dataloader_idx):
         x, y = batch
         y_hat = self.model(x)
         metric = self.metrics[dataloader_idx] # select metric
         metric.compute(y, y_hat) # compute metric

model = MyModule()

# Train with multiple dataloaders
trainer = pl.Trainer()
trainer.fit(model, val_dataloaders=[
     DataLoader(...),
     DataLoader(...),
     DataLoader(...),
])
1 Like