Grouping custom metrics by configuration

implementation help
1

Hi, I am new to PyTorch Lightning and trying to integrate my workflow with the WandB logger.

I am using a LightningModule which retrieves the input data and labels. I have also associated each input/output pair with json configuration file which describes the capture environment (e.g., if it is multi host or single host, bandwidth, delay, BDP factor). I know how to log values such as F1 score and accuracy for each sample but I am confused how to associate each value with a configuration. Is there any guides available that addresses this or something similar?

For instance, in test_step from the LightningModule I have

def test_step(self, batch, batch_idx):
  x, y, config_file =  batch
  y_pred = self.forward(x)
  loss = self.loss(y_pred, y)
  self.log("test/loss", loss)
  return loss

I would like to do something like this:

def test_step(self, batch, batch_idx):
  x, y, config_file =  batch
  y_pred = self.forward(x)
  loss = self.loss(y_pred, y)
  self.log(
    "test/loss", 
    loss, 
    configuration = {
      "delay": "10ms",
      "BDP": 3,
       # etc ...
    }
  )
  return loss

The config_file is a json file with configuration.

My guess is that I could probably change "test/loss" to "test/loss/10ms/3", but I am not sure if this is the best way to go about it. I want to be able to compare different environment settings somehow.

I have looked a bit into torchmetrics, is it possible to address this issue with it?

Thanks in advance!

2

it’s not possible with torchmetrics, neither this is supported natively with pytorch lightning. If it’s supported with your logger then you can directly use the logger to log the metrics along with your configuration per step using:

self.logger.experiment.log(...)

We have moved the discussions to GitHub Discussions. You might want to check that out instead to get a quick response. The forums will be marked read-only after some time.

Thank you