In a simple training setup, I would like to directly access the lists/dicts of losses and other metrics logged during training and validation so that I can make some custom plots. Where can I find these stored?
As far as I understand you need to use such callback:
from pytorch_lightning import Callback
class MetricsCallback(Callback):
"""PyTorch Lightning metric callback."""
def __init__(self):
super().__init__()
self.metrics = []
def on_validation_end(self, trainer, pl_module):
self.metrics.append(trainer.callback_metrics)
Unfortunatelly this does not work:
trainer.callback_metrics
Contains only whatever was passed to EvalResult as a checkpoint_on
or early_stop_on
parameter.
Interesting - this code worked for me when I used it with old API (dictionaries instead of EvalResult). I hope there will be a solution.
Yep, I just tried the old dict API, and it works. EvalResults do not.
can you try: trainer.logged_metrics
or trainer.logger_connector.logged_metrics
?
Since I am returning dictionaries for now instead of Result objects, the solution suggested by @Erlemar works.
But it does look like we need clarity around a solution for collecting metrics from Train or EvalResults going forward, instead of dicts. If I get to try it out, I will update this thread.
Thanks, trainer.logged_metrics
works in both on_epoch_end
and on_validation_end
It is still unclear to me what metrics can be found under trainer.logged_metrics
in each callback, specifically are those reduced (by the on_epoch=True
param in result.log
) metrics or unreduced metrics.
It would be awesome to have this clearly described in the docs.
Thanks, but it doesn’t work for me. I return the same metrics for each epoch, all equal to the metric in the last epoch.
I find another way to do that is to extract all metrics from the logged tensorboard using the EventAccumulator:
https://stackoverflow.com/questions/36700404/tensorflow-opening-log-data-written-by-summarywriter
_New upates
@Erlemar 's answer works for me now if change the trainer.callback_metrics as deepcopy:
class MetricsCallback(Callback):
"""PyTorch Lightning metric callback."""
def __init__(self):
super().__init__()
self.metrics = []
def on_validation_epoch_end(self, trainer, pl_module):
each_me = copy.deepcopy(trainer.callback_metrics)
self.metrics.append(each_me)