How to access the logged results such as losses

Results object
1

In a simple training setup, I would like to directly access the lists/dicts of losses and other metrics logged during training and validation so that I can make some custom plots. Where can I find these stored?

2

As far as I understand you need to use such callback:

from pytorch_lightning import Callback
class MetricsCallback(Callback):
    """PyTorch Lightning metric callback."""

    def __init__(self):
        super().__init__()
        self.metrics = []

    def on_validation_end(self, trainer, pl_module):
        self.metrics.append(trainer.callback_metrics)
1 Like
3

@Erlemar

Unfortunatelly this does not work:

trainer.callback_metrics

Contains only whatever was passed to EvalResult as a checkpoint_on or early_stop_on parameter.

4

Interesting - this code worked for me when I used it with old API (dictionaries instead of EvalResult). I hope there will be a solution.

5

Yep, I just tried the old dict API, and it works. EvalResults do not.

1 Like
6

can you try: trainer.logged_metrics or trainer.logger_connector.logged_metrics?

2 Likes
7

Since I am returning dictionaries for now instead of Result objects, the solution suggested by @Erlemar works.

But it does look like we need clarity around a solution for collecting metrics from Train or EvalResults going forward, instead of dicts. If I get to try it out, I will update this thread.

2 Likes
8

@goku

Thanks, trainer.logged_metrics works in both on_epoch_end and on_validation_end

It is still unclear to me what metrics can be found under trainer.logged_metrics in each callback, specifically are those reduced (by the on_epoch=True param in result.log) metrics or unreduced metrics.

It would be awesome to have this clearly described in the docs.

1 Like
9

Thanks, but it doesn’t work for me. I return the same metrics for each epoch, all equal to the metric in the last epoch.
I find another way to do that is to extract all metrics from the logged tensorboard using the EventAccumulator:
https://stackoverflow.com/questions/36700404/tensorflow-opening-log-data-written-by-summarywriter

_New upates
@Erlemar 's answer works for me now if change the trainer.callback_metrics as deepcopy:

class MetricsCallback(Callback):
    """PyTorch Lightning metric callback."""

    def __init__(self):
        super().__init__()
        self.metrics = []

    def on_validation_epoch_end(self, trainer, pl_module):
        each_me = copy.deepcopy(trainer.callback_metrics)
        self.metrics.append(each_me)