Does EvalResults also work with early stopping?

Results object
1

A related question about EvalResults: Does it also work with early stopping? I was not able to do so, yet. When I use

result.log('val_loss', loss, on_epoch=True, on_step=False, reduce_fx=torch.mean)

EarlyStopping warns that ‘val_loss’ is not found. I debugged a bit and saw that only callback_metrics (i.e. {'val_early_stop_on': None, 'val_checkpoint_on': None} are considered as early stopping metrics when EvalResult is used. I could set early_stop_on, but I would have to do that in validation_epoch_end to be able to calculate the mean, which would diminish the advantage of reduce_fx. I can also create an “improvment”-issue, if you want.

2

Early stopping with eval results works by setting the key:

EvalResult(early_stop_on=X)

and enabling it in the trainer:

Trainer(early...=True)

If you try to change the monitor key of your own EarlyStopping, that will not work with results.

    # has no effect with EvalResult... 
    # only the EvalResult(early_stop_on=x) key can be used
    EarlyStopping(monitor='a')
3

How would you choose the monitor programmatically then? Say between {tr,val}_loss and {tr,va}_accuracy.

4

to stop on training_step:

def training_step(...)
    acc = x
    loss = x

    # you choose
   result = TrainResult(early_stop_on=acc)

or same for val step

def validation_step(...)
    acc = x
    loss = x

    # you choose
   result = EvalResult(early_stop_on=acc)

If you do it both places, val step will take priority

We chose to do it this way because it enables new behavior like changing a key during training:

acc = x
loss = x

if epoch <= 20
   result = EvalResult(early_stop_on=loss)
else:
   result = EvalResult(early_stop_on=-acc)