String "best" at argument "ckpt_path" for test method of Trainer class

Trainer
1

Hi,

The test method of the Trainer class, has the input argument ckpt_path. According to the docs:

ckpt_path (Optional[str]) – Either best or path to the checkpoint you wish to test. If None and the model instance was passed, use the current weights. Otherwise, the best model from the previous trainer.fit call will be loaded.

Also, in the Documentation of PyTorch Lightning for the test set, using Trainer, there is the following:

# run full training
trainer.fit(model)

# (1) load the best checkpoint automatically (lightning tracks this for you)
trainer.test(ckpt_path="best")

My question is, according to what the “best” checkpoint is decided? That is, is the “best” decided on maximising or minimising some value? What would be that value? Can someone configure the policy (i.e. minimising or maximising) and the value? How one should use this “best” string?

Links for reference:

  1. Test set documentation
  2. Test method of Trainer class documentation

P.S. Please note that I’m not referring to using the ModelChekpoint callback, but explicitly to the above

2

Hi :slight_smile:
It depends on what you want to save according to the model checkpoint :slight_smile:
For exemple :

checkpoint_callback = ModelCheckpoint(dirpath="checkpoints", # where the ckpt will be saved
                                      filename="best-checkpoint", # the name of the best ckpt
                                      save_top_k=1, # save only the best ckpt
                                      verbose=True,
                                      monitor="Validation loss", # ckpt will be save according to the validation loss that you need to calculate on the validation step when you train your model
                                      mode="min" # validation loos need to be min
                                      )

after that, in the test mode, you just need to lad the name of the best ckpt :slight_smile:
Hope this will helps you!