Train a new model with for loop over episodic few-shot data

I have an episodic few-shot dataset, and I want to iterate over each episode and train a new pre-trained T5 model in each iteration, the best model is chosen by the min val_loss among all the models.

  1. If I load the pre-trained T5 and the checkpoint outside the loop and then create a new trainer in the loop does the checkpoint continue the previous model or save the checkpoint of the new model?
  2. should I reset the trainer after each iteration →
checkpoint_callback = pl.callbacks.ModelCheckpoint(dirpath=f'{opt.output_dir}/train', monitor="avg_val_loss", mode="min")
early_stopping_callback = EarlyStopping(monitor="avg_val_loss")
model = T5FineTuner(opt=opt)
# iterate over the training episodes
for x in range(len(train_data)):
  # save_top_k=1,
    train_params = dict(
        precision=16 if opt.fp16 else 32,
        callbacks=[LoggingCallback(), early_stopping_callback, checkpoint_callback],

    train_dataloader = model.train_dataloader(train_data[x])
    val_dataloader = model.val_dataloader(train_data[x])
    trainer = pl.Trainer(**train_params, default_root_dir=f'{opt.output_dir}/train'), train_dataloaders=train_dataloader, val_dataloaders=val_dataloader)

best_model_path = checkpoint_callback.best_model_path
best_score = checkpoint_callback.best_model_score

@jirka @williamfalcon