Train a new model with for loop over episodic few-shot data

Trainer
1

Hi,
I have an episodic few-shot dataset, and I want to iterate over each episode and train a new pre-trained T5 model in each iteration, the best model is chosen by the min val_loss among all the models.

  1. If I load the pre-trained T5 and the checkpoint outside the loop and then create a new trainer in the loop does the checkpoint continue the previous model or save the checkpoint of the new model?
  2. should I reset the trainer after each iteration → trainer.fit()?
checkpoint_callback = pl.callbacks.ModelCheckpoint(dirpath=f'{opt.output_dir}/train', monitor="avg_val_loss", mode="min")
early_stopping_callback = EarlyStopping(monitor="avg_val_loss")
model = T5FineTuner(opt=opt)
# iterate over the training episodes
for x in range(len(train_data)):
  # save_top_k=1,
    train_params = dict(
        accumulate_grad_batches=opt.gradient_accumulation_steps,
        gpus=opt.n_gpu,
        max_epochs=opt.num_train_epochs,
        precision=16 if opt.fp16 else 32,
        amp_level=opt.opt_level,
        gradient_clip_val=opt.max_grad_norm,
        log_every_n_steps=10,
        callbacks=[LoggingCallback(), early_stopping_callback, checkpoint_callback],

    )
    train_dataloader = model.train_dataloader(train_data[x])
    val_dataloader = model.val_dataloader(train_data[x])
    trainer = pl.Trainer(**train_params, default_root_dir=f'{opt.output_dir}/train')
    trainer.fit(model, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader)

best_model_path = checkpoint_callback.best_model_path
best_score = checkpoint_callback.best_model_score

@jirka @williamfalcon