Retraining a model with new data

Trainer
1

Hello,

I have a set of samples (X1, Y1) that I train a neural network on (Input X1, predict Y1). This is the code that I use:

net = NN(nparams=nparams)

# Setup trainer.
trainer = pl.Trainer(
    enable_model_summary=None,
    max_epochs=1000,
    log_every_n_steps=1,
    callbacks=[EarlyStopping(monitor="val_loss", patience=100)],
)

# Fit model.
trainer.fit(net, datamodule=data_module)
trainer.save_checkpoint("model.ckpt")

Now, I obtain new data X2, Y2, and I would like to retrain the model on (X1 + X2, Y1 + Y2) where by + denote concatenation. I do this by

trainer = pl.Trainer(
                    enable_model_summary=None,
                    max_epochs=1000 * (t + 2),
                    log_every_n_steps=1,
                    resume_from_checkpoint="model.ckpt",
                    callbacks=[EarlyStopping(monitor="val_loss", patience=100)],
                )
trainer.fit(net, datamodule=datamodule)
trainer.save_checkpoint("model.ckpt")

and then iterate this many more iterations. Here t is the iteration number, so if t = 0, I have gathered (X2, Y2) and would like to train for 2000 epochs (so 1000 from the first run, 1000 for the second run). However, after iteration 1 no retraining occurs. Why is this?

Thanks!

2

Previously I wouldn’t load the previous checkpoint, but I would simply call trainer.fit on the pretrained model net, with a new datamodule. However, I am not sure whether this is the correct way to go…