Hi, I am trying to use the Tuner in order to find the optimal batch size and learning rate to finetune my model with.
I run the code a few days ago and It worked fine, then I had to make some adjustment to the model’s
training_step method. After making those adjustments, I went to run the code of the Tuner again and it’s been stuck on batch size 4 since then. Any ideas why this is happening?
The Tuner’s code is the following:
def main(args): os.environ["TOKENIZERS_PARALLELISM"] = '1' model = CLIPWrapper(batch_size=args.batch_size) datamodule = CaptioningDataModule(data=args.data, num_workers=args.num_workers, batch_size=args.batch_size) trainer = Trainer(default_root_dir=args.default_root_dir, max_epochs=5) tuner = Tuner(trainer) optimal_batch_size = tuner.scale_batch_size(model, datamodule=datamodule, init_val=args.batch_size, batch_arg_name="batch_size") model.batch_size = optimal_batch_size datamodule.batch_size = optimal_batch_size # finds learning rate automatically # sets hparams.lr or hparams.learning_rate to that learning rate # Pick point based on plot, or get suggestion optimal_lr = tuner.lr_find(model, datamodule=datamodule).suggestion()
I do have a
batch_size property both in my
CaptioningDataModule classes. A little side note, is that the
batch_size property inside the
CLIPWrapper is actually the
minibatch size that I use inside the
training_step, I had to rename it to
batch_size to match the
BatchSizeFinder expectations, even though now I do not know if it was the right thing to do since I also get the following warning:
UserWarning: Field `model.batch_size` and `model.hparams.batch_size` are mutually exclusive! `model.batch_size` will be used as the initial batch size for scaling. If this is not the intended behavior, please remove either one.