Tuner

class lightning.pytorch.tuner.tuning.Tuner(trainer)

Bases: object

Tuner class to tune your model.

lr_find(model, train_dataloaders=None, val_dataloaders=None, dataloaders=None, datamodule=None, method='fit', min_lr=1e-08, max_lr=1, num_training=100, mode='exponential', early_stop_threshold=4.0, update_attr=True, attr_name='')

Enables the user to do a range test of good initial learning rates, to reduce the amount of guesswork in picking a good starting learning rate.

Parameters

• model (LightningModule) – Model to tune.

• train_dataloaders (Union[Any, LightningDataModule, None]) – A collection of torch.utils.data.DataLoader or a LightningDataModule specifying training samples. In the case of multiple dataloaders, please see this section.

• val_dataloaders (Optional[Any]) – A torch.utils.data.DataLoader or a sequence of them specifying validation samples.

• dataloaders (Optional[Any]) – A torch.utils.data.DataLoader or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction.

• datamodule (Optional[LightningDataModule]) – An instance of LightningDataModule.

• method (Literal[‘fit’, ‘validate’, ‘test’, ‘predict’]) – Method to run tuner on. It can be any of ("fit", "validate", "test", "predict").

• min_lr (float) – minimum learning rate to investigate

• max_lr (float) – maximum learning rate to investigate

• num_training (int) – number of learning rates to test

• mode (str) –

  Search strategy to update learning rate after each batch:

  • 'exponential': Increases the learning rate exponentially.

  • 'linear': Increases the learning rate linearly.

• early_stop_threshold (float) – Threshold for stopping the search. If the loss at any point is larger than early_stop_threshold*best_loss then the search is stopped. To disable, set to None.

• update_attr (bool) – Whether to update the learning rate attribute or not.

• attr_name (str) – Name of the attribute which stores the learning rate. The names ‘learning_rate’ or ‘lr’ get automatically detected. Otherwise, set the name here.

Raises

MisconfigurationException – If learning rate/lr in model or model.hparams isn’t overridden, or if you are using more than one optimizer.

Return type

Optional[_LRFinder]

scale_batch_size(model, train_dataloaders=None, val_dataloaders=None, dataloaders=None, datamodule=None, method='fit', mode='power', steps_per_trial=3, init_val=2, max_trials=25, batch_arg_name='batch_size')

Iteratively try to find the largest batch size for a given model that does not give an out of memory (OOM) error.

Parameters

• model (LightningModule) – Model to tune.

• train_dataloaders (Union[Any, LightningDataModule, None]) – A collection of torch.utils.data.DataLoader or a LightningDataModule specifying training samples. In the case of multiple dataloaders, please see this section.

• val_dataloaders (Optional[Any]) – A torch.utils.data.DataLoader or a sequence of them specifying validation samples.

• dataloaders (Optional[Any]) – A torch.utils.data.DataLoader or a sequence of them specifying val/test/predict samples used for running tuner on validation/testing/prediction.

• datamodule (Optional[LightningDataModule]) – An instance of LightningDataModule.

• method (Literal[‘fit’, ‘validate’, ‘test’, ‘predict’]) – Method to run tuner on. It can be any of ("fit", "validate", "test", "predict").

• mode (str) –

  Search strategy to update the batch size:

  • 'power': Keep multiplying the batch size by 2, until we get an OOM error.

  • 'binsearch': Initially keep multiplying by 2 and after encountering an OOM error

    do a binary search between the last successful batch size and the batch size that failed.

• steps_per_trial (int) – number of steps to run with a given batch size. Ideally 1 should be enough to test if an OOM error occurs, however in practise a few are needed

• init_val (int) – initial batch size to start the search with

• max_trials (int) – max number of increases in batch size done before algorithm is terminated

• batch_arg_name (str) –

  name of the attribute that stores the batch size. It is expected that the user has provided a model or datamodule that has a hyperparameter with that name. We will look for this attribute name in the following places

  • model

  • model.hparams

  • trainer.datamodule (the datamodule passed to the tune method)

Return type

Optional[int]