Eliminate config boilerplate (Advanced)¶

Customize arguments by command¶

To customize arguments by subcommand, pass the config before the subcommand:

$ python main.py [before] [subcommand] [after]
$ python main.py  ...         fit       ...

For example, here we set the Trainer argument [max_steps = 100] for the full training routine and [max_steps = 10] for testing:

# config1.yaml
fit:
    trainer:
        max_steps: 100
test:
    trainer:
        max_epochs: 10

now you can toggle this behavior by subcommand:

# full routine with max_steps = 100
$ python main.py --config config1.yaml fit

# test only with max_epochs = 10
$ python main.py --config config1.yaml test

Use groups of options¶

Groups of options can also be given as independent config files:

$ python trainer.py fit --trainer trainer.yaml --model model.yaml --data data.yaml [...]

Run from cloud yaml configs¶

For certain enterprise workloads, Lightning CLI supports running from hosted configs:

$ python trainer.py [subcommand] --config s3://bucket/config.yaml

For more options, refer to Remote filesystems.

Use a config via environment variables¶

For certain CI/CD systems, it’s useful to pass in config files as environment variables:

$ python trainer.py fit --trainer "$TRAINER_CONFIG" --model "$MODEL_CONFIG" [...]

Run from environment variables directly¶

The Lightning CLI can convert every possible CLI flag into an environment variable. To enable this, set the env_parse argument:

LightningCLI(env_parse=True)

now use the --help CLI flag with any subcommand:

$ python main.py fit --help

which will show you ALL possible environment variables you can now set:

usage: main.py [options] fit [-h] [-c CONFIG]
                            [--trainer.max_epochs MAX_EPOCHS] [--trainer.min_epochs MIN_EPOCHS]
                            [--trainer.max_steps MAX_STEPS] [--trainer.min_steps MIN_STEPS]
                            ...
                            [--ckpt_path CKPT_PATH]

optional arguments:
...
--model CONFIG        Path to a configuration file.
--model.out_dim OUT_DIM
                        (type: int, default: 10)
--model.learning_rate LEARNING_RATE
                        (type: float, default: 0.02)

now you can customize the behavior via environment variables:

# set the options via env vars
$ export LEARNING_RATE=0.01
$ export OUT_DIM=5

$ python main.py fit

Set default config files¶

To set a path to a config file of defaults, use the default_config_files argument:

cli = LightningCLI(MyModel, MyDataModule, parser_kwargs={"default_config_files": ["my_cli_defaults.yaml"]})

or if you want defaults per subcommand:

cli = LightningCLI(MyModel, MyDataModule, parser_kwargs={"fit": {"default_config_files": ["my_fit_defaults.yaml"]}})

For more configuration options, refer to the ArgumentParser API documentation.

Enable variable interpolation¶

In certain cases where multiple configs need to share variables, consider using variable interpolation. Variable interpolation allows you to add variables to your yaml configs like so:

model:
  encoder_layers: 12
  decoder_layers:
  - ${model.encoder_layers}
  - 4

To enable variable interpolation, first install omegaconf:

pip install omegaconf

Then set omegaconf when instantiating the LightningCLI class:

cli = LightningCLI(MyModel, parser_kwargs={"parser_mode": "omegaconf"})

After this, the CLI will automatically perform interpolation in yaml files:

python main.py --model.encoder_layers=12