• Docs >
  • Eliminate config boilerplate (Advanced)
Shortcuts

Eliminate config boilerplate (Advanced)

Customize arguments by command

To customize arguments by subcommand, pass the config before the subcommand:

$ python main.py [before] [subcommand] [after]
$ python main.py  ...         fit       ...

For example, here we set the Trainer argument [max_steps = 100] for the full training routine and [max_steps = 10] for testing:

# config1.yaml
fit:
    trainer:
        max_steps: 100
test:
    trainer:
        max_epochs: 10

now you can toggle this behavior by subcommand:

# full routine with max_steps = 100
$ python main.py --config config1.yaml fit

# test only with max_epochs = 10
$ python main.py --config config1.yaml test

Use groups of options

Groups of options can also be given as independent config files:

$ python trainer.py fit --trainer trainer.yaml --model model.yaml --data data.yaml [...]

Run from cloud yaml configs

For certain enterprise workloads, Lightning CLI supports running from hosted configs:

$ python trainer.py [subcommand] --config s3://bucket/config.yaml

For more options, refer to Remote filesystems.


Use a config via environment variables

For certain CI/CD systems, it’s useful to pass in config files as environment variables:

$ python trainer.py fit --trainer "$TRAINER_CONFIG" --model "$MODEL_CONFIG" [...]

Run from environment variables directly

The Lightning CLI can convert every possible CLI flag into an environment variable. To enable this, set the env_parse argument:

LightningCLI(env_parse=True)

now use the --help CLI flag with any subcommand:

$ python main.py fit --help

which will show you ALL possible environment variables you can now set:

usage: main.py [options] fit [-h] [-c CONFIG]
                            [--trainer.max_epochs MAX_EPOCHS] [--trainer.min_epochs MIN_EPOCHS]
                            [--trainer.max_steps MAX_STEPS] [--trainer.min_steps MIN_STEPS]
                            ...
                            [--ckpt_path CKPT_PATH]

optional arguments:
...
--model CONFIG        Path to a configuration file.
--model.out_dim OUT_DIM
                        (type: int, default: 10)
--model.learning_rate LEARNING_RATE
                        (type: float, default: 0.02)

now you can customize the behavior via environment variables:

# set the options via env vars
$ export LEARNING_RATE=0.01
$ export OUT_DIM=5

$ python main.py fit

Set default config files

To set a path to a config file of defaults, use the default_config_files argument:

cli = LightningCLI(MyModel, MyDataModule, parser_kwargs={"default_config_files": ["my_cli_defaults.yaml"]})

or if you want defaults per subcommand:

cli = LightningCLI(MyModel, MyDataModule, parser_kwargs={"fit": {"default_config_files": ["my_fit_defaults.yaml"]}})

For more configuration options, refer to the ArgumentParser API documentation.


Enable variable interpolation

In certain cases where multiple configs need to share variables, consider using variable interpolation. Variable interpolation allows you to add variables to your yaml configs like so:

model:
  encoder_layers: 12
  decoder_layers:
  - ${model.encoder_layers}
  - 4

To enable variable interpolation, first install omegaconf:

pip install omegaconf

Then set omegaconf when instantiating the LightningCLI class:

cli = LightningCLI(MyModel, parser_kwargs={"parser_mode": "omegaconf"})

After this, the CLI will automatically perform interpolation in yaml files:

python main.py --model.encoder_layers=12

© Copyright Copyright (c) 2018-2023, Lightning AI et al...

Built with Sphinx using a theme provided by Read the Docs.