• Docs >
  • Eliminate config boilerplate (expert)
Shortcuts

Eliminate config boilerplate (expert)

Troubleshooting

The standard behavior for CLIs, when they fail, is to terminate the process with a non-zero exit code and a short message to hint the user about the cause. This is problematic while developing the CLI since there is no information to track down the root of the problem. A simple change in the instantiation of the LightningCLI can be used such that when there is a failure an exception is raised and the full stack trace printed.

cli = LightningCLI(MyModel, parser_kwargs={"error_handler": None})

Note

When asking about problems and reporting issues please set the error_handler to None and include the stack trace in your description. With this, it is more likely for people to help out identifying the cause without needing to create a reproducible script.


Reproducibility with the LightningCLI

The topic of reproducibility is complex and it is impossible to guarantee reproducibility by just providing a class that people can use in unexpected ways. Nevertheless, the LightningCLI tries to give a framework and recommendations to make reproducibility simpler.

When an experiment is run, it is good practice to use a stable version of the source code, either being a released package or at least a commit of some version controlled repository. For each run of a CLI the config file is automatically saved including all settings. This is useful to figure out what was done for a particular run without requiring to look at the source code. If by mistake the exact version of the source code is lost or some defaults changed, having the full config means that most of the information is preserved.

The class is targeted at implementing CLIs because running a command from a shell provides a separation with the Python source code. Ideally the CLI would be placed in your path as part of the installation of a stable package, instead of running from a clone of a repository that could have uncommitted local modifications. Creating installable packages that include CLIs is out of the scope of this document. This is mentioned only as a teaser for people who would strive for the best practices possible.

For every CLI implemented, users are encouraged to learn how to run it by reading the documentation printed with the --help option and use the --print_config option to guide the writing of config files. A few more details that might not be clear by only reading the help are the following.

LightningCLI is based on argparse and as such follows the same arguments style as many POSIX command line tools. Long options are prefixed with two dashes and its corresponding values should be provided with an empty space or an equal sign, as --option value or --option=value. Command line options are parsed from left to right, therefore if a setting appears multiple times the value most to the right will override the previous ones. If a class has an init parameter that is required (i.e. no default value), it is given as --option which makes it explicit and more readable instead of relying on positional arguments.


What is a subcommand?

A subcommand is what is the action the LightningCLI applies to the script:

python main.py [subcommand]

See the Potential subcommands with:

python main.py --help

which prints:

...

fit                 Runs the full optimization routine.
validate            Perform one evaluation epoch over the validation set.
test                Perform one evaluation epoch over the test set.
predict             Run inference on your data.
tune                Runs routines to tune hyperparameters before training.

use a subcommand as follows:

python main.py fit
python main.py test

What is the CLI?

CLI is short for commandline interface. Use your terminal to enter these commands.


© Copyright Copyright (c) 2018-2023, Lightning AI et al...

Built with Sphinx using a theme provided by Read the Docs.