Problem LightningCLI with default_config_files

implementation help
1

Hello everyone,

I have a problem with using a subcommand together with “default_config_files” and “class_path” inside a config file as presented in the docs.

In the following code the first example for the LightningCLI object using “default_config_files” does not work with the given config file whereas the second one works by slightly changing the object and the config file.
However, according to the docs, both should work. What am I missing here?

Cheers,
Fabian

# main.py
from lightning.pytorch.cli import LightningCLI
from lightning.pytorch.demos.boring_classes import BoringDataModule


def cli_main():
    cli = LightningCLI(
        datamodule_class=BoringDataModule,
        parser_kwargs={
            "default_config_files": ["config1.yaml"],
        },
    )
    # cli = LightningCLI(
    #     datamodule_class=BoringDataModule,
    #     parser_kwargs={
    #         "fit": {"default_config_files": ["config2.yaml"]},
    #     },
    # )


if __name__ == "__main__":
    cli_main()

# config1.yaml
fit:
  trainer:
    max_epochs: 1
  model:
    class_path: lightning.pytorch.demos.boring_classes.DemoModel
    init_args:
      out_dim: 10
      learning_rate: 0.02

# config2.yaml
trainer:
  max_epochs: 1
model:
  class_path: lightning.pytorch.demos.boring_classes.DemoModel
  init_args:
    out_dim: 10
    learning_rate: 0.02
2

The code and configs work fine. The issue seems to be a confusion, see Mixing the order of `--config` and `fit` in LightningCLI can cause confusion · Issue #19714 · Lightning-AI/pytorch-lightning · GitHub.

That is, a config can be given without a subcommand like main.py --config=config1.yaml in which case the subcommand must be inside the config, or the config given after the subcommand like main.py fit --config=config2.yaml.

The difference between "default_config_files": [...] and "fit": {"default_config_files": [...]} is that in the former the parser_kwargs gets applied to all subcommands fit, validate, test and predict, and the latter it only gets applied to fit.

3

First, thank you for your detailed answer.

However, I think it does not really fit my questions, and I am not sure if I have made points clear previously.

Does the code and configs work as they are right now, as you mentioned? In my environment, they do not, unless I remove the “parser_kwargs” argument in the LightningCLI object. Can you please provide some details?

I know that you can run these commands with the subcommand included or not and adapt the config.yaml accordingly if you provide it in the command.
However, that is not what I wanted to do originally. I want to run in the command itself without using the --config flag (using the “default_config_files” and let everything handled by the config.yaml itself), rather than run the command with the --config flag.
What I intend to achieve is having one config.yaml with different parameters for fit and test stage, for example, and utilize the comfort of “default_config_files” combined with the corresponding subcommand.

4

Okay, then it does seem to be a bug. I would need to look at it when I have the time.

6

I think this should be fixed with the latest jsonargparse release.