If I use the code from the documentation (https://pytorch-lightning.readthedocs.io/en/stable/logging.html#logging):
tb_logger = pl_loggers.TensorBoardLogger('logs/')
csv_logger = pl_loggers.CSVLogger(save_dir='logs/')
trainer = Trainer(logger=[tb_logger, csv_logger])
and enable ddp in the Trainer distributed_backend='ddp', I get a folder structure like this:
logs
--default
----version_1
----version_2
with an version_x folder for each ddp process. Thats quite confusing and does not make any sense (to me)
Is there a way to set a path for the entire trainer once and have the loggers and checkpoints to that path or subdirs of that path?
I tried to have a fixed version with version=str(datetime.datetime.now()) but this does not work as the other DDP processes start at slightly different times.