I put together a minimal example on this repository: GitHub - azouaoui-cv/PL-Hydra-template: PyTorch Lightning + Hydra template to use DDP that reproduces the aforementioned issue whenever I’m using 2+ GPUs in DDP mode and not setting the hydra.run.dir manually.
I will take a closer look on your custom logging callback @shreeyak.
EDIT: Note that I updated the repository linked above into an actual Minimal Working Example such that it no longer uses actual but dummy data instead.
It reproduces the issue whether I use the latest versions (requirements_latest.txt) or my own environment (requirements.txt).