The model should be initialized in __init__ and it will be moved to all the devices automatically in a distributed environment. setup should be used to initialize anything else that is required in every sub-process in DDP mode. You can also alter the model in setup too if you want to dynamically change or alter it somehow on each sub-process. If you initialize anything else in __init__ it will be initialized on the main process and won’t be able to access it on the sub-process in case of DDP.