Effective learning rate and batch size with Lightning in DDP

[teddy]

@sm000

The given learning rate for DDP (unlike DP) should correspond to the per-GPU (given) batch size if I understand correctly now. So why would half the given learning rate mean the same effective learning rate? To keep things the same, should we keep the same learning rate?

I believe they should in fact keep the same learning rate in this case, since the per-GPU batch size is the same. They may have overlooked this, but as I mentioned before a 2x change in learning rate will probably not change much.

@sm000

Why would one want to scale the batch size given to DDP (like the ImageNet example/Moco do) by the number of GPUs?

The only reason I can think to scale the batch size is to be consistent with a paper. If the paper reports a batch size of 256 but you can only use 32, it may appear cleaner to still say you have a batch size of 256, but over 8 gpus.

@sm000

My understanding is Lightning chooses not to scale for the user since the other hyperparameters (learning rate in particular) mostly vary with the per-GPU/given batch size and not the effective batch size.

Exactly. In this way you can scale up without having to change any hyperparameters.