I have a model, that uses gradient checkpointing and ddp. It works fine, when I train it on a single gpu. It also works fine if I turn off checkpointing. However with multiple GPUs loss initially looks innocent, but then suddenly becomes NaN: checkpointing no checkpointing gpus = 1 works w…

Gradient checkpointing + ddp = NaN

jw3126 November 20, 2020, 1:57pm 4

show post in topic

Powered by Discourse, best viewed with JavaScript enabled