Go pass the sanity check but get CUDA OUT OF MEMORY when in validation loop

LightningModule
1

Hi, when I run the train code.

It pass the sanity check and use about 15GB/24GB memory.

But when the code went to validation loop, I got CUDA OUT OF MEMORY error (it was fine in train loop. my validation input images were larger than training, but they passed the sanity check), that’s strange.

Can anyone explain this? @awaelchli @SkafteNicki @jirka