Since my dataset is too large for memory, I have to split it into multiple subsets like:
–sub_0
–0001.h5
–0002.h5
…
–sub_1
–sub_2
For every epoch, I want to choose one dir(‘sub_x’) and use it to create a dataset during training. Could anyone tell me how to implement it in lightning, especially in DDP setting？