I have trained a model in 2 node * 2gpus in deepspeed, and saved the checkpoint. But we also want to continue training in 4 node * 4 gpus, what should i do ?
I try this link : Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported. · Issue #3810 · microsoft/DeepSpeed · GitHub
zero_to_fp32.py extracts fp32 consolidated weights from a zero 2 and 3 DeepSpeed checkpoints.
Some epoch and global setp saved in checkpoint. What should do next? Thank you very much.