Unit 9.3 Deep Dive Into Data Parallelism

References

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, https://arxiv.org/abs/1706.02677

Code

• Part 3: Multi-GPU Hands-On Code Demo, 9.3-multi-gpu/

What we covered in this video lecture

In today’s lecture, we focused on the concept of data parallelism and its extension, distributed data parallelism, both essential strategies for accelerating machine learning training using multiple computational resources.

Data parallelism is a technique where the training data is divided into multiple subsets, and each subset is processed independently across multiple GPUs or computing nodes. This allows for simultaneous computation, significantly reducing the training time. The computed gradients from each subset are then aggregated to update the model parameters. However, in the context of a single machine with multiple GPUs, this method can be limited by the inter-GPU communication speed and the machine’s memory capacity.

To overcome the limitations of regular data parallelism in PyTorch, we discussed distributed data parallelism, an extension of data parallelism that spans across multiple machines, each with one or more GPUs. In distributed data parallelism, the same model is replicated on each machine, and every machine processes a different subset of the training data. This not only facilitates handling larger datasets and models but also improves the training speed, taking advantage of the collective memory and computational power of multiple machines.

Additional resources if you want to learn more

There are several guides on the PyTorch Lightning documentation website that I highly recommend reading for more advanced usages:

• GPU Training (Basic)
• GPU Training (Intermediate)
• GPU Training (Advanced)
• GPU Trainining FAQ

Watch Video 1 Mark complete and go to Unit 9.4 →