Unit 9.5 Increasing Batch Sizes to Increase Throughput

References

• Measuring the Effects of Data Parallelism on Neural Network Training, https://arxiv.org/abs/1811.03600

• Group Normalization, https://arxiv.org/abs/1803.08494

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, https://arxiv.org/abs/1706.02677

• Measuring the Effects of Data Parallelism on Neural Network Training, https://arxiv.org/abs/1811.03600

• Group Normalization, https://arxiv.org/abs/1803.08494

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, https://arxiv.org/abs/1706.02677

• Measuring the Effects of Data Parallelism on Neural Network Training, https://arxiv.org/abs/1811.03600

• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima, https://arxiv.org/abs/1609.04836

Code

• Part 2: Code Demo, 9.5-batchsize-finder/

What we covered in this video lecture

In this lecture, we discussed the topic of increasing batch sizes to boost throughput in machine learning model training. The batch size, or the number of training samples processed before the model is updated, plays a critical role in the efficiency and effectiveness of model training. By increasing the batch size, we can process more data simultaneously, leading to higher computational efficiency and increased throughput, particularly on hardware like GPUs which excel in parallel processing.

However, in practice, throughput is not always everything, and we have to make sure to strike a careful balance between batch size, learning rate, computational resources, and the potential impact on model performance, which are all crucial considerations in machine learning training pipelines.

Additional resources if you want to learn more

I highly recommend checking out the various papers referenced in the lecture and in the reference section above if you want to learn more about the impact of batch sizes on the computational and predictive performance.

Watch Video 1 Mark complete and go to Unit 9 Exercises →