Unit 9.1 Accelerated Model Training via Mixed-Precision Training

References

• What Every User Should Know About Mixed Precision Training in PyTorch

Code

• Part 2: Hands-On Code Demo, 9.1-mixed-precision

What we covered in this video lecture

In this lecture, we delve into the concept of mixed-precision training in deep learning, which involves using a combination of different numerical precisions (typically float32 and float16 or bfloat16) during model training to improve computational efficiency and speed.

Traditional training methods tend to use 32-bit floating-point numbers (float32) to represent weights, biases, activations, and gradients for neural networks. However, this can be computationally expensive and memory-intensive, particularly for large models and data sets. To address this, mixed-precision training employs lower-precision formats, namely 16-bit floating-point numbers (float16) and Brain Floating Point (bfloat16), in parts of the training process where higher precision is not critical.

The balance between speed, memory usage, and precision makes mixed-precision training an increasingly popular approach for training large-scale machine learning models.

Additional resources if you want to learn more

If you want to learn more details about mixed precision training, including benchmarks, check out my blog article Accelerating Large Language Models with Mixed-Precision Techniques.

Furthermore, you might be interested in the related topic of quantization. An interesting research article on this topic is LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale.

Watch Video 1 Mark complete and go to Unit 9.2 →