6.8 Debugging Deep Neural Networks
- Part 1, 2, and 3: 6.8-debugging/
What we covered in this video lecture
This lecture covered three simple approaches for debugging deep neural network training. First, we discussed that doing a fast dev run is often a good idea before initiating an expensive training procedure. This helps us to test whether everything is set up correctly quickly.
Next, we discussed looking at model summaries to better understand whether the layers are connected as we intended. It can also give us useful information on the number of parameters and model sizes at a glance.
Lastly, we discussed batch overfitting. Neural networks are great overfitters if we let them. Or in other words, a neural network should always reach 90-100% accuracy when we train it on a single batch. This is a quick and easy diagnostic for determining whether we implemented everything correctly before moving on to the more expensive training procedure on the full dataset.
Additional resources if you want to learn more
You might enjoy this article by on Debugging in PyTorch where the author mentions common mistakes such as messing up the loss function choice or embedding dimensions among others.
Log in or create a free Lightning.ai account to access:
- Completion badges
- Progress tracking
- Additional downloadable content
- Additional AI education resources
- Notifications when new units are released
- Free cloud computing credits