6.8 Debugging Deep Neural Networks

Code

• Part 1, 2, and 3: 6.8-debugging/

What we covered in this video lecture

This lecture covered three simple approaches for debugging deep neural network training. First, we discussed that doing a fast dev run is often a good idea before initiating an expensive training procedure. This helps us to test whether everything is set up correctly quickly.

Next, we discussed looking at model summaries to better understand whether the layers are connected as we intended. It can also give us useful information on the number of parameters and model sizes at a glance.

Lastly, we discussed batch overfitting. Neural networks are great overfitters if we let them. Or in other words, a neural network should always reach 90-100% accuracy when we train it on a single batch. This is a quick and easy diagnostic for determining whether we implemented everything correctly before moving on to the more expensive training procedure on the full dataset.

Additional resources if you want to learn more

You might enjoy this article by on Debugging in PyTorch where the author mentions common mistakes such as messing up the loss function choice or embedding dimensions among others.

Watch Video 1 Mark complete and go to Unit 6 Exercises →