Unit 6.4 – Choosing Activation Functions

References

• Dubey, Singh, Chaudhuri (2022). Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark.

• ReLU PyTorch docs

• GELU PyTorch docs

• Swish / SiLU docs

• Mish PyTorch docs

  Note to Olya: the capitalization above is intentional. No typos there.

What we covered in this video lecture

In Unit 4, we learned that non-linear activation functions are essential elements of a multi-layer neural network. In this lecture, we expand our repertoire of non-linear activation functions, including ReLU, GELU, Swish, and Mish activations.

Does it matter which one you choose? Yes, sometimes it can matter. The choice of the activation function may impact the predictive performance, training time, and stability of your deep learning model. Different activation functions may work better for specific tasks and model architectures. The best way to determine which activation function to use is through experimentation. Try different activation functions and evaluate their performance on your specific problem to find the one that best suits our needs.

Additional resources if you want to learn more

If you are interested in reading more about the performance of the different activation functions, you might like to read the A Comprehensive Survey and Performance Analysis of Activation Functions in Deep Learning article.

Watch Video 1 Mark complete and go to Unit 6.5 →