{"cells": [{"cell_type": "markdown", "id": "ed3501af", "metadata": {"papermill": {"duration": 0.023462, "end_time": "2021-09-16T12:33:15.496875", "exception": false, "start_time": "2021-09-16T12:33:15.473413", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 2: Activation Functions\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2021-09-16T14:32:18.973374\n", "\n", "In this tutorial, we will take a closer look at (popular) activation functions and investigate their effect on optimization properties in neural networks.\n", "Activation functions are a crucial part of deep learning models as they add the non-linearity to neural networks.\n", "There is a great variety of activation functions in the literature, and some are more beneficial than others.\n", "The goal of this tutorial is to show the importance of choosing a good activation function (and how to do so), and what problems might occur if we don't.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/02-activation-functions.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/latest/)\n", "| Join us [on Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)"]}, {"cell_type": "markdown", "id": "66b84901", "metadata": {"papermill": {"duration": 0.021804, "end_time": "2021-09-16T12:33:15.540678", "exception": false, "start_time": "2021-09-16T12:33:15.518874", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "4d7c8160", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2021-09-16T12:33:15.587749Z", "iopub.status.busy": "2021-09-16T12:33:15.587280Z", "iopub.status.idle": "2021-09-16T12:33:15.589355Z", "shell.execute_reply": "2021-09-16T12:33:15.589747Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 0.027383, "end_time": "2021-09-16T12:33:15.589924", "exception": false, "start_time": "2021-09-16T12:33:15.562541", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# ! pip install --quiet \"torchmetrics>=0.3\" \"torch>=1.6, <1.9\" \"pytorch-lightning>=1.3\" \"torchvision\" \"seaborn\" \"matplotlib\""]}, {"cell_type": "markdown", "id": "94e4c637", "metadata": {"papermill": {"duration": 0.021895, "end_time": "2021-09-16T12:33:15.635408", "exception": false, "start_time": "2021-09-16T12:33:15.613513", "status": "completed"}, "tags": []}, "source": ["