{"cells": [{"cell_type": "markdown", "id": "8ecd5f2d", "metadata": {"papermill": {"duration": 0.013104, "end_time": "2023-03-15T09:54:13.367259", "exception": false, "start_time": "2023-03-15T09:54:13.354155", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 2: Activation Functions\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2023-03-15T09:52:39.179933\n", "\n", "In this tutorial, we will take a closer look at (popular) activation functions and investigate their effect on optimization properties in neural networks.\n", "Activation functions are a crucial part of deep learning models as they add the non-linearity to neural networks.\n", "There is a great variety of activation functions in the literature, and some are more beneficial than others.\n", "The goal of this tutorial is to show the importance of choosing a good activation function (and how to do so), and what problems might occur if we don't.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/02-activation-functions.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/Lightning-AI/lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)"]}, {"cell_type": "markdown", "id": "3e50a070", "metadata": {"papermill": {"duration": 0.011958, "end_time": "2023-03-15T09:54:13.388750", "exception": false, "start_time": "2023-03-15T09:54:13.376792", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "b5fc1714", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2023-03-15T09:54:13.433230Z", "iopub.status.busy": "2023-03-15T09:54:13.432760Z", "iopub.status.idle": "2023-03-15T09:54:16.770682Z", "shell.execute_reply": "2023-03-15T09:54:16.769697Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 3.348973, "end_time": "2023-03-15T09:54:16.773391", "exception": false, "start_time": "2023-03-15T09:54:13.424418", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\r\n", "\u001b[0m"]}], "source": ["! pip install --quiet \"matplotlib\" \"torchvision\" \"torchmetrics>=0.7, <0.12\" \"setuptools==67.4.0\" \"seaborn\" \"lightning>=2.0.0rc0\" \"pytorch-lightning>=1.4, <2.0.0\" \"torch>=1.8.1, <1.14.0\" \"ipython[notebook]>=8.0.0, <8.12.0\""]}, {"cell_type": "markdown", "id": "0349ac26", "metadata": {"papermill": {"duration": 0.006921, "end_time": "2023-03-15T09:54:16.795535", "exception": false, "start_time": "2023-03-15T09:54:16.788614", "status": "completed"}, "tags": []}, "source": ["