{"cells": [{"cell_type": "markdown", "id": "876fdcae", "metadata": {"papermill": {"duration": 0.010966, "end_time": "2024-09-01T11:55:58.776082", "exception": false, "start_time": "2024-09-01T11:55:58.765116", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 2: Activation Functions\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2024-09-01T11:55:51.589497\n", "\n", "In this tutorial, we will take a closer look at (popular) activation functions and investigate their effect on optimization properties in neural networks.\n", "Activation functions are a crucial part of deep learning models as they add the non-linearity to neural networks.\n", "There is a great variety of activation functions in the literature, and some are more beneficial than others.\n", "The goal of this tutorial is to show the importance of choosing a good activation function (and how to do so), and what problems might occur if we don't.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/02-activation-functions.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/Lightning-AI/lightning/)\n", "| Check out [the documentation](https://lightning.ai/docs/)\n", "| Join us [on Discord](https://discord.com/invite/tfXFetEZxv)"]}, {"cell_type": "markdown", "id": "7172502b", "metadata": {"papermill": {"duration": 0.010647, "end_time": "2024-09-01T11:55:58.796819", "exception": false, "start_time": "2024-09-01T11:55:58.786172", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "a3cc424b", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2024-09-01T11:55:58.818406Z", "iopub.status.busy": "2024-09-01T11:55:58.817728Z", "iopub.status.idle": "2024-09-01T11:56:00.098083Z", "shell.execute_reply": "2024-09-01T11:56:00.096888Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 1.29407, "end_time": "2024-09-01T11:56:00.100844", "exception": false, "start_time": "2024-09-01T11:55:58.806774", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.\u001b[0m\u001b[33m\r\n", "\u001b[0m"]}], "source": ["! pip install --quiet \"matplotlib\" \"torchvision\" \"numpy <3.0\" \"torchmetrics >=1.0,<1.5\" \"torch >=1.8.1,<2.5\" \"seaborn\" \"pytorch-lightning >=2.0,<2.5\""]}, {"cell_type": "markdown", "id": "870769a9", "metadata": {"papermill": {"duration": 0.016442, "end_time": "2024-09-01T11:56:00.134252", "exception": false, "start_time": "2024-09-01T11:56:00.117810", "status": "completed"}, "tags": []}, "source": ["