{ "cells": [ { "cell_type": "markdown", "id": "7e3b7339", "metadata": { "papermill": { "duration": 0.011104, "end_time": "2023-10-11T15:29:11.954277", "exception": false, "start_time": "2023-10-11T15:29:11.943173", "status": "completed" }, "tags": [] }, "source": [ "\n", "# Tutorial 2: Activation Functions\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2023-10-11T15:26:46.260596\n", "\n", "In this tutorial, we will take a closer look at (popular) activation functions and investigate their effect on optimization properties in neural networks.\n", "Activation functions are a crucial part of deep learning models as they add the non-linearity to neural networks.\n", "There is a great variety of activation functions in the literature, and some are more beneficial than others.\n", "The goal of this tutorial is to show the importance of choosing a good activation function (and how to do so), and what problems might occur if we don't.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/02-activation-functions.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/Lightning-AI/lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)" ] }, { "cell_type": "markdown", "id": "a5b63cba", "metadata": { "papermill": { "duration": 0.010013, "end_time": "2023-10-11T15:29:11.975526", "exception": false, "start_time": "2023-10-11T15:29:11.965513", "status": "completed" }, "tags": [] }, "source": [ "## Setup\n", "This notebook requires some packages besides pytorch-lightning." ] }, { "cell_type": "code", "execution_count": 1, "id": "756ff121", "metadata": { "colab": {}, "colab_type": "code", "execution": { "iopub.execute_input": "2023-10-11T15:29:11.996484Z", "iopub.status.busy": "2023-10-11T15:29:11.996249Z", "iopub.status.idle": "2023-10-11T15:34:16.729212Z", "shell.execute_reply": "2023-10-11T15:34:16.727829Z" }, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": { "duration": 304.746763, "end_time": "2023-10-11T15:34:16.732085", "exception": false, "start_time": "2023-10-11T15:29:11.985322", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\r\n", "\u001b[0m" ] } ], "source": [ "! pip install --quiet \"matplotlib\" \"setuptools>=68.0.0, <68.3.0\" \"torch>=1.8.1, <2.1.0\" \"torchmetrics>=0.7, <1.3\" \"seaborn\" \"lightning>=2.0.0\" \"torchvision\" \"matplotlib>=3.0.0, <3.9.0\" \"pytorch-lightning>=1.4, <2.1.0\" \"urllib3\" \"ipython[notebook]>=8.0.0, <8.17.0\"" ] }, { "cell_type": "markdown", "id": "5ff9ab7e", "metadata": { "papermill": { "duration": 0.016777, "end_time": "2023-10-11T15:34:16.766464", "exception": false, "start_time": "2023-10-11T15:34:16.749687", "status": "completed" }, "tags": [] }, "source": [ "