{"cells": [{"cell_type": "markdown", "id": "cbcf0f41", "metadata": {"papermill": {"duration": 0.015123, "end_time": "2022-05-03T01:17:51.912530", "exception": false, "start_time": "2022-05-03T01:17:51.897407", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 13: Self-Supervised Contrastive Learning with SimCLR\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2022-05-03T02:43:21.313398\n", "\n", "In this tutorial, we will take a closer look at self-supervised contrastive learning.\n", "Self-supervised learning, or also sometimes called unsupervised learning, describes the scenario where we have given input data, but no accompanying labels to train in a classical supervised way.\n", "However, this data still contains a lot of information from which we can learn: how are the images different from each other?\n", "What patterns are descriptive for certain images?\n", "Can we cluster the images?\n", "To get an insight into these questions, we will implement a popular, simple contrastive learning method, SimCLR, and apply it to the STL10 dataset.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/13-contrastive-learning.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)"]}, {"cell_type": "markdown", "id": "8dfc821d", "metadata": {"papermill": {"duration": 0.011721, "end_time": "2022-05-03T01:17:51.936328", "exception": false, "start_time": "2022-05-03T01:17:51.924607", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "00a149ee", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2022-05-03T01:17:51.962905Z", "iopub.status.busy": "2022-05-03T01:17:51.962281Z", "iopub.status.idle": "2022-05-03T01:17:55.701176Z", "shell.execute_reply": "2022-05-03T01:17:55.700261Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 3.755451, "end_time": "2022-05-03T01:17:55.703535", "exception": false, "start_time": "2022-05-03T01:17:51.948084", "status": "completed"}, "tags": []}, "outputs": [], "source": ["! pip install --quiet \"torch>=1.8\" \"torchmetrics>=0.7\" \"ipython[notebook]\" \"pytorch-lightning>=1.4\" \"torchvision\" \"setuptools==59.5.0\" \"seaborn\" \"matplotlib\""]}, {"cell_type": "markdown", "id": "472e0e40", "metadata": {"papermill": {"duration": 0.011868, "end_time": "2022-05-03T01:17:55.728779", "exception": false, "start_time": "2022-05-03T01:17:55.716911", "status": "completed"}, "tags": []}, "source": ["\n", "Methods for self-supervised learning try to learn as much as possible from the data alone, so it can quickly be finetuned for a specific classification task.\n", "The benefit of self-supervised learning is that a large dataset can often easily be obtained.\n", "For instance, if we want to train a vision model on semantic segmentation for autonomous driving, we can collect large amounts of data by simply installing a camera in a car, and driving through a city for an hour.\n", "In contrast, if we would want to do supervised learning, we would have to manually label all those images before training a model.\n", "This is extremely expensive, and would likely take a couple of months to manually label the same amount of data.\n", "Further, self-supervised learning can provide an alternative to transfer learning from models pretrained on ImageNet since we could pretrain a model on a specific dataset/situation, e.g. traffic scenarios for autonomous driving.\n", "\n", "Within the last two years, a lot of new approaches have been proposed for self-supervised learning, in particular for images, that have resulted in great improvements over supervised models when few labels are available.\n", "The subfield that we will focus on in this tutorial is contrastive learning.\n", "Contrastive learning is motivated by the question mentioned above: how are images different from each other?\n", "Specifically, contrastive learning methods train a model to cluster an image and its slightly augmented version in latent space, while the distance to other images should be maximized.\n", "A very recent and simple method for this is [SimCLR](https://arxiv.org/abs/2006.10029), which is visualized below (figure credit - [Ting Chen et al. ](https://simclr.github.io/)).\n", "\n", " {width=\"500px\"} \n", "\n", "The general setup is that we are given a dataset of images without any labels, and want to train a model on this data such that it can quickly adapt to any image recognition task afterward.\n", "During each training iteration, we sample a batch of images as usual.\n", "For each image, we create two versions by applying data augmentation techniques like cropping, Gaussian noise, blurring, etc.\n", "An example of such is shown on the left with the image of the dog.\n", "We will go into the details and effects of the chosen augmentation techniques later.\n", "On those images, we apply a CNN like ResNet and obtain as output a 1D feature vector on which we apply a small MLP.\n", "The output features of the two augmented images are then trained to be close to each other, while all other images in that batch should be as different as possible.\n", "This way, the model has to learn to recognize the content of the image that remains unchanged under the data augmentations, such as objects which we usually care about in supervised tasks.\n", "\n", "We will now implement this framework ourselves and discuss further details along the way.\n", "Let's first start with importing our standard libraries below:"]}, {"cell_type": "code", "execution_count": 2, "id": "944f6485", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:17:55.754645Z", "iopub.status.busy": "2022-05-03T01:17:55.754325Z", "iopub.status.idle": "2022-05-03T01:17:58.993701Z", "shell.execute_reply": "2022-05-03T01:17:58.993022Z"}, "papermill": {"duration": 3.254732, "end_time": "2022-05-03T01:17:58.995369", "exception": false, "start_time": "2022-05-03T01:17:55.740637", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/usr/lib/python3.8/site-packages/apex/pyprof/__init__.py:5: FutureWarning: pyprof will be removed by the end of June, 2022\n", " warnings.warn(\"pyprof will be removed by the end of June, 2022\", FutureWarning)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["WARNING:root:Bagua cannot detect bundled NCCL library, Bagua will try to use system NCCL instead. If you encounter any error, please run `import bagua_core; bagua_core.install_deps()` or the `bagua_install_deps.py` script to install bundled libraries.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["/tmp/ipykernel_4616/1782126012.py:24: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n", " set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "Global seed set to 42\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Device: cuda:0\n", "Number of workers: 12\n"]}], "source": ["import os\n", "import urllib.request\n", "from copy import deepcopy\n", "from urllib.error import HTTPError\n", "\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "import pytorch_lightning as pl\n", "import seaborn as sns\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "import torch.utils.data as data\n", "import torchvision\n", "from IPython.display import set_matplotlib_formats\n", "from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint\n", "from torchvision import transforms\n", "from torchvision.datasets import STL10\n", "from tqdm.notebook import tqdm\n", "\n", "plt.set_cmap(\"cividis\")\n", "%matplotlib inline\n", "set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "matplotlib.rcParams[\"lines.linewidth\"] = 2.0\n", "sns.set()\n", "\n", "# Import tensorboard\n", "%load_ext tensorboard\n", "\n", "# Path to the folder where the datasets are/should be downloaded (e.g. CIFAR10)\n", "DATASET_PATH = os.environ.get(\"PATH_DATASETS\", \"data/\")\n", "# Path to the folder where the pretrained models are saved\n", "CHECKPOINT_PATH = os.environ.get(\"PATH_CHECKPOINT\", \"saved_models/ContrastiveLearning/\")\n", "# In this notebook, we use data loaders with heavier computational processing. It is recommended to use as many\n", "# workers as possible in a data loader, which corresponds to the number of CPU cores\n", "NUM_WORKERS = os.cpu_count()\n", "\n", "# Setting the seed\n", "pl.seed_everything(42)\n", "\n", "# Ensure that all operations are deterministic on GPU (if used) for reproducibility\n", "torch.backends.cudnn.determinstic = True\n", "torch.backends.cudnn.benchmark = False\n", "\n", "device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device(\"cpu\")\n", "print(\"Device:\", device)\n", "print(\"Number of workers:\", NUM_WORKERS)"]}, {"cell_type": "markdown", "id": "69f46cbe", "metadata": {"papermill": {"duration": 0.011969, "end_time": "2022-05-03T01:17:59.020819", "exception": false, "start_time": "2022-05-03T01:17:59.008850", "status": "completed"}, "tags": []}, "source": ["As in many tutorials before, we provide pre-trained models.\n", "Note that those models are slightly larger as normal (~100MB overall) since we use the default ResNet-18 architecture.\n", "If you are running this notebook locally, make sure to have sufficient disk space available."]}, {"cell_type": "code", "execution_count": 3, "id": "75701a79", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:17:59.046279Z", "iopub.status.busy": "2022-05-03T01:17:59.045797Z", "iopub.status.idle": "2022-05-03T01:18:01.223802Z", "shell.execute_reply": "2022-05-03T01:18:01.223161Z"}, "papermill": {"duration": 2.19263, "end_time": "2022-05-03T01:18:01.225338", "exception": false, "start_time": "2022-05-03T01:17:59.032708", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/SimCLR.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/ResNet.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/tensorboards/SimCLR/events.out.tfevents.SimCLR...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/tensorboards/classification/ResNet/events.out.tfevents.ResNet...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/LogisticRegression_10.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/LogisticRegression_20.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/LogisticRegression_50.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/LogisticRegression_100.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/LogisticRegression_200.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/LogisticRegression_500.ckpt...\n"]}], "source": ["# Github URL where saved models are stored for this tutorial\n", "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial17/\"\n", "# Files to download\n", "pretrained_files = [\n", " \"SimCLR.ckpt\",\n", " \"ResNet.ckpt\",\n", " \"tensorboards/SimCLR/events.out.tfevents.SimCLR\",\n", " \"tensorboards/classification/ResNet/events.out.tfevents.ResNet\",\n", "]\n", "pretrained_files += [f\"LogisticRegression_{size}.ckpt\" for size in [10, 20, 50, 100, 200, 500]]\n", "# Create checkpoint path if it doesn't exist yet\n", "os.makedirs(CHECKPOINT_PATH, exist_ok=True)\n", "\n", "# For each file, check whether it already exists. If not, try downloading it.\n", "for file_name in pretrained_files:\n", " file_path = os.path.join(CHECKPOINT_PATH, file_name)\n", " if \"/\" in file_name:\n", " os.makedirs(file_path.rsplit(\"/\", 1)[0], exist_ok=True)\n", " if not os.path.isfile(file_path):\n", " file_url = base_url + file_name\n", " print(f\"Downloading {file_url}...\")\n", " try:\n", " urllib.request.urlretrieve(file_url, file_path)\n", " except HTTPError as e:\n", " print(\n", " \"Something went wrong. Please try to download the file from the GDrive folder, or contact the author with the full output including the following error:\\n\",\n", " e,\n", " )"]}, {"cell_type": "markdown", "id": "281a5764", "metadata": {"papermill": {"duration": 0.012844, "end_time": "2022-05-03T01:18:01.251854", "exception": false, "start_time": "2022-05-03T01:18:01.239010", "status": "completed"}, "tags": []}, "source": ["## SimCLR\n", "\n", "We will start our exploration of contrastive learning by discussing the effect of different data augmentation techniques, and how we can implement an efficient data loader for such.\n", "Next, we implement SimCLR with PyTorch Lightning, and finally train it on a large, unlabeled dataset."]}, {"cell_type": "markdown", "id": "25dc7369", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.01205, "end_time": "2022-05-03T01:18:01.276160", "exception": false, "start_time": "2022-05-03T01:18:01.264110", "status": "completed"}, "tags": []}, "source": ["### Data Augmentation for Contrastive Learning\n", "\n", "To allow efficient training, we need to prepare the data loading such that we sample two different, random augmentations for each image in the batch.\n", "The easiest way to do this is by creating a transformation that, when being called, applies a set of data augmentations to an image twice.\n", "This is implemented in the class `ContrastiveTransformations` below:"]}, {"cell_type": "code", "execution_count": 4, "id": "587db13b", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:18:01.302101Z", "iopub.status.busy": "2022-05-03T01:18:01.301709Z", "iopub.status.idle": "2022-05-03T01:18:01.306189Z", "shell.execute_reply": "2022-05-03T01:18:01.305542Z"}, "papermill": {"duration": 0.019313, "end_time": "2022-05-03T01:18:01.307666", "exception": false, "start_time": "2022-05-03T01:18:01.288353", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ContrastiveTransformations:\n", " def __init__(self, base_transforms, n_views=2):\n", " self.base_transforms = base_transforms\n", " self.n_views = n_views\n", "\n", " def __call__(self, x):\n", " return [self.base_transforms(x) for i in range(self.n_views)]"]}, {"cell_type": "markdown", "id": "f47cb2a1", "metadata": {"papermill": {"duration": 0.012177, "end_time": "2022-05-03T01:18:01.332086", "exception": false, "start_time": "2022-05-03T01:18:01.319909", "status": "completed"}, "tags": []}, "source": ["The contrastive learning framework can easily be extended to have more _positive_ examples by sampling more than two augmentations of the same image.\n", "However, the most efficient training is usually obtained by using only two.\n", "\n", "Next, we can look at the specific augmentations we want to apply.\n", "The choice of the data augmentation to use is the most crucial hyperparameter in SimCLR since it directly affects how the latent space is structured, and what patterns might be learned from the data.\n", "Let's first take a look at some of the most popular data augmentations (figure credit - [Ting Chen and Geoffrey Hinton](https://ai.googleblog.com/2020/04/advancing-self-supervised-and-semi.html)):\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-03T03:19:59.845213 \n", " image/svg+xml \n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/ \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["# Visualize some examples\n", "pl.seed_everything(42)\n", "NUM_IMAGES = 6\n", "imgs = torch.stack([img for idx in range(NUM_IMAGES) for img in unlabeled_data[idx][0]], dim=0)\n", "img_grid = torchvision.utils.make_grid(imgs, nrow=6, normalize=True, pad_value=0.9)\n", "img_grid = img_grid.permute(1, 2, 0)\n", "\n", "plt.figure(figsize=(10, 5))\n", "plt.title(\"Augmented image examples of the STL10 dataset\")\n", "plt.imshow(img_grid)\n", "plt.axis(\"off\")\n", "plt.show()\n", "plt.close()"]}, {"cell_type": "markdown", "id": "b7287516", "metadata": {"papermill": {"duration": 0.452002, "end_time": "2022-05-03T01:20:01.057929", "exception": false, "start_time": "2022-05-03T01:20:00.605927", "status": "completed"}, "tags": []}, "source": ["We see the wide variety of our data augmentation, including randomly cropping, grayscaling, gaussian blur, and color distortion.\n", "Thus, it remains a challenging task for the model to match two, independently augmented patches of the same image."]}, {"cell_type": "markdown", "id": "2b153298", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.451656, "end_time": "2022-05-03T01:20:01.951310", "exception": false, "start_time": "2022-05-03T01:20:01.499654", "status": "completed"}, "tags": []}, "source": ["### SimCLR implementation\n", "\n", "Using the data loader pipeline above, we can now implement SimCLR.\n", "At each iteration, we get for every image $x$ two differently augmented versions, which we refer to as $\\tilde{x}_i$ and $\\tilde{x}_j$.\n", "Both of these images are encoded into a one-dimensional feature vector, between which we want to maximize similarity which minimizes it to all other images in the batch.\n", "The encoder network is split into two parts: a base encoder network $f(\\cdot)$, and a projection head $g(\\cdot)$.\n", "The base network is usually a deep CNN as we have seen in e.g. [Tutorial 5](https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial5/Inception_ResNet_DenseNet.html) before, and is responsible for extracting a representation vector from the augmented data examples.\n", "In our experiments, we will use the common ResNet-18 architecture as $f(\\cdot)$, and refer to the output as $f(\\tilde{x}_i)=h_i$.\n", "The projection head $g(\\cdot)$ maps the representation $h$ into a space where we apply the contrastive loss, i.e., compare similarities between vectors.\n", "It is often chosen to be a small MLP with non-linearities, and for simplicity, we follow the original SimCLR paper setup by defining it as a two-layer MLP with ReLU activation in the hidden layer.\n", "Note that in the follow-up paper, [SimCLRv2](https://arxiv.org/abs/2006.10029), the authors mention that larger/wider MLPs can boost the performance considerably.\n", "This is why we apply an MLP with four times larger hidden dimensions, but deeper MLPs showed to overfit on the given dataset.\n", "The general setup is visualized below (figure credit - [Ting Chen et al. ](https://arxiv.org/abs/2006.10029)):\n", "\n", ""]}, "metadata": {}, "output_type": "display_data"}], "source": ["%tensorboard --logdir ../saved_models/tutorial17/tensorboards/SimCLR/"]}, {"cell_type": "markdown", "id": "7a78d8fd", "metadata": {"papermill": {"duration": 0.462746, "end_time": "2022-05-03T01:20:11.659979", "exception": false, "start_time": "2022-05-03T01:20:11.197233", "status": "completed"}, "tags": []}, "source": [" {width=\"1200px\"} \n", "\n", "One thing to note is that contrastive learning benefits a lot from long training.\n", "The shown plot above is from a training that took approx.\n", "1 day on a NVIDIA TitanRTX.\n", "Training the model for even longer might reduce its loss further, but we did not experience any gains from it for the downstream task on image classification.\n", "In general, contrastive learning can also benefit from using larger models, if sufficient unlabeled data is available."]}, {"cell_type": "markdown", "id": "74e45567", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.461604, "end_time": "2022-05-03T01:20:12.562657", "exception": false, "start_time": "2022-05-03T01:20:12.101053", "status": "completed"}, "tags": []}, "source": ["## Logistic Regression\n", "\n", "\n", "After we have trained our model via contrastive learning, we can deploy it on downstream tasks and see how well it performs with little data.\n", "A common setup, which also verifies whether the model has learned generalized representations, is to perform Logistic Regression on the features.\n", "In other words, we learn a single, linear layer that maps the representations to a class prediction.\n", "Since the base network $f(\\cdot)$ is not changed during the training process, the model can only perform well if the representations of $h$ describe all features that might be necessary for the task.\n", "Further, we do not have to worry too much about overfitting since we have very few parameters that are trained.\n", "Hence, we might expect that the model can perform well even with very little data.\n", "\n", "First, let's implement a simple Logistic Regression setup for which we assume that the images already have been encoded in their feature vectors.\n", "If very little data is available, it might be beneficial to dynamically encode the images during training so that we can also apply data augmentations.\n", "However, the way we implement it here is much more efficient and can be trained within a few seconds.\n", "Further, using data augmentations did not show any significant gain in this simple setup."]}, {"cell_type": "code", "execution_count": 12, "id": "f5f4cf43", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:12.968818Z", "iopub.status.busy": "2022-05-03T01:20:12.968204Z", "iopub.status.idle": "2022-05-03T01:20:12.976988Z", "shell.execute_reply": "2022-05-03T01:20:12.976285Z"}, "papermill": {"duration": 0.396891, "end_time": "2022-05-03T01:20:12.978486", "exception": false, "start_time": "2022-05-03T01:20:12.581595", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class LogisticRegression(pl.LightningModule):\n", " def __init__(self, feature_dim, num_classes, lr, weight_decay, max_epochs=100):\n", " super().__init__()\n", " self.save_hyperparameters()\n", " # Mapping from representation h to classes\n", " self.model = nn.Linear(feature_dim, num_classes)\n", "\n", " def configure_optimizers(self):\n", " optimizer = optim.AdamW(self.parameters(), lr=self.hparams.lr, weight_decay=self.hparams.weight_decay)\n", " lr_scheduler = optim.lr_scheduler.MultiStepLR(\n", " optimizer, milestones=[int(self.hparams.max_epochs * 0.6), int(self.hparams.max_epochs * 0.8)], gamma=0.1\n", " )\n", " return [optimizer], [lr_scheduler]\n", "\n", " def _calculate_loss(self, batch, mode=\"train\"):\n", " feats, labels = batch\n", " preds = self.model(feats)\n", " loss = F.cross_entropy(preds, labels)\n", " acc = (preds.argmax(dim=-1) == labels).float().mean()\n", "\n", " self.log(mode + \"_loss\", loss)\n", " self.log(mode + \"_acc\", acc)\n", " return loss\n", "\n", " def training_step(self, batch, batch_idx):\n", " return self._calculate_loss(batch, mode=\"train\")\n", "\n", " def validation_step(self, batch, batch_idx):\n", " self._calculate_loss(batch, mode=\"val\")\n", "\n", " def test_step(self, batch, batch_idx):\n", " self._calculate_loss(batch, mode=\"test\")"]}, {"cell_type": "markdown", "id": "5e2787ea", "metadata": {"papermill": {"duration": 0.018113, "end_time": "2022-05-03T01:20:13.017534", "exception": false, "start_time": "2022-05-03T01:20:12.999421", "status": "completed"}, "tags": []}, "source": ["The data we use is the training and test set of STL10.\n", "The training contains 500 images per class, while the test set has 800 images per class."]}, {"cell_type": "code", "execution_count": 13, "id": "9dd16d86", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:13.052558Z", "iopub.status.busy": "2022-05-03T01:20:13.052187Z", "iopub.status.idle": "2022-05-03T01:20:24.436773Z", "shell.execute_reply": "2022-05-03T01:20:24.436125Z"}, "papermill": {"duration": 11.40389, "end_time": "2022-05-03T01:20:24.438275", "exception": false, "start_time": "2022-05-03T01:20:13.034385", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Files already downloaded and verified\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Files already downloaded and verified\n", "Number of training examples: 5000\n", "Number of test examples: 8000\n"]}], "source": ["img_transforms = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])\n", "\n", "train_img_data = STL10(root=DATASET_PATH, split=\"train\", download=True, transform=img_transforms)\n", "test_img_data = STL10(root=DATASET_PATH, split=\"test\", download=True, transform=img_transforms)\n", "\n", "print(\"Number of training examples:\", len(train_img_data))\n", "print(\"Number of test examples:\", len(test_img_data))"]}, {"cell_type": "markdown", "id": "32c05b8d", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.016638, "end_time": "2022-05-03T01:20:24.473910", "exception": false, "start_time": "2022-05-03T01:20:24.457272", "status": "completed"}, "tags": []}, "source": ["Next, we implement a small function to encode all images in our datasets.\n", "The output representations are then used as inputs to the Logistic Regression model."]}, {"cell_type": "code", "execution_count": 14, "id": "ef7009bb", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:24.508714Z", "iopub.status.busy": "2022-05-03T01:20:24.508414Z", "iopub.status.idle": "2022-05-03T01:20:24.514649Z", "shell.execute_reply": "2022-05-03T01:20:24.513993Z"}, "papermill": {"duration": 0.025424, "end_time": "2022-05-03T01:20:24.516127", "exception": false, "start_time": "2022-05-03T01:20:24.490703", "status": "completed"}, "tags": []}, "outputs": [], "source": ["@torch.no_grad()\n", "def prepare_data_features(model, dataset):\n", " # Prepare model\n", " network = deepcopy(model.convnet)\n", " network.fc = nn.Identity() # Removing projection head g(.)\n", " network.eval()\n", " network.to(device)\n", "\n", " # Encode all images\n", " data_loader = data.DataLoader(dataset, batch_size=64, num_workers=NUM_WORKERS, shuffle=False, drop_last=False)\n", " feats, labels = [], []\n", " for batch_imgs, batch_labels in tqdm(data_loader):\n", " batch_imgs = batch_imgs.to(device)\n", " batch_feats = network(batch_imgs)\n", " feats.append(batch_feats.detach().cpu())\n", " labels.append(batch_labels)\n", "\n", " feats = torch.cat(feats, dim=0)\n", " labels = torch.cat(labels, dim=0)\n", "\n", " # Sort images by labels\n", " labels, idxs = labels.sort()\n", " feats = feats[idxs]\n", "\n", " return data.TensorDataset(feats, labels)"]}, {"cell_type": "markdown", "id": "9fb59bbd", "metadata": {"papermill": {"duration": 0.016593, "end_time": "2022-05-03T01:20:24.549347", "exception": false, "start_time": "2022-05-03T01:20:24.532754", "status": "completed"}, "tags": []}, "source": ["Let's apply the function to both training and test set below."]}, {"cell_type": "code", "execution_count": 15, "id": "3a248a10", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:24.584160Z", "iopub.status.busy": "2022-05-03T01:20:24.583612Z", "iopub.status.idle": "2022-05-03T01:20:31.517060Z", "shell.execute_reply": "2022-05-03T01:20:31.515987Z"}, "papermill": {"duration": 6.953263, "end_time": "2022-05-03T01:20:31.519406", "exception": false, "start_time": "2022-05-03T01:20:24.566143", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "db53944efaab43f9a2debdb5ed115889", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/79 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-03T03:20:33.492213 \n", " image/svg+xml \n", " \n", " \n", " Matplotlib v3.5.1, https://matplotlib.org/ \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Test accuracy for 10 images per label: 62.79%\n", "Test accuracy for 20 images per label: 68.60%\n", "Test accuracy for 50 images per label: 74.44%\n", "Test accuracy for 100 images per label: 77.20%\n", "Test accuracy for 200 images per label: 79.06%\n", "Test accuracy for 500 images per label: 81.33%\n"]}], "source": ["dataset_sizes = sorted(k for k in results)\n", "test_scores = [results[k][\"test\"] for k in dataset_sizes]\n", "\n", "fig = plt.figure(figsize=(6, 4))\n", "plt.plot(\n", " dataset_sizes,\n", " test_scores,\n", " \"--\",\n", " color=\"#000\",\n", " marker=\"*\",\n", " markeredgecolor=\"#000\",\n", " markerfacecolor=\"y\",\n", " markersize=16,\n", ")\n", "plt.xscale(\"log\")\n", "plt.xticks(dataset_sizes, labels=dataset_sizes)\n", "plt.title(\"STL10 classification over dataset size\", fontsize=14)\n", "plt.xlabel(\"Number of images per class\")\n", "plt.ylabel(\"Test accuracy\")\n", "plt.minorticks_off()\n", "plt.show()\n", "\n", "for k, score in zip(dataset_sizes, test_scores):\n", " print(f\"Test accuracy for {k:3d} images per label: {100*score:4.2f}%\")"]}, {"cell_type": "markdown", "id": "032436ce", "metadata": {"papermill": {"duration": 0.020744, "end_time": "2022-05-03T01:20:33.741530", "exception": false, "start_time": "2022-05-03T01:20:33.720786", "status": "completed"}, "tags": []}, "source": ["As one would expect, the classification performance improves the more data we have.\n", "However, with only 10 images per class, we can already classify more than 60% of the images correctly.\n", "This is quite impressive, considering that the images are also higher dimensional than e.g. CIFAR10.\n", "With the full dataset, we achieve an accuracy of 81%.\n", "The increase between 50 to 500 images per class might suggest a linear increase in performance with an exponentially larger dataset.\n", "However, with even more data, we could also finetune $f(\\cdot)$ in the training process, allowing for the representations to adapt more to the specific classification task given.\n", "\n", "To set the results above into perspective, we will train the base\n", "network, a ResNet-18, on the classification task from scratch."]}, {"cell_type": "markdown", "id": "6423423e", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.019905, "end_time": "2022-05-03T01:20:33.782372", "exception": false, "start_time": "2022-05-03T01:20:33.762467", "status": "completed"}, "tags": []}, "source": ["## Baseline\n", "\n", "As a baseline to our results above, we will train a standard ResNet-18 with random initialization on the labeled training set of STL10.\n", "The results will give us an indication of the advantages that contrastive learning on unlabeled data has compared to using only supervised training.\n", "The implementation of the model is straightforward since the ResNet\n", "architecture is provided in the torchvision library."]}, {"cell_type": "code", "execution_count": 20, "id": "eca664b0", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:33.824029Z", "iopub.status.busy": "2022-05-03T01:20:33.823695Z", "iopub.status.idle": "2022-05-03T01:20:33.831435Z", "shell.execute_reply": "2022-05-03T01:20:33.830838Z"}, "papermill": {"duration": 0.0306, "end_time": "2022-05-03T01:20:33.832939", "exception": false, "start_time": "2022-05-03T01:20:33.802339", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ResNet(pl.LightningModule):\n", " def __init__(self, num_classes, lr, weight_decay, max_epochs=100):\n", " super().__init__()\n", " self.save_hyperparameters()\n", " self.model = torchvision.models.resnet18(pretrained=False, num_classes=num_classes)\n", "\n", " def configure_optimizers(self):\n", " optimizer = optim.AdamW(self.parameters(), lr=self.hparams.lr, weight_decay=self.hparams.weight_decay)\n", " lr_scheduler = optim.lr_scheduler.MultiStepLR(\n", " optimizer, milestones=[int(self.hparams.max_epochs * 0.7), int(self.hparams.max_epochs * 0.9)], gamma=0.1\n", " )\n", " return [optimizer], [lr_scheduler]\n", "\n", " def _calculate_loss(self, batch, mode=\"train\"):\n", " imgs, labels = batch\n", " preds = self.model(imgs)\n", " loss = F.cross_entropy(preds, labels)\n", " acc = (preds.argmax(dim=-1) == labels).float().mean()\n", "\n", " self.log(mode + \"_loss\", loss)\n", " self.log(mode + \"_acc\", acc)\n", " return loss\n", "\n", " def training_step(self, batch, batch_idx):\n", " return self._calculate_loss(batch, mode=\"train\")\n", "\n", " def validation_step(self, batch, batch_idx):\n", " self._calculate_loss(batch, mode=\"val\")\n", "\n", " def test_step(self, batch, batch_idx):\n", " self._calculate_loss(batch, mode=\"test\")"]}, {"cell_type": "markdown", "id": "124ecc38", "metadata": {"papermill": {"duration": 0.020248, "end_time": "2022-05-03T01:20:33.873624", "exception": false, "start_time": "2022-05-03T01:20:33.853376", "status": "completed"}, "tags": []}, "source": ["It is clear that the ResNet easily overfits on the training data since its parameter count is more than 1000 times larger than the dataset size.\n", "To make the comparison to the contrastive learning models fair, we apply data augmentations similar to the ones we used before: horizontal flip, crop-and-resize, grayscale, and gaussian blur.\n", "Color distortions as before are not used because the color distribution of an image showed to be an important feature for the classification.\n", "Hence, we observed no noticeable performance gains when adding color distortions to the set of augmentations.\n", "Similarly, we restrict the resizing operation before cropping to the max.\n", "125% of its original resolution, instead of 1250% as done in SimCLR.\n", "This is because, for classification, the model needs to recognize the full object, while in contrastive learning, we only want to check whether two patches belong to the same image/object.\n", "Hence, the chosen augmentations below are overall weaker than in the contrastive learning case."]}, {"cell_type": "code", "execution_count": 21, "id": "1862c25b", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:33.918218Z", "iopub.status.busy": "2022-05-03T01:20:33.917353Z", "iopub.status.idle": "2022-05-03T01:20:39.568352Z", "shell.execute_reply": "2022-05-03T01:20:39.567677Z"}, "papermill": {"duration": 5.676295, "end_time": "2022-05-03T01:20:39.569930", "exception": false, "start_time": "2022-05-03T01:20:33.893635", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Files already downloaded and verified\n"]}], "source": ["train_transforms = transforms.Compose(\n", " [\n", " transforms.RandomHorizontalFlip(),\n", " transforms.RandomResizedCrop(size=96, scale=(0.8, 1.0)),\n", " transforms.RandomGrayscale(p=0.2),\n", " transforms.GaussianBlur(kernel_size=9, sigma=(0.1, 0.5)),\n", " transforms.ToTensor(),\n", " transforms.Normalize((0.5,), (0.5,)),\n", " ]\n", ")\n", "\n", "train_img_aug_data = STL10(root=DATASET_PATH, split=\"train\", download=True, transform=train_transforms)"]}, {"cell_type": "markdown", "id": "80f5d59c", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.020075, "end_time": "2022-05-03T01:20:39.612636", "exception": false, "start_time": "2022-05-03T01:20:39.592561", "status": "completed"}, "tags": []}, "source": ["The training function for the ResNet is almost identical to the Logistic Regression setup.\n", "Note that we allow the ResNet to perform validation every 2 epochs to\n", "also check whether the model overfits strongly in the first iterations\n", "or not."]}, {"cell_type": "code", "execution_count": 22, "id": "0f2275d1", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:39.654503Z", "iopub.status.busy": "2022-05-03T01:20:39.653862Z", "iopub.status.idle": "2022-05-03T01:20:39.661745Z", "shell.execute_reply": "2022-05-03T01:20:39.661150Z"}, "papermill": {"duration": 0.030517, "end_time": "2022-05-03T01:20:39.663202", "exception": false, "start_time": "2022-05-03T01:20:39.632685", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def train_resnet(batch_size, max_epochs=100, **kwargs):\n", " trainer = pl.Trainer(\n", " default_root_dir=os.path.join(CHECKPOINT_PATH, \"ResNet\"),\n", " gpus=1 if str(device) == \"cuda:0\" else 0,\n", " max_epochs=max_epochs,\n", " callbacks=[\n", " ModelCheckpoint(save_weights_only=True, mode=\"max\", monitor=\"val_acc\"),\n", " LearningRateMonitor(\"epoch\"),\n", " ],\n", " progress_bar_refresh_rate=1,\n", " check_val_every_n_epoch=2,\n", " )\n", " trainer.logger._default_hp_metric = None\n", "\n", " # Data loaders\n", " train_loader = data.DataLoader(\n", " train_img_aug_data,\n", " batch_size=batch_size,\n", " shuffle=True,\n", " drop_last=True,\n", " pin_memory=True,\n", " num_workers=NUM_WORKERS,\n", " )\n", " test_loader = data.DataLoader(\n", " test_img_data, batch_size=batch_size, shuffle=False, drop_last=False, pin_memory=True, num_workers=NUM_WORKERS\n", " )\n", "\n", " # Check whether pretrained model exists. If yes, load it and skip training\n", " pretrained_filename = os.path.join(CHECKPOINT_PATH, \"ResNet.ckpt\")\n", " if os.path.isfile(pretrained_filename):\n", " print(\"Found pretrained model at %s, loading...\" % pretrained_filename)\n", " model = ResNet.load_from_checkpoint(pretrained_filename)\n", " else:\n", " pl.seed_everything(42) # To be reproducable\n", " model = ResNet(**kwargs)\n", " trainer.fit(model, train_loader, test_loader)\n", " model = ResNet.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)\n", "\n", " # Test best model on validation set\n", " train_result = trainer.test(model, dataloaders=train_loader, verbose=False)\n", " val_result = trainer.test(model, dataloaders=test_loader, verbose=False)\n", " result = {\"train\": train_result[0][\"test_acc\"], \"test\": val_result[0][\"test_acc\"]}\n", "\n", " return model, result"]}, {"cell_type": "markdown", "id": "c4714748", "metadata": {"papermill": {"duration": 0.020137, "end_time": "2022-05-03T01:20:39.703439", "exception": false, "start_time": "2022-05-03T01:20:39.683302", "status": "completed"}, "tags": []}, "source": ["Finally, let's train the model and check its results:"]}, {"cell_type": "code", "execution_count": 23, "id": "88cd7c3d", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:39.745718Z", "iopub.status.busy": "2022-05-03T01:20:39.745165Z", "iopub.status.idle": "2022-05-03T01:20:44.677634Z", "shell.execute_reply": "2022-05-03T01:20:44.676950Z"}, "papermill": {"duration": 4.955437, "end_time": "2022-05-03T01:20:44.679190", "exception": false, "start_time": "2022-05-03T01:20:39.723753", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:96: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=1)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.\n", " rank_zero_deprecation(\n", "GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Missing logger folder: saved_models/ContrastiveLearning/ResNet/lightning_logs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model at saved_models/ContrastiveLearning/ResNet.ckpt, loading...\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"name": "stderr", "output_type": "stream", "text": ["/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:486: PossibleUserWarning: Your `test_dataloader`'s sampler has shuffling enabled, it is strongly recommended that you turn shuffling off for val/test/predict dataloaders.\n", " rank_zero_warn(\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "b2ad4f8f9c6048a1a5e119627c964924", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "4a1ca1d04c6d4a62a0c8bbdf62815bf2", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Accuracy on training set: 99.70%\n", "Accuracy on test set: 73.31%\n"]}], "source": ["resnet_model, resnet_result = train_resnet(batch_size=64, num_classes=10, lr=1e-3, weight_decay=2e-4, max_epochs=100)\n", "print(f\"Accuracy on training set: {100*resnet_result['train']:4.2f}%\")\n", "print(f\"Accuracy on test set: {100*resnet_result['test']:4.2f}%\")"]}, {"cell_type": "markdown", "id": "534121f5", "metadata": {"papermill": {"duration": 0.020667, "end_time": "2022-05-03T01:20:44.722731", "exception": false, "start_time": "2022-05-03T01:20:44.702064", "status": "completed"}, "tags": []}, "source": ["The ResNet trained from scratch achieves 73.31% on the test set.\n", "This is almost 8% less than the contrastive learning model, and even slightly less than SimCLR achieves with 1/10 of the data.\n", "This shows that self-supervised, contrastive learning provides\n", "considerable performance gains by leveraging large amounts of unlabeled\n", "data when little labeled data is available."]}, {"cell_type": "markdown", "id": "2587ddd2", "metadata": {"papermill": {"duration": 0.021179, "end_time": "2022-05-03T01:20:44.764832", "exception": false, "start_time": "2022-05-03T01:20:44.743653", "status": "completed"}, "tags": []}, "source": ["## Conclusion\n", "\n", "In this tutorial, we have discussed self-supervised contrastive learning and implemented SimCLR as an example method.\n", "We have applied it to the STL10 dataset and showed that it can learn generalizable representations that we can use to train simple classification models.\n", "With 500 images per label, it achieved an 8% higher accuracy than a similar model solely trained from supervision and performs on par with it when only using a tenth of the labeled data.\n", "Our experimental results are limited to a single dataset, but recent works such as [Ting Chen et al. ](https://arxiv.org/abs/2006.10029) showed similar trends for larger datasets like ImageNet.\n", "Besides the discussed hyperparameters, the size of the model seems to be important in contrastive learning as well.\n", "If a lot of unlabeled data is available, larger models can achieve much stronger results and come close to their supervised baselines.\n", "Further, there are also approaches for combining contrastive and supervised learning, leading to performance gains beyond supervision (see [Khosla et al.](https://arxiv.org/abs/2004.11362)).\n", "Moreover, contrastive learning is not the only approach to self-supervised learning that has come up in the last two years and showed great results.\n", "Other methods include distillation-based methods like [BYOL](https://arxiv.org/abs/2006.07733) and redundancy reduction techniques like [Barlow Twins](https://arxiv.org/abs/2103.03230).\n", "There is a lot more to explore in the self-supervised domain, and more, impressive steps ahead are to be expected.\n", "\n", "### References\n", "\n", "[1] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020).\n", "A simple framework for contrastive learning of visual representations.\n", "In International conference on machine learning (pp.\n", "1597-1607).\n", "PMLR.\n", "([link](https://arxiv.org/abs/2002.05709))\n", "\n", "[2] Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. (2020).\n", "Big self-supervised models are strong semi-supervised learners.\n", "NeurIPS 2021 ([link](https://arxiv.org/abs/2006.10029)).\n", "\n", "[3] Oord, A. V. D., Li, Y., and Vinyals, O.\n", "(2018).\n", "Representation learning with contrastive predictive coding.\n", "arXiv preprint arXiv:1807.03748.\n", "([link](https://arxiv.org/abs/1807.03748))\n", "\n", "[4] Grill, J.B., Strub, F., Altch\u00e9, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G.\n", "and Piot, B.\n", "(2020).\n", "Bootstrap your own latent: A new approach to self-supervised learning.\n", "arXiv preprint arXiv:2006.07733.\n", "([link](https://arxiv.org/abs/2006.07733))\n", "\n", "[5] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C. and Krishnan, D. (2020).\n", "Supervised contrastive learning.\n", "arXiv preprint arXiv:2004.11362.\n", "([link](https://arxiv.org/abs/2004.11362))\n", "\n", "[6] Zbontar, J., Jing, L., Misra, I., LeCun, Y. and Deny, S. (2021).\n", "Barlow twins: Self-supervised learning via redundancy reduction.\n", "arXiv preprint arXiv:2103.03230.\n", "([link](https://arxiv.org/abs/2103.03230))"]}, {"cell_type": "markdown", "id": "e02c21fe", "metadata": {"papermill": {"duration": 0.02056, "end_time": "2022-05-03T01:20:44.806421", "exception": false, "start_time": "2022-05-03T01:20:44.785861", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://www.pytorchlightning.ai/community)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "[{height=\"60px\" width=\"240px\"}](https://pytorchlightning.ai)"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Tutorial 13: Self-Supervised Contrastive Learning with SimCLR\n", " :card_description: In this tutorial, we will take a closer look at self-supervised contrastive learning. Self-supervised learning, or also sometimes called unsupervised learning, describes the...\n", " :tags: Image,Self-Supervised,Contrastive-Learning,GPU/TPU,UvA-DL-Course\n", " :image: _static/images/course_UvA-DL/13-contrastive-learning.jpg"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab,colab_type,id,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10"}, "papermill": {"default_parameters": {}, "duration": 177.423578, "end_time": "2022-05-03T01:20:47.748293", "environment_variables": {}, "exception": null, "input_path": "course_UvA-DL/13-contrastive-learning/SimCLR.ipynb", "output_path": ".notebooks/course_UvA-DL/13-contrastive-learning.ipynb", "parameters": {}, "start_time": "2022-05-03T01:17:50.324715", "version": "2.3.4"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"00e27bbf1ad746ac8b38113edba37fb9": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "04307e8b79224b07816302a0abf3029a": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "043eb2e39d1d4a27a6aefc5b8a115627": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_e6899aaa29ed44dbaefd3c06a11667ce", "IPY_MODEL_205dba58ef084e98a14b250d4045be76", "IPY_MODEL_9df5be0f3d804da5a47abc2307fa16b1"], "layout": "IPY_MODEL_601f7b23fed2470cab5c5345279de811"}}, "044236478d5a49288728e28cf31b7436": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "07493b3196b5495c8bb109f290fbb0fd": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "13fce431063a407aa69e6d9eebdeae27": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "1a54df1a5b974b319f1fc98f4c8da6d5": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "1ee3daae85c84ac2acd5f6ee3e5f3991": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "205dba58ef084e98a14b250d4045be76": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_d8a981d41b054a34a03261376e80102e", "max": 125.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_33c9b5f58c2d41a48cc25e4481f1f1e3", "value": 125.0}}, "243fa3e12d424fba873037e498215fa8": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "264603d530684d0199cf34f8bdc9f001": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "299ca9d6b6c84c4ea43e2790abc7befb": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_00e27bbf1ad746ac8b38113edba37fb9", "placeholder": "\u200b", "style": "IPY_MODEL_653857e1847f4504922fd7fe2839afaf", "value": " 2640397312/? [01:11<00:00, 40183941.25it/s]"}}, "2d6cc1ea54ae478cb37f5b1d5fbc8cf0": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "2f9497c58400453fafc5d2752032211d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b3c659d0a0004dfba0897dfa4ad4d36e", "max": 79.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_6c4ba510a6974b36b57989ea2cbb740d", "value": 79.0}}, "33c9b5f58c2d41a48cc25e4481f1f1e3": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "366e7e0e78824270ae9887b048078c50": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_480439506d1245fbb50af9fe6242cb0c", "placeholder": "\u200b", "style": "IPY_MODEL_381356794a3743ce841fbce9cdde0bcb", "value": "Testing DataLoader 0: 100%"}}, "381356794a3743ce841fbce9cdde0bcb": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "47f963c08b004920bd1adf2bad99f8cd": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "480439506d1245fbb50af9fe6242cb0c": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "4a1ca1d04c6d4a62a0c8bbdf62815bf2": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_838527cbb7d143f7926056f8eacc67a4", "IPY_MODEL_6e0287841cbe4c51abd04a4c1995e04b", "IPY_MODEL_a53ab1e1530c4972b485a3253e857aa1"], "layout": "IPY_MODEL_5a5c620782e646d1bcd3e7319482c986"}}, "5310dd83d81049e197ef20029c8e21bf": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "56df4593adf043af8f3a696e84420350": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "5a5c620782e646d1bcd3e7319482c986": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "601f7b23fed2470cab5c5345279de811": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "62324770a38042179fba4c6f4a3e0239": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "653857e1847f4504922fd7fe2839afaf": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "6c4ba510a6974b36b57989ea2cbb740d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "6e0287841cbe4c51abd04a4c1995e04b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_c568acd119024252ba2f2d5001f7427f", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_07493b3196b5495c8bb109f290fbb0fd", "value": 1.0}}, "72a8bc03aeac4c5ea8e32b6e98f17037": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "7d304cb53aff40379380780ff584a74c": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "80f951080478447286c86ddf4b6c5f73": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "838527cbb7d143f7926056f8eacc67a4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_1a54df1a5b974b319f1fc98f4c8da6d5", "placeholder": "\u200b", "style": "IPY_MODEL_a17edfb1142e406faf205addd2e024d8", "value": "Testing DataLoader 0: 100%"}}, "86d26235ddac41b88d9052197fb77767": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8fdc83a860dc4e219b0247935f1e644f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "91b05fc1d540459a81497f53bde87950": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_2d6cc1ea54ae478cb37f5b1d5fbc8cf0", "max": 2640397119.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_e133a2896b67452ca3c78e6f0d2a9fd7", "value": 2640397119.0}}, "9df5be0f3d804da5a47abc2307fa16b1": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_264603d530684d0199cf34f8bdc9f001", "placeholder": "\u200b", "style": "IPY_MODEL_8fdc83a860dc4e219b0247935f1e644f", "value": " 125/125 [00:01<00:00, 100.02it/s]"}}, "a17edfb1142e406faf205addd2e024d8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "a53ab1e1530c4972b485a3253e857aa1": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_80f951080478447286c86ddf4b6c5f73", "placeholder": "\u200b", "style": "IPY_MODEL_13fce431063a407aa69e6d9eebdeae27", "value": " 125/125 [00:01<00:00, 75.65it/s]"}}, "ac5e41f1640040738b17ab9c2a1d1ca5": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "b2ad4f8f9c6048a1a5e119627c964924": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_366e7e0e78824270ae9887b048078c50", "IPY_MODEL_da78c363b06d44b39a0168fb28111c67", "IPY_MODEL_d92e42fe29744a9bb011f3691161676f"], "layout": "IPY_MODEL_04307e8b79224b07816302a0abf3029a"}}, "b3c659d0a0004dfba0897dfa4ad4d36e": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "b78a17b1b4094697b5df196ba96685b2": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "bb0ce858c1f845a58e795f6357c1b354": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "c485c055a8004a9392c1f5f0f656e384": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_fb58f4c5ddb04185bd9277ea335205de", "IPY_MODEL_91b05fc1d540459a81497f53bde87950", "IPY_MODEL_299ca9d6b6c84c4ea43e2790abc7befb"], "layout": "IPY_MODEL_f40336122d1a40ec9cbdeada8c72d124"}}, "c568acd119024252ba2f2d5001f7427f": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "c74401b1bf5341a8b6dc264057e782f3": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_044236478d5a49288728e28cf31b7436", "placeholder": "\u200b", "style": "IPY_MODEL_56df4593adf043af8f3a696e84420350", "value": " 79/79 [00:01<00:00, 85.11it/s]"}}, "d6f83b5682e84e5e880e00d5ffc868a4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_7d304cb53aff40379380780ff584a74c", "placeholder": "\u200b", "style": "IPY_MODEL_ac5e41f1640040738b17ab9c2a1d1ca5", "value": "100%"}}, "d8a981d41b054a34a03261376e80102e": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "d92e42fe29744a9bb011f3691161676f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b78a17b1b4094697b5df196ba96685b2", "placeholder": "\u200b", "style": "IPY_MODEL_47f963c08b004920bd1adf2bad99f8cd", "value": " 78/78 [00:02<00:00, 27.54it/s]"}}, "da78c363b06d44b39a0168fb28111c67": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_62324770a38042179fba4c6f4a3e0239", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_72a8bc03aeac4c5ea8e32b6e98f17037", "value": 1.0}}, "db53944efaab43f9a2debdb5ed115889": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_d6f83b5682e84e5e880e00d5ffc868a4", "IPY_MODEL_2f9497c58400453fafc5d2752032211d", "IPY_MODEL_c74401b1bf5341a8b6dc264057e782f3"], "layout": "IPY_MODEL_86d26235ddac41b88d9052197fb77767"}}, "e133a2896b67452ca3c78e6f0d2a9fd7": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "e6899aaa29ed44dbaefd3c06a11667ce": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_5310dd83d81049e197ef20029c8e21bf", "placeholder": "\u200b", "style": "IPY_MODEL_bb0ce858c1f845a58e795f6357c1b354", "value": "100%"}}, "f40336122d1a40ec9cbdeada8c72d124": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "fb58f4c5ddb04185bd9277ea335205de": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_243fa3e12d424fba873037e498215fa8", "placeholder": "\u200b", "style": "IPY_MODEL_1ee3daae85c84ac2acd5f6ee3e5f3991", "value": ""}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5} ![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/13-contrastive-learning/simclr_data_augmentations.jpg\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/13-contrastive-learning/crop_views.svg\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/13-contrastive-learning/simclr_network_setup.svg\")