{"cells": [{"cell_type": "markdown", "id": "9b0d3dbf", "metadata": {"papermill": {"duration": 0.012709, "end_time": "2021-10-25T20:00:43.480755", "exception": false, "start_time": "2021-10-25T20:00:43.468046", "status": "completed"}, "tags": []}, "source": ["\n", "# Barlow Twins Tutorial\n", "\n", "* **Author:** Ananya Harsh Jha (ananya@pytorchlightning.ai)\n", "* **License:** CC BY-SA\n", "* **Generated:** 2021-10-25T22:00:34.269471\n", "\n", "This notebook describes the self-supervised learning method Barlow Twins.\n", "Barlow Twins differs from other recently proposed algorithms as it doesn't\n", "fall under the category of either contrastive learning, or methods like knowledge\n", "distillation or clustering. The simplicity of the loss function and its effectiveness\n", "in comparison to the current state of the art makes Barlow Twins an interesting\n", "case study.\n", "\n", "\n", "---\n", "Open in [![Open In Colab](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHUAAAAUCAYAAACzrHJDAAAIuUlEQVRoQ+1ZaVRURxb+qhdolmbTUVSURpZgmLhHbQVFZIlGQBEXcMvJhKiTEzfigjQg7oNEJ9GMGidnjnNMBs2czIzajksEFRE1xklCTKJiQLRFsUGkoUWw+82pamn79etGYoKek1B/4NW99/tu3e/dquJBAGD27NkHALxKf39WY39gyrOi+i3xqGtUoePJrFmznrmgtModorbTu8YRNZk5cybXTvCtwh7o6NR2KzuZMWNGh6jtVt7nA0ymT5/eJlF9POrh7PAQl6s8bGYa3PUum//htmebVtLRqW0q01M5keTk5FZFzU0oRle3+zxwg5Hgtb+PZiL/ZVohxCI+hL5JgjmfjPxZ26+33BG3dA+ealHPM4gQAo5rU59gsI8bRvl54t3Ca62mvHyUAhtOlLd5WSQpKcluBjumnoCLs1EARkVd9E8l3p9y2i7RbQ1B6pFwu/YDgW8KbHJHMTQrwnjz2oZm9M4pavOCfo5jWrgCaaMVcMs6/pNhDr0+AMN93XlxV7R6DNpyzi7W/OE+yIrsjU6rTrbKV5cd/pNyItOmTbMp6sbBB+EqaYJY4cWE3VUciNt1TpgfcRFv71Fi54xT5kSoyLvOBEJMOMxWXkFlBeBSX4u6Zkcs+3KszYRtiapbNRqF31UgetVuc8z9vBXIv1qD+F1f83B6uDlCUyfsZGepGPpmg01OB7EITQbhS9ribKy+DmP1DUiClLz4bnIHVOqa7BY+Z1wg5g3zgUvyehiNpnJKxSLc/ts76LKm0BzX3c0RNy1yXjDcB5lWoro4iNHQxM+f1kWeWQARAWQS++trISJTp061Kep25X/MycwtjuctSC5rxo7ppi7VNUox5+PhPHtrsS2O1qJ6yx1QujQUzm9sh6hbkBlvvGcN8hYnwjUjH6kjfZEd5c/jitz5Jc5U3ENnFynKl4eB7nyEgP2UZ+Yz3/rVEbyYr27qELrtC4FIC0J7sc7xWnmccdHfRRTs0VB+cA4lt+oFcRR/wUeH8FG5w2Mbx8FQ8TXEvv1xYf4wBP3O2WyL3/UVjpXWgIqaFeUPr+wTmDvUB7njH6/bOv+HRg4SqioAg5GDe1aB3ZeMTJkyRSBqkLsWqSEm0fZVBEN94zEZnYvrdx1JL5cxe+a+AbhSJecRRHW/ikTFRTa38dtQlNZ5CRKwFvUtZU/kvBoEF9Uxni/XqIM+dwKbTw3rhcxIf7gmr2M+H6SMwx8iBzJbw5oxeG3Lv5FX9B3AGaHPS8e8z77H7v9VMpvPG5ug1enh7eGK8h0LBTwUb+GInqzInlRUK65DmTPQu4c3+uQKjwKK77zwUxBX4Tq7yR1RuiwUsqlrABCM6esHdXoy47fk4+prYKy8ZF574x4V5BnHQBuf4g9Z9ld8U36L2aktZNNplNfw7zotwWTy5MkCUft4aLEopJj5/OPHl1BQqeAVOnHgNSQOqmBzq9V9cfEm/yx5ubMGKS9cYPZ3vx2OS/c6PVHUuUO7Y1Pci3BO/1zgq18byebfGemLtNF+6JRtOvMk926ibussZqM+1mNz4TWkH7rCbM5phwGRGDAaoF8fY5OHFnlldAA8sgoEXKnDukA1NgSeNjqkJT9brbN4pC9WRweYXyLugR73c+MYvyWfu0yC6+mjzN1Isfw3FKJS98CU/zI1IHFkFPR52cHL2FJk0sB6kMTERIGo9GzcPkLNfA0cwdwi/hfEYO86ZMd9w+y1egfM2T2Eh/vesMNwljSzuZRT420SW3eqy8N6aHMmwmnFUZ7/PGVPbIoNZvNU1BURdHs0bT2+HjL8sDSM2e6vi4Lj5NW8WOLVA6RTT2azxLV+bglaFNqLieqemS/gWkw7NyoAHo+2dEsiivengjKsPFoqWOvbSh/kxPaxyW/JRzH2Fl3EzD9/xjAefJqB3usKUFn/0Gb+S/d/jy3FN2yLOmnSJJtn6oehByEiHPSeXnDxFGPRnoFoaBJjcdQlbDwcjL1zTNuQpoxD7R0OG0uUTMi0fkVwdzBdYIwcwZunxrVJVLplNm54BZp7jfDfYLoNyqQi1K6KxIdHzmN+QQ2WjFIwUT2zTGdlRXo4NFXVUO4sgX5dFC7f0aP/ZlNeUjFBuL8Xjl6uRuP6aMjSjpjzsH62FDU7JhBuGccEXIvDfJFFBc/gHw80dklfCVYnRaDfpiJcutPA4F7qJsfJeUPQI+1fqMlNhFx1FM0GDqkjFVg7NojlQ0Vt4aM5ReSqcbpaCg8nCW5lRsBvbT4T1TLfFptsfh7gItzuKTdJSEiwKSrt1vcmnEXXrsLbYnWDA1bu+z2WKy9Arq+1KRqdfKsoBo0GcdtEpS/B1bO4v0cFiUhkjskvKcMrWwtAPHuwQq8Z+4LZ1vTQANfXt4J0DwZX9gWa9qh4XDM/voC9JXfwYEMMHJcfNtusn82ihvliVUwg5KrPGVf6GH94ZJpEZBen6EC4qYTHA1dXhW0JIex8txzv//c8lhzXIi/BFxOH9jGbQhZsRalTIBZZ8KkGyZAxeRQvXkFF1TWz/Hm46jNYUnjPbt3JxIkT7f6dSj8qfJJyVvBxgaIlblOyjtysNHWN9fjjqWi7glJfW3/S0Hlj2XnA8PhKT9w6g3Qx3XiXhvuxQsuT1proxBKI/AaZqY1Xz5muvY8G8XkRRCaHsfQsRAFDH/tZPbcYuHotOG0FRIqB4HR3wNVoIPLtz8ycTguu+jpEigE218vd1YCr5m+HpHMvEI9u4LTXwNWaLjl0iPwGAmIpeHx1VeCqTJdPs1/vweweQPO3HC24NhOhnTphwoQnfv6QSY2ICbkNmdSA4h87oaLaiYfn5diIEd4att2erOwJXbPUHp953p6orQVSUVWRAXBT8c/dJ5L9xhzaJGp71GR/wFP8P5V2z10NSC9T93QM2xUg8fHxT+zU9ijeU4naHon8CjFJXFzc8/kn+dN06q9QgF98SYSo2Xen2NjYZy5sR6f+4nLSK5Iam2PH/x87a1YN/t5sBgAAAABJRU5ErkJggg==){height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/barlow-twins.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/latest/)\n", "| Join us [on Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)"]}, {"cell_type": "markdown", "id": "a0f8a9cf", "metadata": {"papermill": {"duration": 0.011144, "end_time": "2021-10-25T20:00:43.503475", "exception": false, "start_time": "2021-10-25T20:00:43.492331", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "9ba1fb3e", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2021-10-25T20:00:43.529599Z", "iopub.status.busy": "2021-10-25T20:00:43.529097Z", "iopub.status.idle": "2021-10-25T20:00:43.531815Z", "shell.execute_reply": "2021-10-25T20:00:43.531280Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 0.017035, "end_time": "2021-10-25T20:00:43.531927", "exception": false, "start_time": "2021-10-25T20:00:43.514892", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# ! pip install --quiet \"torch>=1.6, <1.9\" \"torchmetrics>=0.3\" \"torchvision\" \"matplotlib\" \"pytorch-lightning>=1.3\""]}, {"cell_type": "markdown", "id": "23d70cf8", "metadata": {"papermill": {"duration": 0.011167, "end_time": "2021-10-25T20:00:43.554606", "exception": false, "start_time": "2021-10-25T20:00:43.543439", "status": "completed"}, "tags": []}, "source": ["## Barlow Twins\n", "\n", "Barlow Twins finds itself in unique place amongst the current state-of-the-art self-supervised learning methods. It does not fall under the existing categories of contrastive learning, knowledge distillation or clustering based methods. Instead, it creates its own category of redundancy reductionand achieves competitive performance with a simple yet effective loss function. In this tutorial, we look at coding up a small version of Barlow Twins algorithm using PyTorch Lightning."]}, {"cell_type": "code", "execution_count": 2, "id": "29341bf4", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:43.584461Z", "iopub.status.busy": "2021-10-25T20:00:43.583974Z", "iopub.status.idle": "2021-10-25T20:00:44.983203Z", "shell.execute_reply": "2021-10-25T20:00:44.982750Z"}, "papermill": {"duration": 1.417567, "end_time": "2021-10-25T20:00:44.983322", "exception": false, "start_time": "2021-10-25T20:00:43.565755", "status": "completed"}, "tags": []}, "outputs": [], "source": ["from functools import partial\n", "from typing import Sequence, Tuple, Union\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pytorch_lightning as pl\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torchvision.transforms as transforms\n", "import torchvision.transforms.functional as VisionF\n", "from pytorch_lightning import Callback, LightningModule, Trainer\n", "from pytorch_lightning.callbacks import ModelCheckpoint\n", "from pytorch_lightning.metrics.functional import accuracy\n", "from torch.utils.data import DataLoader\n", "from torchvision.datasets import CIFAR10\n", "from torchvision.models.resnet import resnet18\n", "from torchvision.utils import make_grid\n", "\n", "batch_size = 32\n", "num_workers = 0 # to run notebook on CPU\n", "max_epochs = 200\n", "z_dim = 128"]}, {"cell_type": "markdown", "id": "721a96db", "metadata": {"papermill": {"duration": 0.011192, "end_time": "2021-10-25T20:00:45.006236", "exception": false, "start_time": "2021-10-25T20:00:44.995044", "status": "completed"}, "tags": []}, "source": ["### Transforms\n", "\n", "We first define the data augmentation pipeline used in Barlow Twins. Here, we use pipeline proposed in SimCLR, which generates two copies/views of an input image by applying the following transformations in a sequence.\n", "\n", "First it takes a random crop of the image and resizes it to a fixed pre-specified size. Then, it applies a left-to-right random flip with a probability of 0.5. This step is followed by a composition of color jitter, conversion to grayscale with a probability of 0.2 and the application of a Gaussian blur filter. Finally, we normalize the image and convert it to a tensor.\n", "\n", "Within this transform, we add a third view for our online finetuner, which we explain later on. But, to explain things quickly here, we add a another transform to perform perform test our encoder on a downstream classification task."]}, {"cell_type": "code", "execution_count": 3, "id": "6cb8b894", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:45.037011Z", "iopub.status.busy": "2021-10-25T20:00:45.030646Z", "iopub.status.idle": "2021-10-25T20:00:45.038691Z", "shell.execute_reply": "2021-10-25T20:00:45.039079Z"}, "papermill": {"duration": 0.021697, "end_time": "2021-10-25T20:00:45.039192", "exception": false, "start_time": "2021-10-25T20:00:45.017495", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class BarlowTwinsTransform:\n", " def __init__(self, train=True, input_height=224, gaussian_blur=True, jitter_strength=1.0, normalize=None):\n", "\n", " self.input_height = input_height\n", " self.gaussian_blur = gaussian_blur\n", " self.jitter_strength = jitter_strength\n", " self.normalize = normalize\n", " self.train = train\n", "\n", " color_jitter = transforms.ColorJitter(\n", " 0.8 * self.jitter_strength,\n", " 0.8 * self.jitter_strength,\n", " 0.8 * self.jitter_strength,\n", " 0.2 * self.jitter_strength,\n", " )\n", "\n", " color_transform = [transforms.RandomApply([color_jitter], p=0.8), transforms.RandomGrayscale(p=0.2)]\n", "\n", " if self.gaussian_blur:\n", " kernel_size = int(0.1 * self.input_height)\n", " if kernel_size % 2 == 0:\n", " kernel_size += 1\n", "\n", " color_transform.append(transforms.RandomApply([transforms.GaussianBlur(kernel_size=kernel_size)], p=0.5))\n", "\n", " self.color_transform = transforms.Compose(color_transform)\n", "\n", " if normalize is None:\n", " self.final_transform = transforms.ToTensor()\n", " else:\n", " self.final_transform = transforms.Compose([transforms.ToTensor(), normalize])\n", "\n", " self.transform = transforms.Compose(\n", " [\n", " transforms.RandomResizedCrop(self.input_height),\n", " transforms.RandomHorizontalFlip(p=0.5),\n", " self.color_transform,\n", " self.final_transform,\n", " ]\n", " )\n", "\n", " self.finetune_transform = None\n", " if self.train:\n", " self.finetune_transform = transforms.Compose(\n", " [\n", " transforms.RandomCrop(32, padding=4, padding_mode=\"reflect\"),\n", " transforms.RandomHorizontalFlip(),\n", " transforms.ToTensor(),\n", " ]\n", " )\n", " else:\n", " self.finetune_transform = transforms.ToTensor()\n", "\n", " def __call__(self, sample):\n", " return self.transform(sample), self.transform(sample), self.finetune_transform(sample)"]}, {"cell_type": "markdown", "id": "69b57f48", "metadata": {"papermill": {"duration": 0.011342, "end_time": "2021-10-25T20:00:45.061887", "exception": false, "start_time": "2021-10-25T20:00:45.050545", "status": "completed"}, "tags": []}, "source": ["### Dataset\n", "\n", "We select CIFAR10 as the dataset to demonstrate the pre-training process for Barlow Twins. CIFAR10 images are 32x32 in size and we do not apply a Gaussian blur transformation on them. In this step, we create the training and validation dataloaders for CIFAR10."]}, {"cell_type": "code", "execution_count": 4, "id": "273da819", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:45.091605Z", "iopub.status.busy": "2021-10-25T20:00:45.086769Z", "iopub.status.idle": "2021-10-25T20:00:49.752232Z", "shell.execute_reply": "2021-10-25T20:00:49.751801Z"}, "papermill": {"duration": 4.679129, "end_time": "2021-10-25T20:00:49.752355", "exception": false, "start_time": "2021-10-25T20:00:45.073226", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./cifar-10-python.tar.gz\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "b13dfae7160c4c8586c5780f597cd8e4", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/170498071 [00:00"]}, "metadata": {}, "output_type": "display_data"}], "source": ["for batch in val_loader:\n", " (img1, img2, _), label = batch\n", " break\n", "\n", "img_grid = make_grid(img1, normalize=True)\n", "\n", "\n", "def show(imgs):\n", " if not isinstance(imgs, list):\n", " imgs = [imgs]\n", " fix, axs = plt.subplots(ncols=len(imgs), squeeze=False)\n", " for i, img in enumerate(imgs):\n", " img = img.detach()\n", " img = VisionF.to_pil_image(img)\n", " axs[0, i].imshow(np.asarray(img))\n", " axs[0, i].set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])\n", "\n", "\n", "show(img_grid)"]}, {"cell_type": "markdown", "id": "a0233366", "metadata": {"papermill": {"duration": 0.018776, "end_time": "2021-10-25T20:00:50.033993", "exception": false, "start_time": "2021-10-25T20:00:50.015217", "status": "completed"}, "tags": []}, "source": ["### Barlow Twins Loss\n", "\n", "Here we define the loss function for Barlow Twins. It first normalizes the D dimensinonal vectors from the projection head and then computes the DxD cross-correlation matrix between the normalized vectors of the 2 views of each image.\n", "\n", "Then it splits this cross-correlation matrix into two parts. The first part, the diagonal of this matrix is brought closer to 1, which pushes up the cosine similarity between the latent vectors of two views of each image, thus making the backbone invariant to the transformations applied to the views. The second part of the loss pushes the non-diagonal elements of the cross-corrlelation matrix closes to 0. This reduces the redundancy between the different dimensions of the latent vector."]}, {"cell_type": "code", "execution_count": 6, "id": "7f3f6369", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.069422Z", "iopub.status.busy": "2021-10-25T20:00:50.068917Z", "iopub.status.idle": "2021-10-25T20:00:50.071108Z", "shell.execute_reply": "2021-10-25T20:00:50.070701Z"}, "papermill": {"duration": 0.02278, "end_time": "2021-10-25T20:00:50.071208", "exception": false, "start_time": "2021-10-25T20:00:50.048428", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class BarlowTwinsLoss(nn.Module):\n", " def __init__(self, batch_size, lambda_coeff=5e-3, z_dim=128):\n", " super().__init__()\n", "\n", " self.z_dim = z_dim\n", " self.batch_size = batch_size\n", " self.lambda_coeff = lambda_coeff\n", "\n", " def off_diagonal_ele(self, x):\n", " # taken from: https://github.com/facebookresearch/barlowtwins/blob/main/main.py\n", " # return a flattened view of the off-diagonal elements of a square matrix\n", " n, m = x.shape\n", " assert n == m\n", " return x.flatten()[:-1].view(n - 1, n + 1)[:, 1:].flatten()\n", "\n", " def forward(self, z1, z2):\n", " # N x D, where N is the batch size and D is output dim of projection head\n", " z1_norm = (z1 - torch.mean(z1, dim=0)) / torch.std(z1, dim=0)\n", " z2_norm = (z2 - torch.mean(z2, dim=0)) / torch.std(z2, dim=0)\n", "\n", " cross_corr = torch.matmul(z1_norm.T, z2_norm) / self.batch_size\n", "\n", " on_diag = torch.diagonal(cross_corr).add_(-1).pow_(2).sum()\n", " off_diag = self.off_diagonal_ele(cross_corr).pow_(2).sum()\n", "\n", " return on_diag + self.lambda_coeff * off_diag"]}, {"cell_type": "markdown", "id": "050c8b3c", "metadata": {"papermill": {"duration": 0.014594, "end_time": "2021-10-25T20:00:50.101352", "exception": false, "start_time": "2021-10-25T20:00:50.086758", "status": "completed"}, "tags": []}, "source": ["### Backbone\n", "\n", "This is a standard Resnet backbone that we pre-train using the Barlow Twins method. To accommodate the 32x32 CIFAR10 images, we replace the first 7x7 convolution of the Resnet backbone by a 3x3 filter. We also remove the first Maxpool layer from the network for CIFAR10 images."]}, {"cell_type": "code", "execution_count": 7, "id": "a1df9ad9", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.134356Z", "iopub.status.busy": "2021-10-25T20:00:50.133886Z", "iopub.status.idle": "2021-10-25T20:00:50.315984Z", "shell.execute_reply": "2021-10-25T20:00:50.315481Z"}, "papermill": {"duration": 0.200049, "end_time": "2021-10-25T20:00:50.316105", "exception": false, "start_time": "2021-10-25T20:00:50.116056", "status": "completed"}, "tags": []}, "outputs": [], "source": ["encoder = resnet18()\n", "\n", "# for CIFAR10, replace the first 7x7 conv with smaller 3x3 conv and remove the first maxpool\n", "encoder.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)\n", "encoder.maxpool = nn.MaxPool2d(kernel_size=1, stride=1)\n", "\n", "# replace classification fc layer of Resnet to obtain representations from the backbone\n", "encoder.fc = nn.Identity()"]}, {"cell_type": "markdown", "id": "c85b1d8a", "metadata": {"papermill": {"duration": 0.014604, "end_time": "2021-10-25T20:00:50.345738", "exception": false, "start_time": "2021-10-25T20:00:50.331134", "status": "completed"}, "tags": []}, "source": ["### Projection head\n", "\n", "Unlike SimCLR and BYOL, the downstream performance of Barlow Twins greatly benefits from having a larger projection head after the backbone network. The paper utilizes a 3 layer MLP with 8192 hidden dimensions and 8192 as the output dimenion of the projection head. For the purposes of the tutorial, we use a smaller projection head. But, it is imperative to mention here that in practice, Barlow Twins needs to be trained using a bigger projection head as it is highly sensitive to its architecture and output dimensionality."]}, {"cell_type": "code", "execution_count": 8, "id": "62392e1c", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.379426Z", "iopub.status.busy": "2021-10-25T20:00:50.378950Z", "iopub.status.idle": "2021-10-25T20:00:50.380731Z", "shell.execute_reply": "2021-10-25T20:00:50.381114Z"}, "papermill": {"duration": 0.020782, "end_time": "2021-10-25T20:00:50.381232", "exception": false, "start_time": "2021-10-25T20:00:50.360450", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ProjectionHead(nn.Module):\n", " def __init__(self, input_dim=2048, hidden_dim=2048, output_dim=128):\n", " super().__init__()\n", "\n", " self.projection_head = nn.Sequential(\n", " nn.Linear(input_dim, hidden_dim, bias=True),\n", " nn.BatchNorm1d(hidden_dim),\n", " nn.ReLU(),\n", " nn.Linear(hidden_dim, output_dim, bias=False),\n", " )\n", "\n", " def forward(self, x):\n", " return self.projection_head(x)"]}, {"cell_type": "markdown", "id": "3b69afcd", "metadata": {"papermill": {"duration": 0.014461, "end_time": "2021-10-25T20:00:50.410233", "exception": false, "start_time": "2021-10-25T20:00:50.395772", "status": "completed"}, "tags": []}, "source": ["### Learning rate warmup\n", "\n", "For the purposes of this tutorial, we keep things simple and use a linear warmup schedule with Adam optimizer. In our previous experiments we have found that linear warmup part is much more important for the final performance of a model than the cosine decay component of the schedule."]}, {"cell_type": "code", "execution_count": 9, "id": "1df8bf31", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.442982Z", "iopub.status.busy": "2021-10-25T20:00:50.442512Z", "iopub.status.idle": "2021-10-25T20:00:50.444620Z", "shell.execute_reply": "2021-10-25T20:00:50.444217Z"}, "papermill": {"duration": 0.019933, "end_time": "2021-10-25T20:00:50.444716", "exception": false, "start_time": "2021-10-25T20:00:50.424783", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def fn(warmup_steps, step):\n", " if step < warmup_steps:\n", " return float(step) / float(max(1, warmup_steps))\n", " else:\n", " return 1.0\n", "\n", "\n", "def linear_warmup_decay(warmup_steps):\n", " return partial(fn, warmup_steps)"]}, {"cell_type": "markdown", "id": "0f165fdf", "metadata": {"papermill": {"duration": 0.014653, "end_time": "2021-10-25T20:00:50.473995", "exception": false, "start_time": "2021-10-25T20:00:50.459342", "status": "completed"}, "tags": []}, "source": ["### Barlow Twins Lightning Module\n", "\n", "We keep the LightningModule for Barlow Twins neat and simple. It takes in an backbone encoder and initializes the projection head and the loss function. We configure the optimizer and the learning rate scheduler in the ``configure_optimizers`` method."]}, {"cell_type": "code", "execution_count": 10, "id": "ad94b021", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.511807Z", "iopub.status.busy": "2021-10-25T20:00:50.511324Z", "iopub.status.idle": "2021-10-25T20:00:50.513468Z", "shell.execute_reply": "2021-10-25T20:00:50.512978Z"}, "papermill": {"duration": 0.024798, "end_time": "2021-10-25T20:00:50.513565", "exception": false, "start_time": "2021-10-25T20:00:50.488767", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class BarlowTwins(LightningModule):\n", " def __init__(\n", " self,\n", " encoder,\n", " encoder_out_dim,\n", " num_training_samples,\n", " batch_size,\n", " lambda_coeff=5e-3,\n", " z_dim=128,\n", " learning_rate=1e-4,\n", " warmup_epochs=10,\n", " max_epochs=200,\n", " ):\n", " super().__init__()\n", "\n", " self.encoder = encoder\n", " self.projection_head = ProjectionHead(input_dim=encoder_out_dim, hidden_dim=encoder_out_dim, output_dim=z_dim)\n", " self.loss_fn = BarlowTwinsLoss(batch_size=batch_size, lambda_coeff=lambda_coeff, z_dim=z_dim)\n", "\n", " self.learning_rate = learning_rate\n", " self.warmup_epochs = warmup_epochs\n", " self.max_epochs = max_epochs\n", "\n", " self.train_iters_per_epoch = num_training_samples // batch_size\n", "\n", " def forward(self, x):\n", " return self.encoder(x)\n", "\n", " def shared_step(self, batch):\n", " (x1, x2, _), _ = batch\n", "\n", " z1 = self.projection_head(self.encoder(x1))\n", " z2 = self.projection_head(self.encoder(x2))\n", "\n", " return self.loss_fn(z1, z2)\n", "\n", " def training_step(self, batch, batch_idx):\n", " loss = self.shared_step(batch)\n", "\n", " self.log(\"train_loss\", loss.item(), on_step=True, on_epoch=False)\n", " return loss\n", "\n", " def validation_step(self, batch, batch_idx):\n", " loss = self.shared_step(batch)\n", "\n", " self.log(\"val_loss\", loss, on_step=False, on_epoch=True)\n", " return loss\n", "\n", " def configure_optimizers(self):\n", " optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n", "\n", " warmup_steps = self.train_iters_per_epoch * self.warmup_epochs\n", "\n", " scheduler = {\n", " \"scheduler\": torch.optim.lr_scheduler.LambdaLR(\n", " optimizer,\n", " linear_warmup_decay(warmup_steps),\n", " ),\n", " \"interval\": \"step\",\n", " \"frequency\": 1,\n", " }\n", "\n", " return [optimizer], [scheduler]"]}, {"cell_type": "markdown", "id": "5ec4d0aa", "metadata": {"papermill": {"duration": 0.014734, "end_time": "2021-10-25T20:00:50.543024", "exception": false, "start_time": "2021-10-25T20:00:50.528290", "status": "completed"}, "tags": []}, "source": ["### Evaluation\n", "\n", "We define a callback which appends a linear layer on top of the encoder and trains the classification evaluation head in an online manner. We make sure not to backpropagate the gradients back to the encoder while tuning the linear layer. This technique was used in SimCLR as well and they showed that the final downstream classification peformance is pretty much similar to the results on online finetuning as the training progresses."]}, {"cell_type": "code", "execution_count": 11, "id": "f271c5a3", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.582976Z", "iopub.status.busy": "2021-10-25T20:00:50.574339Z", "iopub.status.idle": "2021-10-25T20:00:50.585053Z", "shell.execute_reply": "2021-10-25T20:00:50.584582Z"}, "papermill": {"duration": 0.027403, "end_time": "2021-10-25T20:00:50.585148", "exception": false, "start_time": "2021-10-25T20:00:50.557745", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class OnlineFineTuner(Callback):\n", " def __init__(\n", " self,\n", " encoder_output_dim: int,\n", " num_classes: int,\n", " ) -> None:\n", " super().__init__()\n", "\n", " self.optimizer: torch.optim.Optimizer\n", "\n", " self.encoder_output_dim = encoder_output_dim\n", " self.num_classes = num_classes\n", "\n", " def on_pretrain_routine_start(self, trainer: pl.Trainer, pl_module: pl.LightningModule) -> None:\n", "\n", " # add linear_eval layer and optimizer\n", " pl_module.online_finetuner = nn.Linear(self.encoder_output_dim, self.num_classes).to(pl_module.device)\n", " self.optimizer = torch.optim.Adam(pl_module.online_finetuner.parameters(), lr=1e-4)\n", "\n", " def extract_online_finetuning_view(\n", " self, batch: Sequence, device: Union[str, torch.device]\n", " ) -> Tuple[torch.Tensor, torch.Tensor]:\n", " (_, _, finetune_view), y = batch\n", " finetune_view = finetune_view.to(device)\n", " y = y.to(device)\n", "\n", " return finetune_view, y\n", "\n", " def on_train_batch_end(\n", " self,\n", " trainer: pl.Trainer,\n", " pl_module: pl.LightningModule,\n", " outputs: Sequence,\n", " batch: Sequence,\n", " batch_idx: int,\n", " dataloader_idx: int,\n", " ) -> None:\n", " x, y = self.extract_online_finetuning_view(batch, pl_module.device)\n", "\n", " with torch.no_grad():\n", " feats = pl_module(x)\n", "\n", " feats = feats.detach()\n", " preds = pl_module.online_finetuner(feats)\n", " loss = F.cross_entropy(preds, y)\n", "\n", " loss.backward()\n", " self.optimizer.step()\n", " self.optimizer.zero_grad()\n", "\n", " acc = accuracy(F.softmax(preds, dim=1), y)\n", " pl_module.log(\"online_train_acc\", acc, on_step=True, on_epoch=False)\n", " pl_module.log(\"online_train_loss\", loss, on_step=True, on_epoch=False)\n", "\n", " def on_validation_batch_end(\n", " self,\n", " trainer: pl.Trainer,\n", " pl_module: pl.LightningModule,\n", " outputs: Sequence,\n", " batch: Sequence,\n", " batch_idx: int,\n", " dataloader_idx: int,\n", " ) -> None:\n", " x, y = self.extract_online_finetuning_view(batch, pl_module.device)\n", "\n", " with torch.no_grad():\n", " feats = pl_module(x)\n", "\n", " feats = feats.detach()\n", " preds = pl_module.online_finetuner(feats)\n", " loss = F.cross_entropy(preds, y)\n", "\n", " acc = accuracy(F.softmax(preds, dim=1), y)\n", " pl_module.log(\"online_val_acc\", acc, on_step=False, on_epoch=True, sync_dist=True)\n", " pl_module.log(\"online_val_loss\", loss, on_step=False, on_epoch=True, sync_dist=True)"]}, {"cell_type": "markdown", "id": "c7f6bb76", "metadata": {"papermill": {"duration": 0.014775, "end_time": "2021-10-25T20:00:50.614884", "exception": false, "start_time": "2021-10-25T20:00:50.600109", "status": "completed"}, "tags": []}, "source": ["Finally, we define the trainer for training the model. We pass in the ``train_loader`` and ``val_loader`` we had initialized earlier to the ``fit`` function."]}, {"cell_type": "code", "execution_count": 12, "id": "21965f37", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.649228Z", "iopub.status.busy": "2021-10-25T20:00:50.648753Z", "iopub.status.idle": "2021-10-25T20:00:50.731321Z", "shell.execute_reply": "2021-10-25T20:00:50.730897Z"}, "papermill": {"duration": 0.101688, "end_time": "2021-10-25T20:00:50.731435", "exception": false, "start_time": "2021-10-25T20:00:50.629747", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/home/AzDevOps_azpcontainer/.local/lib/python3.9/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:240: LightningDeprecationWarning: `ModelCheckpoint(every_n_val_epochs)` is deprecated in v1.4 and will be removed in v1.6. Please use `every_n_epochs` instead.\n", " rank_zero_deprecation(\n", "/home/AzDevOps_azpcontainer/.local/lib/python3.9/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:432: UserWarning: ModelCheckpoint(save_last=True, save_top_k=None, monitor=None) is a redundant configuration. You can save the last checkpoint with ModelCheckpoint(save_top_k=None, monitor=None).\n", " rank_zero_warn(\n", "ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["/home/AzDevOps_azpcontainer/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:746: UserWarning: You requested multiple GPUs but did not specify a backend, e.g. `Trainer(accelerator=\"dp\"|\"ddp\"|\"ddp2\")`. Setting `accelerator=\"ddp_spawn\"` for you.\n", " rank_zero_warn(\n", "Using native 16bit precision.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}], "source": ["encoder_out_dim = 512\n", "\n", "model = BarlowTwins(\n", " encoder=encoder,\n", " encoder_out_dim=encoder_out_dim,\n", " num_training_samples=len(train_dataset),\n", " batch_size=batch_size,\n", " z_dim=z_dim,\n", ")\n", "\n", "online_finetuner = OnlineFineTuner(encoder_output_dim=encoder_out_dim, num_classes=10)\n", "checkpoint_callback = ModelCheckpoint(every_n_val_epochs=100, save_top_k=-1, save_last=True)\n", "\n", "trainer = Trainer(\n", " max_epochs=max_epochs,\n", " gpus=torch.cuda.device_count(),\n", " precision=16 if torch.cuda.device_count() > 0 else 32,\n", " callbacks=[online_finetuner, checkpoint_callback],\n", ")\n", "\n", "# uncomment this to train the model\n", "# this is done for the tutorial so that the notebook compiles\n", "# trainer.fit(model, train_loader, val_loader)"]}, {"cell_type": "markdown", "id": "9cb9b57c", "metadata": {"papermill": {"duration": 0.016005, "end_time": "2021-10-25T20:00:50.763854", "exception": false, "start_time": "2021-10-25T20:00:50.747849", "status": "completed"}, "tags": []}, "source": ["### Using the trained encoder for downstream tasks\n", "\n", "Once the encoder is pretrained on CIFAR10, we can use it to get image embeddings and use them further downstream on tasks like classification, detection, segmentation etc.\n", "\n", "In this tutorial, we did not completely train our encoder for 100s of epochs using the Barlow Twins pretraining method. So, we will load the pretrained encoder weights from a checkpoint and show the image embeddings obtained from that.\n", "\n", "To create this checkpoint, the encoder was pretrained for 200 epochs, and obtained a online finetune accuracy of x% on CIFAR-10."]}, {"cell_type": "code", "execution_count": 13, "id": "bc2dfce3", "metadata": {"execution": {"iopub.execute_input": "2021-10-25T20:00:50.800041Z", "iopub.status.busy": "2021-10-25T20:00:50.799564Z", "iopub.status.idle": "2021-10-25T20:00:51.171031Z", "shell.execute_reply": "2021-10-25T20:00:51.170535Z"}, "papermill": {"duration": 0.391254, "end_time": "2021-10-25T20:00:51.171147", "exception": false, "start_time": "2021-10-25T20:00:50.779893", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["torch.Size([4, 512])\n"]}], "source": ["# ckpt_model = torch.load('') # upload checkpoint to aws\n", "# encoder = ckpt_model.encoder\n", "encoder = model.encoder\n", "\n", "downstream_dataset = CIFAR10(root=\".\", train=False, transform=transforms.ToTensor())\n", "dataloader = DataLoader(downstream_dataset, batch_size=4, shuffle=False)\n", "\n", "for batch in dataloader:\n", " img, label = batch\n", " print(encoder(img).shape)\n", " break"]}, {"cell_type": "markdown", "id": "68078df0", "metadata": {"papermill": {"duration": 0.01633, "end_time": "2021-10-25T20:00:51.204343", "exception": false, "start_time": "2021-10-25T20:00:51.188013", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "![Pytorch Lightning](data:image/png;base64,H4sIAAAAAAACA9yd25aiyNbv79dT5K5bNsX51GPVGltRQFFRQBRvegQQnOQkR+HF9v1+so2ZVd3Z1VXf171Iv4uVY1RqYsTfiJ9zzpgRhFH/+Mfj55//a6GJpr1fvoR1mvzrH/98PLwkIAu+fILZpxcP1AB18yQv0TT34JdPoKnzr5eTKAhrtA5hOl5//ePrCx4or9+uP55/+tc/Xl7+GULgPZ6MT1NYgxc3BGUF6y+fmtpH+bcySZRdX0qYjPWyCi1K6MPaDT+9hOOzL5/Cui6qXzAsiOqwcT6/PYBqFKk+u3n6dyRAOza0rL5qNBUs3TyrYVb/XaE3AdRN8sb7XFGfQQqGPAPd327Row1olIIA/qRV2Pdqo9L4cgbd+q8CenHLvKryMgqi7C+L/Xeoxp9vSu/kR0vJ8qxP86b69JJCLwLjlST59BKNdYMyqvsvn6oQMASJbnC2uCww2oZuzC/4wmaA7c+4XJVZccVjYXInVeGSnDUDixkzs5dx2VTAAsJl5QgahdVzJc10aTMAkYjuLvBPNZgFX758eutZVfcJrEII/yom7O0p9mbfpE9DILA05fs8S0COpHyc8kmBpQRW4Dn3s1uNPcT+NYXA3b1TOrFL1TjQZktyUYmKPZLGWb7obiUdUWcAiLzympVR3m+2FFHDCsw3t3ioGWkrWIZyk3ZcmkHprHLURQDkQV9/AIFXR3YZ1+U4mmMoliUBxeMQx2mPpxycJDyehX8E8C5gvIsAv3pROiL49O/ySRaWABNOjD1ENQ2nq4o74vJRXZF8c88QJNxchOJm2zkxaJZbbPbJMjFzmoxmGB9Y1N32N21w0y+Ud9jfFbGhj0A7/JDPa/v/NqSv/UNdlsE9xmegwDk4oDgOkgJFC45LuoAjfOEvsQpHu/v14WMlqOp/F1l8HWIn6GvdmPUivdy6pVfAAQFrUtQ1o/Bh2gYaaa7Su8aeg9NJ2cnkzSjJsE4WhLHh+MFv9mcHic7AMg0mzl0ZX8w+EtkfuonykHYhxULcBR7NkDhF4g5P4C5HOhw70vxL5F6vOGOJf9vSpMHglXDNnQ0lvWPePEloyU7aBuFh1LYOF2d1fekFdcsUSbW9Bs4yCG11XQ2lJIHmdmhbDg6251y5FvP4Y+3FslB2H4nt9z6iBEfRjktAguQcxiMJyLqcz9GEwDIQZ1jyv2P2GuA+ANqquLb3zNsWW1ejC7BdJzyV2KbbE3zckvFN5lLlkpl+KSi6hQvRNmwOmFIuETo9aGRWhRvusoGxRmiRtVscrr0laeJH2dr3nURJUhBI32XHmOYC2uVwODopy7sky/gUJzDfqE0c1RaYfExqahetUnuOzHc5MwiW34Hz8bJe5EJf7ux8cyHSQchhFPk4n8n+zsckLmDa2SCJTZHJN2yJJOl2z2/nQYyso/0HxHS/BCns8vJaobjvEwItsJ7nejzPCwKPc3CM7YASXN9j3pOYxoLUFw01jiORshguSGrpmeWOZlBKeQJ587pXUu5ERth+O6zN8N4lx7jy1+u5ac7dRrxr+6OeUd6iF4tl70aYFEb1bJP+2Kv+FgsHhqCN8rJCPYEY7WAEARmX5zncp1gOCgJDsxzDuD77BxTf/ZpCpllrp1seI9RqdwCWb9z6zFYGxrgRxyCc4fP9Zp0yjCcce5OlTrYe1PbWqMiYTNnq0M5317mVnG9Li2sDqgapubOFmftj1/lbZL6mtQ4vjGkOADzvQ8/lKQZnBYbzSOCQgscT/m9YHs5SuWVU1D8F4Y15bzkG6sfDj1AArOvzI7dSGUAsxV2rEidsLhcKxein1MMhto9ad3vYmZFW4qR4G+zeY03lmrDJZaHgjp8tXKWdY1eClOYWHpGzumteHabui8fMpSiSyAV1lGdYPCa1b8399FKV7l/GArM2KvMsHbNflHV8lyIB8TmuPv3rn9ib3FdfmYjCRKS6Z12Vagi6vmaBqQYDSKXwLnHWfvDPvh65O3F2XV8YIaWBvWAzcr50lrtdza59cOBHvy7X0pKMVJkgGUrN9577wSjcsMmu6LtoQvuQYD3XeQKP5dLKKmReO+cT7d44Zsw8lbm439T8+qjQsV/le70J7p4haQeHXFhqc7KFFWC6ZoPJqnQIdsu7dli1mIypTlMoziVR+8MzeLQw88YBlsBphuUc6k8sPsBPePvC75vV5oBfowUG141y6ObCrWQhD+ouakvYLDuDj3hvQ5AL95wPYGEVeRotbjtHxealn8UMHolXkaQqRu6Qg3vjP9g4fg+sPsGznEvBH5GYzsIVhOqy6i3YbG/iZq8t/WTtFpC5mJuWPptFi+nKSjVEjrmPc6g9I5Ez+6jBul3Gfr+3bjJ+nA8XnssKuLJFhqijwzH5Cyxe05A095oEopH35dPnrx8/8FqQudB79Pat0N83oW8aKEd5Puuy9BMciqlPqlFyaTgbjJA27oytS4Hq1558VztFzTTZ7phFco+V8HqwjrotHan5lVltNXmrNrNobSYipZ3tLXUfZLePh2YpHf6CQ/2UWxaloIZonqFj0TxJJgH8XgyFrMfwgAZPIKmO9kO690BRI/tyZUICyIfzLO17ys11mB1WwpKTgTvfecrlcL0v0yw/nGrFydxusfGBiFvEbSWI9oBn/j5WVX9BKsJsAsmyBP0keg8BlOQJ1xM84QnEDpJkJt4SMZval0sZ6bNwfaO8S5UbQ3nftnHeZG17KaKTM3d3AetG9RZooaKHhxm3Wp3JgyoHh0HW7ZtXS94p5Y84300gVvWZi8J7kZf1JHDvdFAaZwiGGicTH8+POHfnIFE3ZyVu6HW2n7dnV7wzAlh7zIl3ZttkflqHoZB1hOQEJ2Mhe/vK1mx21mQVUC2KrgAjnkzSJiqB6l0+8BJ8gu86IJsU7x71UY/hXILjySfwok+tXeprPT2Ql2OWgZmzc/hBuBKsMFvltCor656t0nrO2xJx6H2MS6kBYHjVIfzW3GqX5Szosd0M3BrnoHh+WdeROGGMcHMPplFZ5uUUar+roGPa5Xg88J7ALk+YniP0Fbu/rwajuVuLrINtYpwuw5V2di6lYcs43DGDf2et/c3tF9os2glibvZep106SKrUOm0QnsGWFiGE9IygjhN89fc7EdU0eL/JoIBkIEdC/wn0UvZQnc6yaenbxJ8Pyk6pqbWWEZKA6Xs1XgLLwIRWtNaOjYlYHSStaOLnZVMe8DUix/d70h3M+NQ5ZWmWnAkWK8BRE8aGkUPaZGML0ddlt8hpHlUnkvyhJCo4gCZd5hnjx8IptvT5iqj1ho9EbYVcZuJF3nrKDilZcBYX5ClyWlzSFmzaLjI5ymzJ9mN87u61cseoDZmsm/DcJ2JIdrvFmDbO53BC/Bs77sMxJkzD+KaB4t4IzoXPmFTS7tCvjyGTHoCeLwQC6TvV8YYqTB8XLSzxdaLY0cd8tg68lo3VPOzppeiDIdJ0Mr9dS0zZtfU+QHi5mPvlZn7d65N8+b3BVEXkwRINSlCEE0n+RHWMk67rk+QzxuS2qEnTq0m81lS9LvVTrVlAj8huns+An+zko87em0NwxHC78++tA4YbLpJhqHZNJPIOO+MJKjjLKcKPlroTl3nl2JNsspiUBD7qow4UHI6lnzGugNJPtpJrDdbMBnsy41IuuCZ7MTxVViUuan0PWlJ0WOe2gWufLwInj09rallB9bzaiKubf0+wlgXl3GsugrFJi+3yMsEWPZiAfpxyJTnwoixAYQIfazxTEP5EEmWBw0GGwp9AdVau08JnwarhlWtp0gzZOr5k2XbfEyv5lm2PUeNcTsXJdW/7Lay2s92RP/R73V/FxEYGIDdW+4oiXXW/tZLI5AyG1CdkOl5UuU1VjcXRAgQQbSPYTRptfiiI4qRDsix4hl8vT7RGKWwWApMSevl0IDJi7spNvqvT4XKjMoTb2mIVZuBiURc1wQeVtlLIHfzS4g1Xj5PYP955a35TObb1/KZd+uEEv/4dQIV6IEreDbt5+UFof6SMEgQLcei5T2DcwdXigNs79noR9+xtgctXjS5EGjY8PGPB9brfl5SyUEgWyXYSr1xuCI/J1JrlbCPLlaB1E3PY9zaWJ4vVsrsaWwanP8RqKzSD3VMIf6+LugQkcYr+84LhdL5sjOWGhFydE9LDerdvVyxPW8OsEYqhCVgrJkOEaoiNJzZ6qNNx1kTKxYcHNamo7O6amEEpSL6qOamOgiUGYB4jwRS+JQjGX3kxCeY3ERQCyoc8zT+BnGUYHmIPERltDaSIHJsNF9jJ64v7bOnlDiOR+la9N6o7UwxyWUj2eg/Nc27da3x3DnCq9mPSuwIyXdaKgbRxYRtgMWGUgl5Uo2GeX9EKuiWsP2KY+pkmyjAkLXj+MzL4ZXm7SMloUcPK2eQXpe38m3CJc2bIEZuY0QSewf1CblXkZJnWnsCjnennN0/TjMvO6Img3qir02K1ijSLwpOSHY5rccK86MFgMkOCdIAgEOwzsqXotl3J8kXO+dlhG7v5Kh1HaLO6WJaesGEtGjY3L/S0kPcyNpOQ0w3ZY1qMeXV4Omirq+CKu+1+CSpOI4eZzua3DK+n8ErzOEKLyL2OufVH2OAP9MY8iQS8Sz4jT2qpMjhKW2kbb7aqTO/0k307HOj1TQDcUc0IgvMto5udyTKXPQPSlp1KURh5wl0OWFzTQ1FHosPAillzxxzIeUu9wSZERH8cb518EsKvEqjjcwDizDOoXa7tLFmF5tklTyt33Ymzizlo24EfcMMmOatxAb5Vz3lbDe65CG7zQcMP7Hl9D7cO5mXSeahLkr1UZ9NMXV92RF9C4BRqENRNCVEXJEnefEgw/IkkyrK04+LCM+7EaNWwxFg5SUu7pmabyKPvluRRGO0J7lHB+cyTKo1rK0xXxIF01jxyrJx47vKdjwthL2pQ2DfL4IQri1OxoMj7NsrJCRmmHyX1YyNqVjTTUL7TQSngui7tP2ONzRUut3Z9lE+0f8eLGqikGe0OdaJV1PV2Tra1T0nsHotZ5SwGGjjNWjM+cIyUAN9zFEvwjm3MFrPUVVbp9cJdW5uanyfExmAcQktY5B9hjt9rPe4IjhMd5xlZeEJw86NuiljWJ9FOw8P7cZ3E8QDl+15mF+vdRuV3Pm8Tc3x/MkDN+DuikDpOGMQGYIKhkyKhGbVx6Oz46LPyeeUR3QTvjrLqsXGrQm8NLCetZfxRCRU4BroE84wVtkwKbsVgd8EOChxOXRCXzFahXPRz3KJ3u5lXYjOE4CyKXp684rw4ck23r653/+D73k6Lz9ZCKXb+DFwCMbjiteJdo2yCL0dZG9Vg8hrvOxlUcBmeBIB7xroujuwzh4+1Ob30jvR5cdWq+dy20g0pGVooG0GXnxYF3rcdJ8NBxizcXQTObHOgobEISzoIr+n+UvKMsDJDhYpXkshMyLbjJi3QOp9C7qsEivu0z/vCM1bSqJsgBmZtJQfWv84PelB01067IFStxLV809ZwtkKsgnYcNwrIkEh1IdLXMpmHYrLjEdsMTzNtfo8Gl9exvrzsnfvOmEAtAU3mhq+3pz4iBP5ADvUg8GmSf4YHd9Xx0lNHUzdCtiBuK/e2yxWthAvlAtarvVvEsDP5zDrGIdIyhZPxbWWqjd1ljovg2wOLCb1jeI7A7yl67vfhlbPMKTSjbMxGQjDtzvTvKqhLugwhPGWTVpvWJlPiQevb+m2/hzN1oZGxWF27Q3ADPbc+B4MrH+qZOs6c5aJOJepQaMy2UX0w7OmZo92cQN5Qlhf6RsXSpyWNT9kJ8ej1ZGoOZB2Ggs+Y1XEyZ+1CPV/dfZUFqhkcjWO9VVbsEb8i1RAs/O6yWrpqc1ArRMXPF9m3orap5vgu0XqOH6K7LxkrbqcMaatEV4gwzV/Z1PYzXimsy8hFq9HN3Eelj3Dfn2mi0GWhx1DP4BrVnpCVmmaTNsCOrKqWIDLCc9M1kdYfJFa2FbDakIeSvsxrXc3vwukkJ42SB9eh2a63bMSt7bbI4bbf36zGW7ery2lCJpPldeR/rTAaVVWjfu42k0bkn0iiPPA4zhP+vNluOlXsboGarvTU5KBZSUand6tUnjvNlujt1a2ImNGU79B0DMuywB4PF014mouIbOFk0dUHLsqLWXFH1Aazjcy+RQV1didExryF5WP5fwrGbxqo7xMMA5xnZDWzlmU2pDLj6W6A65Asr+7t5hJ7aUiKzVF02kJJhet+VfBgS5dH8YbP8pply6aJ2Ft7NDlCubfzq5GuZ0M4l1gSV0JkgpcXJUyjJkXLJsvgtAXt76RQnPShI5DPmOWplWBqR9xpoK1IcQmZA6kLl3NHH1YX7+rPYioOkMA+t1HbO+H8sLYEuSvO3YVQGuEUX3R4PHomrbg45igHisRS3jMm+PTY9QpmLkS/fTlyGsY/aKEkcH2Gop6x8kpr1GyrUlKq5uYideadJK8CpNydqIqjkrV02G2MuUQ47lDFuyriB5OsGbo1O5twtzLh5Xd2VxNEp8shbCOiu2tSeJrgxUWZ+9F4rRgnaR8x3vxID329cYU/5Y5rGOSGujGTDb+iVqcSMzfaTjwiO/rqibVXKmJZNh69U8NATzal2eHn4z68zcHCuxwjyRgWB73DMulAXDvSgmaKBwKSTMh5vvb/IxDyLA4IknzGfZXwkmFg5m8SREXYVOGvNtSdWb93XV1uMMG5kbNYwxZdxQxiPZgZfT5Xgovwp/zsnCQX38byrV222GWmxcc7nHl7St9NscImqUb3a+rwcSvu62aRDzDGn8ui4yRV8H32GXSLTvZJZXPTKsPkzxmb7gp6EcAdTZ6FW7/cuvlFX5tEUkAQ14dba0YH7wZOkGkKskGO3XC9ijjXl3iFY8rK2yNGgXgTRpwSAi+F4zTY/Qiof1ZDAYcTgu89g2UZJsCLWFymT7pYS9f1bFmu5hS5mB9qdr7aNoOk63i3Zvns5N7gnUZE5uywZFlpxs5Lmj11bYDRn0xKNjfiwti0TLya4N8l9L8m09N2j77XQQEkBAZwz8jFwyE8rZNNR3WnlG6wQVjmNgsTxGD3V55BmHLVn+UDL7trol/WVw5f32ivHcBpO2aJx1WoyVk2O0U1Qh1bhELmNEXBKXPCMWPJ63EunESFk4PSm7xR6oeCKM9RPMOSzyCKbdRLlVBLfav2C3DU+1Rfz9Z7hsflZt1SSK4sSTrkG77tKCG5xJasn5sLbi5D1h3uhVo6qwHyt2O/mHUyL2YbXKKsCZnQmLwUeVZFLUSbzIPlY9acgXYa0x9Kor7rkILgPGOPuBFutHwDqkWRh/CSpNV6W/VRNQ9Pg6VSLOu4wYaZ31TjYFkktJr5tgSr2qOC1dE7E9qWwxU7XDv20cFiwdAP5vGyUSaMSG8p9Thm5E0xKbn8gxBKjyO6C8EzMiFcsJ1iQcjLY2W3cOcVje8Vw/YoL3b1vpl3LY5huKJdtoUG5cVhq2/WrPz4XmmIH0u4uwn5zKqH3rUKT5THIuPQH/ITPL2CZQQrtAbOtHTovQ7qUR5LOOAZ9wGd2NjzRr0Fgbfc+RR27nn/6hzh3uDandMhZFgEAV5nkBtSDEYD5RDpFj/XoWQa/j3UhIMiLfhOw0JNdJ1M09sdtZvEb5wcv+7rBonbJOAxUHzA8P1fyKIspGh+nIo/wzrvvC03lxYG2SIZpGitmasr2Z9u5HlbqflgtTPsQJJlqVtzbgF2ol8xuzMnBqttvE9wAZcM1vfAfjdf0/q8d8U+ufgTomaVNMEkjGP90RoJn+CJZ3yrTeadaLOQmKOhbqk2pJBrDIbjWRaJhMwS8t4mnNSE3VJn+gTxlMtwutJIT90volerZXlcxGdpg1923UViSZKxdoMhaxPi4diHNw/89sXSSfC+F0PHCS8LSfIZ91U1qkUERVKQHSMR/fWmKwXfECSdi3rJ0Yd9vw96Td1BMIfivLymWL0RQbP0Y3HBKoqTb3GCaLSLtN0jbDZTqv0xd9spJGvgXqGHgjGRnn5n4c9qKOVwjg/9Z8TIzXx9JdTaWIeqTuAzik2r67xuh97YG36gtpdZMXC2vjzd3RPe5QZer7G5KBxISU2JxBGOWKlg1VDhueLPNtQMwYmEmjCzee199XWPw+MrHFGATl2T/JkmSroEydLeM7gulgd9jzXDstg0V1xO1SFFHLIRnOsZLzngFHgViIhxNMuddiurjkdEWdxedL1cOIKX9xwIDOdck/tsfU1qn2iWfC5P4FqDYPoGlN9EUNylaYKgnrEuiQRht1IS+pSbKxX6UlYZOK5LfGe7CLW9qTRchoRe1SzP8MS6N52U0wj9PqzjeEjOlXsQeQIbEg6/d6GLGLGnw3zK/LAOSwg/v70wCd47HdQHOM/hT9mJlxlgldptL0qK7THSOpRF2+Q746AaCU4OfQXZgPCbsPG6gPFN75I7Mz4rOR7r9Yw71BFumpt5Oyw8VRljQqrEvb+fwi+atCt5rI4KHsk+ztJ5Aq2Tylvnjkr84wGz9BM+JIG6D93t8abgjZnNMXWA++Peu4RiiinF3T+fFdWU25gRcQ3eOWRwVltyKQ7HjVObokk3AdVPyGLqHFTTfPQhgDI8eNx/eYZ/Ymqfnc7bWQNMFo4eWmzMaDf0A2moG1rhV3bUq9soXTjNfUAyP+zPMWEZi4FU70R5kZDhkIGruiTPVNg5K0w1O2w35ZyDuoNj73MvatJJ3H6XGWfFwAXcU75p6beuptmgEO/760I/nAbydBjK24Yt9fDM871yknjOd4Rz1pPSjU7KCNd2wJI23HlhmJHZN7RmFo91297fWkUgHwGYstbQZC2MJh0N8aaActDncPiUOyxSxAi7Gc9jdUuW2LAcoHHMSsnkBDkm5wkczlrvcdIpOudBmCG6LA+5YYrRcrkk771Ne1zM27Rc6JXH7BFo8aSymrL6+nqs6JhU1E2FVo2TTtvp/mc1lGBJyHnUM/ISlod5Bw5Vdr7RhNtad3qZ5cy9NNL1dTbHeHw2mzfO5mItinjmqoPAyT234jdn/4rwKxMYAaFTVJaHFAZJxfW91SooJ4wObV5H2aR525sCCh2XA7z7lB1Mhyw+z6WggauEcunt2pJl57Y1QyU4upVm1ac91Vab2y3WvXoPoDDH/HMsEFpHHvtBkyiH5BPJX0RULPblGsFzF2Mm+GwHnSBBO1BOGlh/V0FdAhcgxfx5rvYBRyvpAoxEuOvvjcp6txPDzg9DV67kgze/qjfAt7vjUo4XtSKNsY3qtOGY0WtbOBCIs0QMxql9ZLvSOiW5RalFFzMc2e4WH3y00mNDcBXV+euyFM15HCDx7wPZ5C00lMS6EXuD/qptrPnxOp9vY4vnzTWfIRtVOS5Xt2VoaUSAj3HtpDebZKf49cGaqRQxSwNLwpljSHEnH6PjemFmW+NcfjAHL/L9CuUdjuAA9/2a8D++HTKdgcdBl48J1ONYlk8vX48u/vKpi7w6/OLBNnIh+vrHp29GVEd1Av9V9CNjN3w75TobvRZL8iD/XGTBC6hf0jFxgeXL//u/L/vefJTb/FbsTxUfpeSoVhrnn9ib9rtTsN8a6MG3lo9A3rXRDOHLq0oHH7+/vdVLV44Ax3f38/LlcYoqOv4xPk8fx0a9zFYvjw0FYCz4+cVwQQJf+rwpXx6nXSTV/37J8vqlHoWdPEpgWSSghp9f0JeP6+6n9+dFvp1C+NqaH334eQGzt1ffEUDuafLtuMJ3JT6/Xn4F+OXTG8/vT8T2kyby0Mh9UPzhcYcP+3kt9Cjz6NwPBV8/mKIc37p8OIXv/DK2+dfIe/fREDRO0DzPChRNsQzJv6v49ok+ejlGwrrJHmeMFMW7uuNfjxBJ0BxHcayAM7+fsflOoO6iemT+y+uR4L+MnvFO4VuvfosF/X99cjjB8SzJciSOcQLhCTSPP76uBlGCgA7KM4BDBZ7jOQcnCA73Xo+M/UFLxneC7xrxf97e62elXVC+B1Y1aQrK/tcElAH89bW1P6v5+om8q/qBpvmTd/yPcr8/GtPvZpwHv3zF/j9uR39uxC8gqf8jUP+ggw8/+fVhYO86+DW8/LjCIy6+K5s78esx/D8u+z/gHH94w6ZMfmAx7+Lpv638H+l1b/8dwuPxDyPTW9LyVw/hfUtFxn//HwAA//8AYwKc/TxtZXRhIG5hbWU9InJlcXVlc3QtaWQiIGNvbnRlbnQ9IjA0MDI6Mjk0Mzo3RjMzNzoxMzg3QjU6NjE3NzBDRTIiIGRhdGEtcGpheC10cmFuc2llbnQ9InRydWUiLz48bWV0YSBuYW1lPSJodG1sLXNhZmUtbm9uY2UiIGNvbnRlbnQ9ImYwZTkxYTdmODhkYjI4Njc5N2MyNzlmYmQxZTQ0NDRjMmU5NTU2MGExNzBkZjc2OGQyZmU1ZDRhYzFlZDliNjciIGRhdGEtcGpheC10cmFuc2llbnQ9InRydWUiLz48bWV0YSBuYW1lPSJ2aXNpdG9yLXBheWxvYWQiIGNvbnRlbnQ9ImV5SnlaV1psY25KbGNpSTZJaUlzSW5KbGNYVmxjM1JmYVdRaU9pSXdOREF5T2pJNU5ETTZOMFl6TXpjNk1UTTROMEkxT2pZeE56Y3dRMFV5SWl3aWRtbHphWFJ2Y2w5cFpDSTZJall4TVRNME1qVTFPVEUwTWpFMU56QXlOelFpTENKeVpXZHBiMjVmWldSblpTSTZJbWxoWkNJc0luSmxaMmx2Ymw5eVpXNWtaWElpT2lKcFlXUWlmUT09IiBkYXRhLXBqYXgtdHJhbnNpZW50PSJ0cnVlIi8+PG1ldGEgbmFtZT0idmlzaXRvci1obWFjIiBjb250ZW50PSIwNGNlNjc1ZmE2OGRlYTk2NzVlOTRhYjRlZWRmNTY3ZWFmOWNjZDQ5MWJlYzg1NTkzMTQ3ZWE2MDVmMWZjOTM4IiBkYXRhLXBqYXgtdHJhbnNpZW50PSJ0cnVlIi8+7D1rd9s2lt/zK7jqtrN7Gkp8P5LYcxzn5YkTe+KknXZmDhckIYkxRbJ82FZ6+t/3XoAUKUoyKUvOo2emGZMEgYuLi/smAD14IMD/nsxoToSIzOjBYBpf0dQjqS9mhfuRermYk8lA8OIop1F+MEhpEmdBHqfzR7JpGYphKtJA8ElOxOQjuRHzlERZAFUPHzx4sAx6EuTTwhUv6dyNWQfTOM29Is/Wgn+YxUXqUdGLfbq2g4NBnhbwanSIHbU7y2gIyFNfDIPociBckbCgHLzDAW9Cuk2QSRxPQioCUlQE0gTjwCN5EEcNpD35sngmvlZenfz0wlD1cJ542c/n8TPtMghjXfx45Fw/fTF/rxVvfhkctonSDf71e32SWVPp+oqQyenro59+/s1yXfr2Ovr119dyKv/j7/PsH+pNkZ5+uAv4Xz9Nf5o/f3HtmtcqlcTi7H2Yz6y/Zd6l8kK/yH8Kps8kenOtjLOju4B/+Y9Mfx1/+HD59jgmR79G5vX5WzGX5PP5dWK/UbPnH6PcUZ1ffz73APyDJvTYy+NwngdeJk7jLG9SPA5xeuN0yJmKJMnQi2fIDBsAQA0x8BsgeMNbWtArqCcWadhoNM3zJHs0Gm3of1SyOL3JaRqRcOSm8XVGU4fBqli12R+BWry7MF4h3eiHMH9cQHsRK/8wyR+zEmTiusQNY3cEonTdS0ja/cdJHsyCTzSci9h6HIS0gcDvP/xWxPljmNkMMOMPjwR+1fjlYfmYAkVikOWq0j//Xb3J5wn1jwof0PHomvckiqM54nByXr1EnKvXSRqjGjrxl7uXDVM1TUOSTWkZkSuSBsQN1/U0piQvUvoiJJM1b+lNAhw8g4HXL0sCZDk0zJYReFdEURBNljsn5TBP/DUdMMxwilfg34Jz0Bq3ImmqZaigZpTlrkGtLtfEWYRZ4Q9/PBTu1Jel2bqp2mZHX3kKpEXSlb3dBtK0FcXUFK0D5AREJ586kyn1nDhiFgPI7fBZQhovtw/JnKZtJoHeDNPSLN3Wl2uDfIxBVx2FldS1ZwSGEuTzNfDWE4RG/tn4HYkmtGogS5JUE30juPVzuQpO7QeuL3b6fsEZ+wVn7RccToVUs+Q4Buvv/7QijL//Uff5VQm8LlmKquiy+hkEXldVkE3J0jv62kLgdVnRVdmy5X4CnxRh6KT0t4JmuUM8Rj4HjMAsyXtJvK7YlqIoUkuk7izxGyhyV5naMJl/Pq41ZFvXJPnzcK1t6aokK119bcG1gL6sS7LUJQj+HBypwHMygJz1YlBD1sFpkcyW3ro7g64f/F0ZdMO8/QkZFDxIS5Zk6zMwqGFJtiqZur1HBtUV3ZSsTtes9J+dCY1oFvRkUVMBhwMckz2x6AZS35lF+4Lry6L8xlJV3aAgm4bhq6rhy5akSoomqxb1XMNSb52qqg/ZoqrsWcR2wcuVVVXRDAMusqEQjRDJvxXIHzWuZCVg+n3Bjn6whD6//vNf/Bqn/6ow+r28m5Hcm/6rql7e0Bswrouq5Q0Ghe2K/w0xouMXs9ncIXmeBm6R03Y7jPHa7bwiy+PZ5jYsJXNLZ833f/x7mcptcWjiWBJuuQELl5eanNWR70sQjZTk1BeqMFUAVhGeEu/yGsKOTDgG7wNYxg1C4Me2aILfUiRrtEDmX75uC+PP7z2DXEcvi1+e+S9t6/iXdy6J2lx8FaRxhEyx0hxk2S+8WvIWzFLReIVb2oSSDXCONFNSzS7DhnkHp2pey+QKQFPWVAjDFb0roMsSAvPTCc+STd0wQPK6dGWcTkgUfGKi7CRhRcfbQFsyqDVZb6cMVkAHmRPGkwn1naAbqi2BNtJlS+lCeELjLmCKBKPXDEXVu7zl0kOm/nGRpsCynYRVJNPSYebldiy8xuAVYPHmDmZe28zuxvmLIMwhBK9M9iNhTMJska4hHmveO13DMmMrbFvnYtZ6AKtMaMvAOLKmdZFtOvfT2PHCwLsc+iSbsrzCMKfEmwK753EcuvGN4+WkTc5tMbJksBwwaqnLJ80KdxbkwyY3Z5hLTEKaUycLJpFTJLtjo+mWrahWl7cQxc6MgjLxhmDsvUuQAEzqBT514rFTJwt3x8e0TF3XjU7ni80UEKeZA4K58i7DIMuHxPcdTIfujo8l6ToY6y7+KWer/kzhBLMkTnOYMjTldA+IGDJoZ6UfYaY0TIYhJWnkzOKUOsSF6XIa2DGsFvZiJ7xYlKF1qc4cw3Ym1EM/dqI4d8CQgBp12rZrF1xs2TZUQ+uyOJxGdJbkcwdZiH39AdkesvT4GAS+ptQekNINWTGtfhO3BinOSPtESZElW9M1u8swdQhZAB4JqKIZnbk0zfaAlSwZIGp3JhSXtP0SSjcV4PAdCVUkPiIGnI7fUXbHSkHPyuzMxXVg5QfZLMj2MG+qZqmKYhnbGzSWx7titwvTtjs+uqKiSetvYBsssy91DTG4rsu62mVXG0hwOXKYVJUk2Rcutq4oitGlotdNUMmzJQvvjIpuaODCG0oX614F9HrohSTL0hhiRf7R3mlitgdUwKKCX93FtlyKGrig5wV2i5XvjoUJswMxUtfctAhSh6079a0pEoQo/fRI3XmpZ/c7GxZm1DV5W1zuhTMsWzNUoM3d6LIo2B0RW1YVFRyauyEyDlLwufaKjgG6VenyhdvoTMCrwk82Ht2DtQHDJ6uytq0G2c7f5Tcbv0mvfAW4BV9LlQ2IITqtUZEAmSCcKuPktYmLbYllS4YG//SeUoWabfeQ0pZMMHuq0RlS8k5xnhaDdoMwRCclIZOd1RvgYaq6bnWGbmXQz70hhxNh6cPiTjjYwK622s+nxS8K4BE5GVl8PtihbzBuEA1JnUmtqu9ZQtJl/tsDDrps2JIq9+OFtgzsId1ig/DKtip3JuFKZzmh0aL77DrIMf2zMw46mFlVsjp1ViUR4IrW87CfoAG/3dkmxFc9VQEbej0RuPDrZndxMFVJti2rM7tTeqLTeEZRFTh70kzgY1iqZCj9uHHfvcPAdVOWpJ7mk9twAE/TJA0y6uRpQFa+Nm6JhYJf73TVUuyePsWqq+fklOzqSwAaYJh0VemZggiipMhXaOGM43QPmBggnZLUmY+tVXQQFdQBAZnuiRQG+JsmxIp3npFxSndVEICGZRiqYej9TFVpJityDPGr0B6CEkADnEzNsnumy9polNQp9dbOyMj4rds0tvKfamTojAS7C6wsyxio9WSPNg4JOMDXcbryuW57NFBmNUCmAw3md4OQenvy46BnMJ8Gpiz7GY19swBIBDCl3bWqtuy9cmESMsdO9qOiZBNpoMrbUWCPMbEiYf5I0w2zn9lIKbiTAN2nPnMkMV/CORP9bK8AP3seF6kTUerv6mQibqamgVnrmWtj+xHm4FgFM5LOHfY9ez+iqqgahqedibZOIuFX1tra7QEv/HhuWjtP3l7snaLalmYqcuc69S5sFrZ4L/ZP0UHFmBArdKDVRmjnjnGtN/jnar/ZqVSrR0KvCAluNZHL1PHumOiGYdtGZ65iMyb6nr4G8QX6gInaz1teh4u0P1xM3M+mqf04lvumPplnTop6pV7FeMe00+2Yqbpp9NQ4qHtxG59TbnjZA2FsxTLtniHNmkkq14LtbaIsWZZMqScD36JaSJKEc6da/uHSiI6DfB/4qZJlq53OxEZ6bSlfGxiqkcdcXix6K+qaYql2z2/rjQ+OOcl2/UzC93TYhmL1Y7TOic3ywkfvbLuJvYN4WoYpq6bS7xtTi2iOC36c76XFzL03/DBBKXc7lsvpuTze2jm5C2q4jRhUWz+dS8JLxKu4v6m0NVnTpJ5Z7SqbfLf0zT7F1rZNgNG5wLz6+u47JAwZ+31+FaNL4DxDzN2pYha4tiTm3iZfBxkxcPV7T/VDfMePve2w2bAdaisCyuBIgsx0oVntnqIRblTomRu4C9Us25LszoUhyyLD3SeXRNEePFpd1TXwJHt+soTuC9raW7bAay9fhAAjmEsJl1rfHaOM5kXiXMfp5TiMr3fistsQtXVwpZSeyfKl3Xm7INj3yyrmhMDvlVS5n0fC9mE5LJF8bwhpiipLaufWsJLfKeZpwDBwzKKifwh3B9QMxbYlCPD6cV10Z9Ruw8EED06yjX7ORlK4cK1WhzppTPwZ6fkRZntVZRqyokl25zKc1rfiIr2i86101V1QA/Ul9VVhObmkJV5b8lJvK2PaOjjiSs+gj6TeNLiiDi45LDFawE4pfutsHyBhV4m8P9jxFILw5L9EUXgfC0nK1g8LmFUWxiHJpmD4Hwr5lAr1CnThbxcCSysKeSy4VAiBcagvBBHW44e4TME+Hwo5mcD7cZxSBuHZ2RshxZAhzQRRZN1mXhokueClcZbFaTAJIjyXAw+liItsIPh0TNODAbsMoIOcTtIgnx8MsikBNSv+LPufPvxNPjn6cPxaOsv1T+cnVzE5eeqR83e/TI9/fP1Sf3b91Dy7ntFX7/S/n5Bf3o6j3y4v7B+l5x+K6Ytffzs+tszEeuum797+enJ6/Go0enp9cDAQcFcVoAKhTHmsyugjuSIc3YGQpV59IAk/eKQ6jSQDzcw2EYz47ahxzIfu6rhgQhp+zAaHT0Yc2iE/GYSfftM+ECjL8W7l6BR24Mrhynk57LySMAYqNloM+GlA7brIquyQnq5OVo/lKX0asUxxNM8Qevru6O3xK+f83dn758fvT87eOu8+nD53fn7+9NXZ2euHb47evX7+/vz06Pi5c/787bOTty+dk7cX749OT4+w9sXDFydQ/cP56dnRM+f4w7uLs3fO+dnFCb4cNE5wQdqLYAqDq4PBDT93pTwppYGMRmyDGhoE1ZJMLculpqFijt9SXEunxNJkAp6TZ1vjsWn6su0SVac6aA3ZtryxITeO3FnTn5cla/q0Aa5CLcmQiGf7LGfujolsq7ZsuaZEfd0irg+Bjqoaiks0WSGuCz6lqxLLl12jq89sTZ+qL0myCbZHgn6JRlSDGKZrGB6EtvhZQ7UMa2wqnq54qm/4mPMDjMa+K43NseIrt/f5cV2Xkmu6EjHBySGuqYyJZVGEDMDoeGzLLgVnnkrKeOzJEGW6hmyYuqWOLVMCI2COTdYl56yVY41Evhp/LT+OzufvY1B3p8FkmmPWb5TMcywQw6pEgNpCSzb7tUQRHqwcE9Q4lshH1Y6UEMsdeg0cYXiWYevyLacatZq3BbWNYC9Izd0wTXQaB4RtByW6jm/BaZVo28LnTkeji+qQpK2GmuFXv8sGFLYRbuux0hw9aPB84nxf5GuCvAstH1QKFw9PA3MZHgw8tIhghsKBME3peMXy9ONuflrVjGQ5TUcQOmcjfhjbyMGjAQJvFMzA6GcjYMt4mCAyvU6WK4/ZEhFItuHILpIEwwau+F3wiuS0OqJrxNquSl4FmqZpnN4VNm+8GXjDPIM3BaB3664G53BwywjUkwoTcSkGHmrVtZO64k4kAbjAvogMCB7JMLtiRwOGMfhH30nsf1yP112QkB2FllOB98OW6h4MQKGzQ+PG5IqXc1+HTf+ITXw/hMr2WXXDmKaFQv+OYTw/3sw2sXh350iP1hl24HPO8BBDoFFjJr+TqaIqaIGadVktMfOwSaMykx4QBNQ1o5L5m3MYBWOKp+NxpEdVATh4bNzo1p6Vbi3ofBFiPkyKB6CsSoYYMU+Z3bqxP69IxbdAi3GRCzS6EusNfMwzB4WTJTB4cPr5M4o2eKX5PISh4FoU8TolySMB09uXIhY8Lt3FUrn4wVXVFVNdXIuFoAcAJMwTYgXCgVAgohHKxtCQlEP9DmQ2zcV4LJa0WsxyciMqQjIXNc6dojsRg2gMVj1CN4KWpbjWUbyeAjsIeICeCP2PY6/IsPPsMkjEPF5APryAAow1yoInI7LAJ0tIxBUVrooRMW8ZRw27UiGVxhMgWsbVGQtYUuyq8Si6JBXOy3rCgirj4AZim+vAz6fiuAjDyu1mHVcUx7ePBOn7x4MeuFR9iDD4mbAOM4bKWuqxsAF6BublV06GkhoLR78OJ8qnJ3xCKxRe8emNQ18oZ7rBcEAWHwQjCDM2AySAyFt4xkuEVW4ZazjbykBIY6QFj9S5Hmhw2QKSeBMKvhhOxHFIbwT8w+gAfbFEvpA0mLuidQOO32j3scjyYDwXXbC5lEarwBZsyxi3hDBLRe0WQzoQ8IwSMSQuSvircoFqOa8TIrKQ+2DwP6eMYAIQ7H8FTs6HwiRGLq0WtT5kavcR2lMmgjPUIoeNeXkCOgvojyoGnHil7Hka+KAjKp5BTnoa34DHLUiCbMA/KOPuOHgpQ3A3GfPx9t28B9YDkRLKq4g4iXz0K4JZUT8h+VQYB2EopgXOMGYIYh+8Jf9g8MYSpGN1qMMF/itvLI/daoIyVGzBGOqqoA81UzCHuj3UhhJcdXEo84sKLeDBFoeSLA4thV3loWaLCtwNVROuuiryAgOq2RpUgauChRZ/hmZACyhQoEAGqAZcdQ4LWg8NFW8FeShZiKIMjaHN0FTwFsstE/4BvirUNQBDbM2AwQsd8YEieWhCgSJCFezS5jfq0NZxBFBRxYowRMBGYVhIrP5QNRCAgo9QrjBSDQ3oA7qGXhAThCgjYirUVwAT/AujwEmH9jBgQRHwhYzEAGiMuDjw6jrUNHgJ/wGPDG3sTMDOAPuhbmA9GSGy+rLO5ooNCUatDk0dx6ILOBaYsqGiw+wNTTaHMBUcCyQKo6HNiQn94jgU6BJmF2ZZh7k8suCVKvC/ME4J0AF2ELWhhoTTLZH99wn1GPIV02NXk4acom5vysiS7IPaAFcYbAjoS+DryEPnpvkgglWa0LzS1xAJJqKyJHTLqvHJCOA3OtykajZolZZRHPEv4H+FB9xYcXABjz8WyQ9kljzGsjD2Dri+/ZHr2x9BfSzeos44+F558b16vDh49nv1OS9ZHDzLS9DgwwVNJ2vPXfkSOItHBi0khXpYQQQuDHMavEuhJupYX7XNUSwWmLDDBoILSowuDBN7AEsxJkWYg/LHej6a/tL+ywJoWhX/ZDNRX0WHKSt2JkilU6skKmYgHfQNy6Qlv5AC/Dnwnbxy4269N/QhvyRkjuazbFQCq47LYscgLJbKliDRGy0N4BKkpaC6bBGB7X9YY7KK38XJy7fOh/MlQDylCQhEEwfiiaX6nyWIW8Jm6VghNqA//hiszIM4nREPggdCFVOhiusSkyqyr2hU9zSTUtc2TG8sUcO3xopk4vlcOrWIq8vU1C1dJ5qhS57fnvC20AgCSscPkZslj4ukLVFtLeAWeQ5mq2ma34MEhcCh4P9PytOb2Wt6A34RsOIiJ8CjDA6hj41s+D+lNnHziJ1l3hCWWS7y/OCDhfdVoopGfY0hr+y8om1h1LHy9kY9B21E0YfM7mbTUbeb+hH8H/6hFpdlvFGmaE7IoliW0Szor+R2Zdb800wSVmGYvWGYCOMNayQrjQYSa9AGI0liBWbVtrBU45MR54DDlupvW4GmBeB+3RsaFUJ9KzZ95VaYkMeJKAkpzjRc3Rg6nMENn3tujqoGwEgLF7rlD3f6tbwE1FBcB2arhquh2ZtQaeSXmlzkCNaRRla4OYhUIqpNz/nrkL01wra1qCngNWl7FbWbknjsIaN4QCBJ572kTEf3E/+QJSGRDEE6BSdRlsCzDcEdU0T80xQZVgn/nMoqu1VYPaFdT2xUBIBYNxTX1JR4TZHXxH4BJEMNnHfTalaUWKV+IrbiWwG7LIIwlA/wFFT8A2wqCTNX1PEPPizHYC/B5JFwOXIqwiWnkPE7fv4UWUTOuH7VTQuDulHD+akFj8kL5jwWj0tStyw15RNUWsg5+DrsD5SpAvUnlLVH3bAubsYu6s7aQr/J+xRWxoXCX0bnK3qrkiahkipcmxyS+eIZgu3aXa7TGy34WTHDjRxr4FdvGkXMUsK0SugFqhWdJPQl2XCXiN/0Rdf2LQg/T+fCyyB/Vbh/XfueqQGU8ORmIMzL67KJ1QRrINzMwkdZQsBNHiQ47PQK5Bql82DA+KUaHZNlb0qv4OLH1xAkX+aTegKJm8VhAY7jOlW+YQiVLmDW9aEcGkPlIUrvQ7mWpLWtloOjpRec8GtfNm1BGid8EGjFFnaj8tf5PHFpdEUNVD8MRlvDrc3BLigQ0jG2ZBeUHW3D8OsAqfHJuEzMzSFaCKeYaQINnoFZhWgiJQsmOQVmEsVyI9FyODLVhafFLGH8Jorsx3O2SdO8qFA5rO7KpF6JWQ07m8/cOATixSQXmXEXmEcVxemMhKUFGE/EWYHnmCYpms8fUpKmj8skXSNh2SJMrcda2gujMZgRtfH1fi2AWqct6RtgqwXNZ7HLf1pkG4ovbNkyzcdraH74hnVwr9R7MgqDTaLVlw4V743KtX73SJEj3sO3QxI8b5TpxvukyvGik2+HMIDuJUbU90iW87KLb4coMLoC12HdI1Euyi6+HaKgCIm43I5ej+5ZiATezbdDG7bGOrtPspywHr4hirC1jOS+DdFJo5vPQJz1rsqTUREebni12QGCWFBqBFl5nAgJczPVzU52jxlg3xZxQUZfoi8R+f7c0YsKr0Me6ywKvhhP96Em31zKlv/EabCFmfxMVD0u8RMq/A7bJV+Gulwk1pUvkpKtYh6vr+Q01vTQmLetMhbrEguLicZjCRaT2w72l2aMBf2bY/wtJu899nmIf9cEUJ9t5PX+ws89/ud1z4f1/UZa/Cfd9adIdz2/SUC30P+kuuoXVaprH7ku8CEUzKx8hqzXrb7Nbl4M5UzyFaTRSnYFDcVvylTtF/JabnM1p1q9GG2plyYC7H4WRzH60zBNyiYX9BR/+UEgkc9WJfKfAHoymmqfnxcArcC7xyBiG4vNUTnk1y8YjfXyXfkPSd9vBLaNr9rA57Dx8FUHADlu6GNbF74CCr6vkDms7r5m2lULgELiDpcWoH4FlGTaDekHyJW6Dp9OyZdS7FtRFI8Y5uuihpMi8O/x28cWJD0DpASOlcCwAilfLfvT283jOIpAsQl4uKsQ51OaZnc1m9IuZnPN8js8VGR2nx/K3k+p8A46efMcNxrgT4N+C9LUoBD/xbivQpyec1QO+fXbouOsiO7168lWTkeJDf7sZjFD+Vwq+BYIS/2iXBTc2Lj+NRC3DIWeV/gtcruLkm+BvLjTK9tIWhZKf1n6XiCCfOsWJgyXShNe+kXN6rryu6R5P0e2EzcE0TwJiffZ051vGl0fNh7+k/D8cyc8z/kZnN9uwvMbyXhqZcbznpf4lUeq1lbC/VKpyZKxBofn+EM797u677Nn/Uoyf1euKBD5jwsFWWO//Rf2K/G3jgT2G0fMq6wfvwWnpxn0LD69jcpfj/pKKMxwES7wl6xYXF0/foVZhM+90KN/cNDlwe60VGGbiLYOE750fHBfyxOW4T4ZRaTeILKy03fTAQHNXSNs0OULdg+laKuWd4ushTwLIpH7RWzzibrYfNLeLfzgQRNAucU2o3jqXsPkVoeblC82+IMZDcfA+ynNq8YznxcyIBUeUFh7m/CgCpkXJ9SvoPOeloqg/4/IHnk8eNA+fGEFmWqDEp7695fB/+FpfOz+yQiJCIoameNmlhyyN+xQmxh5Bn9q5fAJ/m1s2GoMW8RX1UkQvGh5L88F7kHkMsHQF/kGsXeLDbdLLwN/6SSm+k01ajykB2xhj32zFTIMBnhMOIVrIMXpJFsFt9S4iNY1XAyWfbPZDif82U7QeN4UAlaaHww+vH8hgk87o/k0BgpMaM3OTxgdK+IjtdkZHUBwgZ27KmYzYYlDF0e44MZEmCxvmsYzGrLjR1iD+oiPte0qplrD0Hc+ioN1XG4NRH5rbhXeYmQcTIkfaH8a+g0pqAuW+bM6Y6ZZyGoGgGoIBXi2YRMjvj86zi/p/CB7OGq84ScH/bZSOado4Niv5uLpTKDkllBu1mcB7jQOfTzm8oLzQxvcguHWVC5TICttNrdoVkWd45EkyEkYfMLDzcbj5msuxmAv3diNb5Yask2XJEvipEi4RV9Xo70dtP2e9V+eNM6hrFQpeQA1TTmvEN6Bfc5WalYaZj0Rq9ZZAeY3Y99VRQzNQFLxx3OT6W/h6CXNL/hr6r9dbG19htWjcqVuA2yW0DBkP/W+bnhLI1uiay0I/w8AAP//ALcASP88aW5wdXQgdHlwZT0iaGlkZGVuIiBkYXRhLWNzcmY9InRydWUiIGNsYXNzPSJqcy1kYXRhLWp1bXAtdG8tc3VnZ2VzdGlvbnMtcGF0aC1jc3JmIiB2YWx1ZT0idFZ4V1BMUDZqMHUzcDFEam1mN01YMGtxWW5TN0VxOGlEdHRwUkNsR1BLMWtIUnh2Y3dxY1NLZ0cvM1VlSm9CanMrVENxUzd1ZC85SkRnSGZNN0Jranc9PSIgLz7sfet227aW8H8/BT6ddToz65Qyb+IliT0rTZumM07adZpmLn/8gSQksSZFHZKS7XTN33mjeaF5ktkbAEmApGzJSeo0R1kxReJ+2dg3bGycEPnvWbpab2pS367Z2WSZJglbTUic0ao6m/xaGVVaM6NitIyXBqYx5inLkglZ0RzSY8iEnJ8Q5d+zarsgN3m2ggKWdb1+cnp6fX09vXamRbk4tU3TPIUUE3KdJvXybGLbE7Jk6WJZw7s5IbRMqSHaAeWXG9a2Ji8NC5LShJVNi67YrVFB7HJy/mxN6yWZp1l2NlkVK8hW1WVxBY38U+iHz8MXE1KsaZzWt2eTqTshydnktTOdTWdLy46tqU9M4hBr6sDT2VpObMKHb/AAA/6WBiQzeDrDwWB4OlsDE/JQHoJJ30NTTrEtWovaNmC1ljUNiHcREGs2tZbGNLywTAxZWkpmHKTzkxNlaJ8l6bYZi2+KG7IuYHLSYmXQqCqyTc1IsWXlPCuu5fiRxMCRIL9u8rVRF0a1WSxYhVkqAlM7EmzExaqm6YqVE31ST55tsqbuptTxEmqWrzMKQKMXdXLyLEu7EuYZuyH4gDIg4xymsaZlLYIA5HJoClvVrCRrwyTzGcDbNl1Q3l2MxtpHgoYNmpCyyAAIijX/wqY8o6SmUbpK2M3ZxLBa+FoVxgYCywwaTdQm0k1djLRsrPd8ykcb0kYpzS7WMElrA1dAyeZnEwAPWlNeiyEWpNIRMSEqEDSVFHGdwmir9TZBvNXVskxXVzCOsIJsUrObuumCmMp2rvnSrdMaB+yvjMNXUd7KNZnRiGV6uBjaNF90S9jyJmSbsmsA0LOJCYvK8ggPY2UFvYAEU6td+xjBe4w5AGDyNbRmVffWfdMX+WuU0IKRvopwCZxav5uha9ejUW6w4WzLVkWSiEVpE3s6ew5/+AvtNi0Xf5fB1J9R+IP/PFS8bi1IpQYb4n1paOGWhdgBUAxmMeylEVCLWJjBhAyWizhGLcSamr4NgSa2xA2xLfjEOAiGkrZG+D4HbDEzrHchFBcbU2fmAU6aeqELmV1AkVPbDN5hM2RVFpS7DN6/nhFotD3bOvCg8Af/eUum7tTOrKk7w+oDX4lC7GdeBFMPMZWv5zGm9tboFwWBM/xbQswgAv53+O1EIrgh4P1UFr+yuNahrg18HJBbi+rHoK6JejDgWXz2n/MfS0KCyamPgLIXJo6+bXlTP3B533hKoBYIsHo2jIXZnL3DkH6cyyfjFQa8h1pFlDbbcvIEbGvhCDf9YDmnr/rFNEDwTtbEW+Aoi8WUi2Xra2vIxDaZW6MXionx731uBFiLvhxFnllvMcqSIPj962C8bme0bgh9roRC3n1A9mfOjegQ24Q9DsAK/mgMXmXMw8EVR8qnLiCmkIgn4jmYmzAMAI71CJggnh4mbxrYEOy71CMezwLYxkOU42XO1HQJPlTMyaMIjzcwjj/GZ+PZKVDFc8ErPYOhbkaFbmHgSkH2dEo4TqJlekkVCc1qpMnqrL5lNAfOsowxouFgA4WDDSZNO0YItVIV8s87WsGjOsajz9Rx+p2xeU2AYbku6ZrEFbBd5WYVA9ulfRjQmwWrJyOjpLQuKkrgegBQkftJgIDoQxUXWVEa0QI4u7JOaXlL1jeQSATzxrQReYa5PTLkECOaLBo5QmE31nSlyBpjiUUNCZvTTVZ3M6NMSQpDskwrUnaMSce7/jCIbOo+xcoPb8kiKyKa7WoIzTJSzMn3af1qEynNeA7hIvCO6ndLPomRrpArNaKsiK/EMG8NmqWLlZFDhgxYuP/97/9RCx1M9u7SP9n8JwaH5q0xL+JNNQQH/GoH6V/gg9TF2IQ8vPv4QhFVZCmMxbPTTYY/d4kysKZKVgGs7ZRk7pFjJGc9FBdAiLlLcuJygGy82nllnCsGLUkoAvibgsisLOGCCAwdkQ3vxkH0W+l2mnTISKZuKFSWVnVU3LRQgXJXLjClIrG0QzEq/TQjp6QaFxIHCR9DRoxB/Ep0evhHlxZ5ly41NHsUGI8C41FgPAqMR4HxKDAeBcajwHgUGI8C41Fg3E9g/L1FkuJ6hXubX55gwjt2eRRPjuLJUTw5iidH8eQonhzFk6N4chRP7hJPinJBV+l7qhj/qAKKGn0UUY4iyu8koghY+aKEE9Glo1hyFEuOYslRLDmKJUex5CiWHMWSo1hy3DU5iiQPEUkagzPS/muTiQ8+kZh8XpT5eRutjqWWrw1Rl1N7+qlkGXD6W4b4xyF5ZLj4yBYwpnqPtCNNIKAIeeA0Kxbp6p9LVm/K1WVdnOGZterPzvM/2y/h/yKtl5toCgQBPn66fVsAiF4gElqlqwUErW9rDDIyJQxqi+AnpxVMGbwkRVzBT1VsypjByyVIZEAl4C3NKaAjeIFGFNP1ajHRTl3Jrr7iZ95es9UGqlld9QBHFan07JyeLW+TsjDiLI2vzia/ffW3TVE/RUpTX6L0I76fiB/Ag0uISGMuOU15HhHztfhZ09usoInMJAuDoeXpL9PV5Rq6oxWJxwflmT2SQwe04jpscZk2ha42WfZ115xhI3/+4fs3l7/8pBVUlCnMIbRitbjclJmWnk/nk9PTbiJP+9N4OpjEU5zCUzGBpzh9p2LyTuXUnYqJO22mTWvNpmKl3qH/+q/JYDKMZU6BvMVOMHdtP8aHRUOTsdCJvHnEqAdPx0xcP4xnUUj9yGdR4Cd2kCRJGPq+E3px4I3M+II20/2PFwWQu4QUm/qfiICirwmPg8CfYV2TdPU1p3RPKlzl6UpbI6RJoy1J2iUZrlh1YWGRmzWsrPllXNMzLOsvm/VXNF8/xTCAmzMBGX/JeDP/As1sYxGSznAJOi9wPDm1/rPznQhBwFFC5HqrlsU1zy/mShbO5VUl+BK/z/ZZyfrQ7rkae0hU03dICiAxPf9oSa4kC4j0bbK+Nax+7cfF/Pe2mB8w5yua3UKPqilPcP90I/e6gCnTJwUXPSxUNTeNESq0ZLxFaHZeifQEqDmhcQyArNfMyb2WtVngT06/ysQQ89X81aJ+ykPa9c1D+PTh4n4qUckT2cIGjTyRMJm12O5LASjTs0PTdWjMgsBmvhfElm/79sykUejE8dy2bM+ZJyaLTTM2Y3fuRoxRGpiBxcxZkOgAdd4DLzmMOvru8LvCtSl8nmTVxJA3nG9LBziHhqcMuNLZKObcxB9gsWWDcSKRYeXMKoqZIiv/x8vCEvhwrH+lN4i9M+i+OLlQGXM87K8dG+C1Nkc1FNlCpGwiRtS6PAERyeYwDYT89puIe4OSIczK+YmegyPnxt3Bs2hT18VKKw3mrqi4QKZ8TqRvBZFel2W+Tas8rSohPOWsqgBudDXxiDzxO+o3bvbSVThT3yb40DVGpodaTeJNQzdzpraNakxbVT20moeLEN8Cnor0U3UKCigslHqKQTqh1RWKjguskgS8SagLCdRkJk+yS8EhJkl8NdOAAAqgIacHAEMBd01ywe8G4DpxRpR1wj1txNkmYca8pAugn7UCrauiTueSVgMzwbK50U8t5y+iFYjKqBkZwV1qMdVpxGp6ygvD3vbLO2/WHHbwRGFw6HqdNU3JYZkRgUV47VBJntbGErUlMS2TymArGmUs6ZIkaQUrGyHwzmQA9htm0FVirMvxhAqfybEK7guhBRl/k/5KhH+RKl6ynHL/Ij8X8/qaluxnjpFfFEkH3A0s8z5JfMIpDUc0HU5RsE93kKlpzEkHGgrS4uUOppfP0im0qhoSlWpTbtntZURXXZU1q/CQV8ZiIDjo+wGTGCJJG95qmUbnU8HAHUPWdcMQaLs75FWDyAzYhXEMaFRxydgKxecZuk+5xZUe0fhqwblTgzOuT8iWlv9oiA8Dqbgs1IgW//R0MkTdrXygolK59Qc1OYhV+SNPQHSHXxDdZ5MdUn+nr8vTlcFxGzDd4nee1lwH0JNhni2tFoGrO45cpzfccbyOjKhk9Mq4BgadzB2hCFwVZU6z9nzeoyNmvgknpkCqm+XRQK6FPe7Efc47cZrSDeWiQh4XrRDtV3XJatzRQKhcl8W6SdNMKm2yAi9L5ujgBjkJmUYykg06bcxvVVOSfhIoZieWalM1u+eDJJPzfojkIJVTqGp/85tW86n0VoLyfGHkm5olk/NTNXtdFquFMh4oGaheoeyR4WsHq8VHZ5M/7UL4O/o25Pwn54Ogrru8mYIt4G2XXdC6f4F8H+FPQ1m0upoV17AFA7uJgA43AwEst6UixaGytDvWjFgZkbIhxLZqoKOQeuO80Va02H5NkwQ69YTY6xtiPhVo+BmqdWUlO9Wme6pHJbiuinmRZcX1UPJ5BL2GyjWRahMBGUy5KQvXcpBrCgD14bqOix+/v/zhzZcimSYRs2YzmvgM5M9oFpvRjM1cy4pclvhOHNihk8ws12GOY87NKIn9xKW+Nfdn1symnqPLQf9RbEi+qWoSMa5OACke95uAxC3pCnhufYZYjSPWnlxvBKv7qWhdFFmdrtdQfPdqVCSqgYLm+CNkvsen7zAqmUbFkWIHUAsFsi130sMACZg/i6em5RtTy4GHGfqSJDkdXYoxMiCcJtlTy3J4NHw8bwrDkscc143xDZAW+AYoljiCbzCRb5htbRC6/NgkU8f1oELTnnoB1gRkOvQzbKnpIKsx8yg0OMA/SSlN04GvbIqdMk0PyzAhu/hz5R/P5eOfoMcQgjGYy8ecS2ASQk98Y5EWpvfwr01viKL0UBdDPV45vvl6NbxxBrZOdAC5nZlLNZMPi3fSgF6+08eFj9T71w6ZURggwZnAoxspcwYlw18Io+HCzDhh1g0TGpbMLF6GGCmYuKkbAFvkvLIxjPbjoYm2h010vExp7XOFLXGIj5W/G2VMOLi9UUVIaV/YkDixvceBsiMJLYv+RdMI1GcRgWnuJQWWH3i259vm3xE9CGgyZ/PY94LQpGbkRGbCYp/F1Ia/OJ5D4CzxPMexosicxXE4C53Yiea2nzDHdKM96QGfBqoaIXxUInANg2twDfZnSBN43xtOkUulUQGdziW7uIfIFxAuviiynecDTrECQBdBgCq1wJoRkI4gwtKUa5AIdXC2HworIQ9QVAiSnIU2SFY4UMQFIEWGFro4BWzk+pjJ97CEMBgmBckwzKa+bWBR7gU0ArWGju/2W4AILsiwfSD+WdaFP3Vsf+p5gWJMZvFOvs9NFCXd2YUHKQH79oXXmedCibwz0A8XBU/bhYcV6LWiQaAHZAVy2KhRNANM5XuhgQKkpw8mWmZlEIvuYCES8jgGz9Iv0uBlYpUGVslThRDqDlsJgTiONsq7W6RSOzG3UE2gsLEbtlSk/TPA00knZwk836LwRl1fxCnN5JpAf8DK9wGC0ykC74K+h9XQ7UmoK96a+WFIEOVUfJGXuDx32hsB0Qyu2m0KquxMSOWYtI0fEqovkTYJi2wG0mxW4S7c1ZFQ3bFHa7HIo6FnzpwwiZKEzmzmWUls2SZjgW/blh0nbjhLGIMkZgyyDJ051PY8K45MOt+TUPFp+DslVFzHggOAmpw9yBJyzPast+eDL30DZIHVLdumqNcjtuBqhcoR+Opp4AfP1RjT5PkDrpBECcWyA57VIuKJmXnedzIN1bPji8F1iAYWvrvaUc2oq1k5AxXy2pLe5w4GAwFT897RccfQ9bKWyatXjKhNLm3sIg4vYTp0bp7y8P0x+IrV11DIac7yqEXjI1RCRGi4PXRdidoFVAwwe7vbAwzCVWNOqKJyVQt10soimqGhrhFTtkEqANoq3TIjZ3wHa2Sfp92q0DbteGXPVnS7lxJx1+md+9dPUyLWxF3uw7KlW2XXBqsiPFXbl9aqCDP1bZd/aSLfQOTu7RXUGN7fOrUw4DiTW4KuAg2uMOyda8rSPcprLaRQKdkcFsOZiosETamjQ/SxcuOMRnwT52ySml0pPEpsmwHeRFutSuyKXQrawgHwMimuV0hvBTxeig1O+QGkntGKya+aLuRbVNJVvGzC1zS+QuIkywNQKm5xQ64i+/dgWdRX7PZssiDxpG/F1wHT1+RNe7BN2PF9TXCLk3T9lUDK4UmaXoxDarwBHgsnB9mJfaBUgwN+LHB5c6mCoQhsrOV3g2t7prCdm/55wyZiom3Eq/8+AyqHgKattPY0YKtfr/JGv74PGXSFGYU9NKPgRg12YyMxMLbQbCSmro8EzxnaPKDgNDNcLh15U9xEU8mJUoTlcPsJLrQN6jK78yJteeoJH1mX0VW242RPT1yRwHo24Xv25/jUrffVHPJA0JuiJnQLbC+aDOwzhy+QQKHZUFO0gC8kN4KifBRExk0bqgNR2anINcRollrcbpwmUgkcxKmQfM/TjFXAcbK90JHeig4rpQdipR9Eaw7DS58TDvqsUY+wncGTzcBLPQQFCWVQiFvm/IBiw1E6pPtGjnL0NptxzZJJAxKIYqC7z8UH6sBNcQoyoIBziDeVpl8O/HQBqAmCkENRhYCyybn4vRdd2KF/EJKA9J8UT6zR5qZkf4O21wejC8w8gi3skUJ3Iw1eCAFWJr7aDz2otXbYYX0gdvgJzS2bNh6RxKdBEosUBk8Bho/Kq/hTy/dRHeo7F4BGfGfq+b6y0YQbXdNg5m6BqUHtsHps2bV57oumDPhWE5ho4eO+f80FZN24SMi4PWYDTyzb8sSyJp9bjlAWbLFory/3S1H73QxEcFvXGnCZG1USeKoam7A0rHfu0ursi4gFhXo7lQmaWZUpynifQy6zp+cYSPuiUbLzln1/3w/EmNrKQxMT5fNe/BkEB6HPIPik2LOzMz0YdypZhxjUGRS8G38qSfdCn8OKOyS6OBCJfqvUfUShn0jEy1GQV0yaPyoStTiSGHcOEYz6hpiNuoaQxoeq84UZypEXaIgAOMZ134U9Ka3x9gCYbcynhPTpIMwcBj4zsM3YlBcmsJCtxwxTeswwX9lb6JnrUNwEE1thvtSmugF+mw7gftNDfLh0plYQ9l1o2CRoGtGPQgxKrKUR8Pah5Ymrj6BikjlmmDqdxdPQ454ReFzgkraG7aA+S3jzsOxXljveKctQehXiuQS0OQ10JN84YLAxHh9b3PYLe1a2fD4szehUnY0DMb2CHibnyscfVaiWpoyHonqZbYjmXa3A3Si+MaE8vLIOtdMDUftzWecRrX8StL7O6O1HR+R7ybf3yMlo9BASWP62ykubHgS5LvKxlqUbpUNyQMAuauI8l1t7hWq8icx2cKEUeygWkZA4OZcvf1TsId1HHS5ly3xD/DHTi7xDxpbpyIpdX6ohRPt4QGs6BBMdKoA3berafkQc+/sn+7i44+iX7A/ol2wXwmxWFsjW8u1elLkXKLZo0vukeLJi8aZM69tD8WSTb4gnPb3IUTzZJOEb2dhwQjNWAnpaF4CvbkldXLHVZRXTFVZGcOet+3pA8zrEWR2IOH9uWnpkzT6NjesyZdnD9jR2qypdz59ajtOz2keJDOTAjCMua+oFfdEPdYBu8A4t9QE9eJ4xdXA31ApsQDCO6YDk6cH7NAxw28QNA2TAAhTpA8sB5GPy5MEMN1lQQPQb5OQAKxij3Oj6KBe6aPhquiEa7s98kdHgGV9YWCXIvBaa/eO2LtrQEv8dNmzQHdvC4jzP513C1+B9PkUxG7jAUBeRrRm0ITPavvdifSAWTtN1ZwaIl+tiPW7Ca85QaxtIgoQmu1zPilpOx4VBcSCVM7VDZFWn4nwJ1OdCn9HgFZB5OMMEMxg7yGfKLHi0MgxeWKi4mJFQ1IUKjABixZvsd7+peJrFF30RnX4d4pg1x0EtENHxR+pubbSm4jRKpyWtOZczRhcOJQMNnpicN2+jZGDHkfEDMNopokyatRa5NI7ZGlcZu6lPm2L/sqzzbIcPgE9BRVJAS9DQh+xpVWxIQny9vN2s9gJQ47I15FnVZRptoI6KJGzNAKWs4luRRgbQqKgvN2sokAGtwcrJmhXrjBHU/G1WiOgPbffeO+SyT0cW/LBdLT59H1t4t3S2skEFyPLiOS0XmGfH8UQCrqVzpwOjF+SEJdObI7ONNvszp1dq46HEMvlBLHgAIgbp3h+kI+YFd3MS6H5VOvObC5/4Qqno+njWwR+kErTNuMsj9C781UDn5Lx5ewTJX5h38orQmlMeUd6mVRqlGazNJwIKnx68fBqd3h2LpDGpJK2fSRpVRbapAUWgn0l4CIPKkhtUkhIHCX37ypFpzPH3wKIiJa8RFVPNN7SM1WTg5lIaeW/yHA+O72tTfq5MmuJOY4hU8tIwpaO61lPxZ7T8r1gEdGFZlOl75K7HzsiG99q08CV/f7IcVXb7G8gMF1bP9UFVGsUqA5bgdVGO2La1voTk3Lbep/acVQk2/Mi6AAt83WeQk7JYoxmsyKt9GdX1RDtgxOtqlqUW2LAMPNvQGleAzDAfZxq69nICpBhG32sj2rVXZN1p3Pr5WwIf4IZiZBgJNwVWDnGpoKWfgrpjxjRrw489Z72p+t2MGQ82whwfX2Fm9qEjPGah9WgD/VADsEPt1MYHVDM/+dBxHdptPNqofpBZyMPMWMYHWNn3/dDh1fdLH21oH7Qde/h28fhwyk2wDx3K3tbR4y3+T7Ez9YC9tR3IQUZ/6Gj3FNCPMdq/o377ASr78eFv9FYfOvw95c2jAfsj64YOVYLt4EDkWO45KfJ8JEgWinBw3jktFaFdAAyuepQeciqyYgwccDlPbxBbqE4+6zRn3L15d17xJtt16rDnn3Lnhomss+fDEjWk6H9XHgNV1VRCRuENFs4MTiRMtbrK5q6UNUNXjugJrQI5so43dW9X6KBVxM+tJ2Eyj2wW2qZpJ0Fk0WROvXlshkHozd3YsubOzAoCa5+T7ZPzn5oGKqdgh64zFc+VygV/rY+3NXrW7Pm5zHEQus/mfsCI1dfo9rNJo90b2ALiyZ2Xb8iplUqPnmZDaDL6eg6uacgjeEw4PAhRSi5eDqmtu02p7GguUqlX2vU7Y5deyKm81uOae6SuU/Q52MpuRUlQovucHC+j8b5o3763RX0kY/l3nuYS1DJx1+mV13nxRNt3a+wgvWZQ3ytcqlOngeNxP582EU+swiP+0m3Lh1K2BprW90zyw9YkX1pk3t/T18506Flgl8X9zuvHFAXO2J1PCsGTvqOFE4s7Lj5qSVZMS7w1Sr9Qp9X+jK27n/nieN2sjfEoIy+STjMmfeMPU0lXwKNNVJLxNTM5762ZU1wwvT7qDtGVIrgb9G+kjlJ3hi62eorFImPoMWIcC3TrUR6vf8HdrAv1lrZO55QT28e5DO7vx1m67ie9fxlAulpv8C6xfHMjRlEIVTB6NFKcGgjXjk+U1H+qgLqyHzCAjKVNtMSCN+OpOyDWso1RzLjINjm/w7UBZQNBGZnTqvWCir6lBeA8EU5QW4ZqfMHBXNfKUupGodnKRFSB26Jt66eCCehf+AH/hCeElbg+lqsgU7yggNcAPyxLRjINmySqxwu2+HjAihjJxtcOYCLOac0N7tehGZaR5O0WEZSGVH0uGf1kv+zotpvf+jAehcssYzW6K57PdzVWIoCXfDh0bDSShTv1WBYZAOfeeQR+wuFXIxUQuONOMhVtcngSiAjecWwmu6GhmSLtio7dCJVLUjoibWuSGLGRgZqdgm9kt/Wl+0EVnb/lNKBf3ugAtbmAYLBsQsZhRgyQhPX7J++O9T1yryN+TZRbqfXlio1pXOzrPKPoveDFhpymmM2qt7hR7hxJ3CDCHtSJrNz9POA4tSV/4lHfiZheLkR1zSzrmSDmZxnRy8OXHx90PQcPf5mWVX0Bca+5hx8tq97kv20YSGQgJK+LFMniPoIS1NcbEu0jpjCvBmfat+YTywOBKTQ9L0C3e75n9a7Dkvs7EkEihoQp+reX75Pv3932xl3cryVhbFcq9CgufGoBNkR4++VV9kvk/Muv9Lub9X/aSzPKs81/3obX7N//uo3zN8XFvynh12e9KV1DZ9ELOd5zkxebiiEQathEF9Pb62s0BKHO0JSX9xIEKJa81e+0UcrZwQ7Ke2VE6RL6f/uN8Dd+z82LYpMl6GSZ4P5Xu8q0axtlFcoFJ/3a9+jFqniNzrTf7r6VZ6Td52+KeomaMPT/uSyuW3WF0hS9kf/PMMjbH7/98Ym8WIfeSHJOUlR2xlgwyJMg9zHU69RLJvvMXRuR62L1DzWpYkBa2ZSQ6+XtPxPD6A1EuzM72lEs50WnzlD3Y4cd5ZV2zEfb2ifEccz1zf52pnsCBE4yjie//WMNErjuvArT4EDJZCOqKM6QN/eDFDdGlb7nTuObZkHYUzJyW4g0ok9RwMG7QiS77dgdn47vY3w6stPodnG1l2kJXaW5URZ1A2PP4rSMUXUHhQaQ6pb/wDr3cdjL4go6InHKC2xpE2oUaxqnNSTHs3htaHMjMYoPON0Gm8/hhbfPqGJU3iwMkXZCTnkDWgMFC48Kocdqm4gnl6p9476WdHXKANS5xXQNgIeXs+zblBEjhZZkjwQ9YI2jTkpb4K3PzQ/wDAnoalNm361QNZ78lc3lFV17KdTGuGOuLNbVzSVAezFw7ynMCXB/FHkpxJrVCzE9pHUyOYZST9p18shHmLHpRO06xva+DZFsP1UTCIoB+gjW5ElT3iDuo3LG73kZbuzZ+PHXUDsda1qKIIpWuyD+eugK2BuVkHdc26WqLjjnZ91xWXdr2cUnUzCWAGcvMDufOggvWwjTlDO8nmYqeXYgTd/KGzwRHO6+5UO9pYSrViVzotx1Iq5R3kXYIMG8KOpRdY6IgPGhh2ivG877/J3YlcoU4o9meKJU3UxJXe133Eiyk+/n8rZimzTO768SQlEt+6Fc/siO033MPtb65TLvB4ktH4fT/4KY+8+cdRdq2f3Z9ocz7R+h7eN8/V7tPrIiR1bkyIo8IivyseXfo/h7FH8/P/G3l+7jMt/C5qBlvAXp3o/pHm0mLFZ196vZnFM4PMW4R7vDWNoBnZz0fKgvbWGZAWTPwBlsAQxvSE3icpNHCg/e4RR5lap6RatyiyrJbwy7ubMVLYDFa54oxYgDKFZrkyK+uQlPjssWHhCqHlDRMG/jUb2EkSTyqpwsmZz3E2GnoMni7GV7nBH6C4SqvQiW4lVsvZsk7zReOsCOX9wUOXKppLxBkra3SI5cJlmxNS1pjVeDno7FP34HT+uSMfVaj6bH+D7SyT947yT713RSfH2x3Wy43Ka78vOL76/k6ptui6+H9Frca9swi+mKZgLLnjfyQnepbIPhl3b7fgipgcIT2aOerN0NOh8uuq43JZPWA2hox6/XVS+M5dMwolHA/LsN8WoFU39f8AtyQB5uu0U7mtPYELaEJy9KZETE4QMZu9eptt/jXKVikzhyTq5vt4QH/4jsymNZKf0BTlCeiGtqHnYOsrsFeY9jjUq+LO0UjQLVJMJgdjWi8aCDOoQx+sA0N6IVExcsi+Ww/z1hnenzvneECUFMK+TlD2++vXz5w8V3l9/88vbtj28OvhHsD3zRVzhjpucGdpjYpunSBK8r9qP53GcW9RPmW1Hgmyy2mTkLYzeYOVHsx/MwSRzfZ4HZLK5xfw6IUDkK+5o0V7M94dewNSc8Jn38pxIqqXp5EPoeE0f7OgeuBD/v8OzY2f1BTs4mVzmKIkJcFQGNymAyoj06f9vdTq6fRxg5i9AcDtGsPXci+MGSkzZKgyJ+3eTruuDiYJPZSFIK8DNqa7rfQWi+mvFiNTTDbxD73cN3x9ofPe1y56Rhtl2T9hGm7WInQHSeGBADS9Ou+ye2P36ArtOkU/wIJDqadTDPsNjWUUHLBGZpfXsIyOgmwphZyKxbmm2gEXvpZw+FkXhTVjDmfHekXZyDYW0bg4Oqd/D+wf20I9ScNWmH6QOw+kc9BvOJpuJTrFl9JO9dWweBgDjCNTiyJaKaY1oDKyrgH8mu/dL2iFAeGU5nOf1wV1TqaboDKHs7jWKfj6tlNfrW65A8sDDCZAl1k27Pqap+r1jGkHbQLeXXR/MfA5mJ/ohkxspCkQefNX9G8Gx105zxfmK765unUkXN3yfnA3XhaP3Ni8FxNKJrY4Z12pPzr1ZRtX7aK+YuU2McEH4Z387hIA9pj7WjKf2TMB2lSdAyvyRLj4gJNFhZopj7gq5wV7JkABvA4hLtzCWthWUenh3sr5cW1gfOynpALwC/g/3ebgRNcgZ4MZ5KhcBEXRxciTguA+I/KXLwryEErm9BJl4Lwbgb/BaKGp1lWVyPTEuDluSllsq88OHMixXwbJ6iVuXuhhSlqN1TilofZw0njO+OH7SCd3uUs8NpQP71mxMViEcOMcJIWvhAXW5fkzyuApa9r3JjAePb5OofZQR2h+GluPw3T7p5HWLKevV9WWzW3SAeolcp6fVBGI+bvdNrA0Spw+8D7i76JU2rO39t8P9b6b/mRLnlVUciz/prmqplN5qC8XuHV9fCXVZecFdZGasqNNmp8YhJzyZGSDoyrkK0uQIyXX0NYlkvpb4Zn7Dqqi7W04716CVX2Zcf1zDPHImghIO3Ln+f1q82EflWlDLWpnFZDu9RI2gfQmQDhsLH42+eJ2ybxijhqC3cw7M03l60w6Gzf5g/Z+mZeeAvml/DgUcyX414nZaXhfjCj7XV+bG2pR9re4kHz/zYmKKfUtPFS+8xLvDRy6rNT4r6M1/1yuxKB9ie7oNvOvPQ/6ftOlCYD9mnQYABVujA07c8UZ7ByxO1jrjPNpuejrrPtt6/DqemFeCNIN40DGw6w2Nz/CGvfvbEmDvoq3UsFp3E3umNT1GG8m80Zv+Hyf9HC3T+jnvyN0DmGGqab/L1OY85Fbw+3z0s8/Nn+GxgqfHSiSuVNGZsezmYQf5gD1W4guGE/1E8sFdWSId/efvSCCYkZ/WySPjB8Xpy/n8AAAD//wCjAFz/PGlucHV0IHR5cGU9ImhpZGRlbiIgZGF0YS1jc3JmPSJ0cnVlIiBuYW1lPSJhdXRoZW50aWNpdHlfdG9rZW4iIHZhbHVlPSJoQTlUNFpaREx0Zi9IVmN3Z3U4THFoWXJuRTBzYXlsMTBpblBpUGtvSXdBSEZxSkhRVFlEenZtVHJUd2RmcDN0TnZnbm5ZeXRLUE9naFI0YkFkRmxXQT09IiAvPuw72ZLktpHv/RUwH7QvAxYBEjxmutshjcKWHaPYCa/WEfs0BkGwSDUvk6zurnE4Yn9jf2+/ZDMBsoqsqh71SLKlDm9PDYkjASQSiTyA5BVZ/F2nu3FsG6IqOQw3Tjo2tFVjqaBokaaZbLa6J2PbVmPZdTpbJGnz4JBx3+kbZ9ildTk6V2T1J/tS0kqmurpx/qvdkXo3jCTVZCi3DfRUNtAZqeWdJm1Pur7t2kETVeCQg0MyOUqalYNMK00fyrG4Pen+erjf2jGKMst0c+OM/U47pNDlthhvHBY65L7UD1+1jzeORzzCQmLKdD+ULYAzlznkoczGwgKbEbEFVW3dtY1uxrnPiUwziWbyjL0cCscidt3JsSB5WVW031VAFH2vmzbLoN8b59vQFYS5kZAuF/ADbDwGbwr/C+6eFsPvz35B/T9jk491AI09KOBQLqEEfgYSpuSKb7jpdlFKsVSYtm8FcaM4IMI8PRIaIBzxLfNcDhSBPpiFgbcZ7tvADZIQQMNlv6bbIOEuE5UbmuovEdy0wXE94QYeZEUhXJYoN8FGbig4dcPYdB0wxCwKsT2F9gukp86py4KkolP/S6IAoYLE5UAtit2fVVGsMy2x54/O7fUG1+P26noDbLJmneuNZX6szNu+vrWFWXmPcFPCrikk57XPaF7pRwK8pmm1te/HimAhbftM95TbzLZvH6jnzGNeZ3qUZTUc+gFGz9qHhkzltNeDHg+5FtizknuS0bKpykbTtGrVnXOcwfWwq2vZ7y9sXWfaDnLo2m7Xzcy73Iewx4YSdhSR0KRtBmdJml/BjrrTqUxp0fblx7YZZbXaXLiRYpJIZjaT5UqP+uSY9zyP+B+/xcwPg9XMJ94PQj3JSpC1KzExiynbVafrTGvd7MgqRweQnMO4RylhqPWasEh0j84Jn1blqcyDQnk2QDnq+kkRTb5HDquBreiuqfQw0K6SI7L9qbg2f2bFZogB0WtgjOFVLdVF+KLX+Y1TjGM3vN5sMj3cjW3nbkFc71IXFv3pQbaSqqpUdzfOnzRwZTm2/f4VaTvdEJT2ZOrLOScBIf9+gPraQp2TaSNPN/2ammfEfYKwzjTFzfv9d22vine4FZqy2W66/YgFtDqU9PJhU8th1P0ma9WwGdpdr/TmwzBK4O5NWUvQbJuq3bZu12zPZvY1jFm1MjvBezURO40FD+6qg6DZTDJk2jJWjs3ibAK6moQaTg1VLmxH/TgetiPsbJq22Z501CMgeVL60EvgpN6smdH2FBsQsi2HEUCN4ENhd5B2R4mJgFTBnofmHfVPRE0nj9aH7QaHgt1W1lsy9Op5JEccP4Pmv4UlupmkYgWy6LgWsJsBowUtL2iDIymvri4I9k/L83UeLKv+ziEliLTvd3U3ttQI+xkqKyWgNgu/WeKbfVO0453eA+prwf5H6AXtKezGzGaWTXZVVv0uVpvgii+UjtFxRo2pttrVDZFNWdNcgrIDey0HQpsREOdPjT+T8TeUkn9z/kIovTXp6w0yhey1BBQf6+7W1GzaDpXR7aSOr/E54wjiCwejM4VM3YFNLcLOpM1uHEwp3QHbFbKHZbhx/vO739HYIbUeixaIvdXjUpeWTbcb56Gwa5h2M/btpNflDi3UHrTCPGlqW5xhVeoqm61hu6NAiCpdtBVw9po6XxS6qsruzSfoR3DgvFW7YYHrZLEbJlAV2MrTauKsV2b4M9QuGA2GDPD7fWv4+SmjaMU4046YxczVUbpcnQkc276WZbPeR8v6+e9qKTbytkWJgQsBjUEugJVltCTNd1UFkgQ2GExqKO9hPrBUMHGEhUmWTd6aaS27M7oFmAOagVqDRisuR1ut12jAaFtQbbHMplEgHUqb9pj9HjyZMt/Pwm0GmYtTPT5o0FDAhyHpUjAMa0zlIUyqanuab2m9G0FTT4IPFNhUMxXY2nn7H22KCoWuMRwAnWY9E4Md9EIZNy/AR3wWtrVBNMWqhfValfPgZh/Aw9Z/odpu/4ZwjzPy+3L8Zpe+In9olLvWs082B217YjqA8F7YDRvdbGxuA8unadeCrbCfikAFgXFC25wOur8vlZ5YvtiD6p6tir998dddO75BL2z8gPvD5l/bl2xktQcFMbgGwBa+sq9O7lENT/BTP0qOegsmyqqX3xlWXTW2kmgFtW1xYxucV6Bm568gUXK8PgcEa3hbNsC8zfbDrl83mem3IN3PrzZX2OyA6B/KmTwN7Mm///18AWgBJuONE4mYc5WpJM7COM5Z7LNQyTTLdcqiPBA8zKXiOkzyLGZ5GLEkT3maK61FnKcKOOU7JAdaQf8Uzur68l6qPUU6aLDVx5fAWhPSz2GuS6AvmL1CGQnupTyMRBB5uQh87scKijxPxFHsh1GoRZR6KsozLZTIPaVUnOtA+Vma89i5fW8J8nkM9utniUGrXV+Oz+KJi7AvmCmSIOShzpgfylAxpj0ptYqyJM4DGWjlZXGShioJfD+SuQhVmiRBwlMBRm4QsTR3TgTIYpozqZzb/5hSP0UyPTw8TIIJSbAbzBAvQeBYdJ/FW+eQL5izApQwMhG+jH0vi/w8SHWUCZ1xJiOuVBaKlIkEtJwfeQFTYZhlOgliYDfBucyBbQw9LjLNQbAcT0bsmrwilurgN3SviCHr1zDLUzY90XPOLQKtR7LHBTYpV+7HN22tO6ASuBLliCa1temctVF+uCPIZvsTONscUJLa2JTB03vnaFL+wCkjDz7jlBGBP/+UETzjO2pRe9bpfUy8t74rYjwUJFMiViYZEO7yhISu8Ilwg4hErkjcwPXgLajL7MuHFpBJqIvn1TE3bzzyphxSrh/BW/jUFoQAlgQAgqfaWBjbPDSDqUMBhwIm8FydusL2Ba3d0MckYa4XI4oMGkMbN+KYxPI4gh/g6wNsCBhia9MZVAjEB4qYG0EBpwCCQyY24buJwBkAoG9O8RPEhhssPAPv+iF2wDEL5dyQyg1hDBgaRkFMsEeGiPkAzwETfMIszE2BDxglhBOsYEgM6M0QFyc+v90ggEr4ByzhJjgYwcEAe1eECMewRwPPhFkrMyWYte9GAuciCM4FlgxvEETgRmYNYSksFkgUQ8PEEhPGxXlwGBJWF1ZZwFp+GUOVT+wTz4kBHWAHGrgBEk7E1Pw7Pzo+HOb9oj7d56ioYdd1bT8u5MpvR7kdbrJ2hDS1YuElqCxznKDG5+isS6AvWGmFYRJyKb3cC3mSgAPGuWaZYErlSvoiS2Mm8iBTOveEF/lexGMWBnES5X4M8hm44q0lyOTn/xSLZ0Ei8EQU0OEl8M57i+pzeOcS6AvmnSwLdZKzJFCxl+g8jdIgDrxU5AmPeaQiJsM8V3mcR0KIJPfTAO0gH4xuxQBaG/8KCXLKND/afX8J/CK78jm8cgr2gvmE8YQBW0gPhYvHVchSkSah8oM8zNFcDvIgiIF1gGWSIPV5FikV5DpMOI+14M7tl+//8ON5ZDoGfGl8MvaybJ4pWC7CvmCOyUHFZFnOci9Qmc88zwt9IWXghZKlvso0uE7cS+JMMpYwFanA0xFIFdBmTPtMOrffTRT5GfRRipdtL4BhEM/nMMsZ3AtmlDhKvTBF5ZN4PBcsD5Qf51ng++BlMwZcw/IozLQMOLjj0vcSxTzuZTxQIY944tx+BUj8OI9bpu1unFxuk/7E0ZCtv/0SX5e87uWl8nlY0SWjvktpeLgCXlyXn90jLW+vT6/9Dxf+4MfK7+Uj1X0PTWs9DMbdn/o8rwKU5FDYp61xiHXYZ5R+8WAhWel+fJ4D73JuItWCReQY+LPgTnrvGHjc4JX64FvzZTW0Aa80LsC/BIP4pAYd3vjdsd+PNbrO6IJHXqTAz0woOuDowQY8mtLgjsYh8SpwddFXRcc/XkfRMRbYOvEN+KAslutadL0DHw8LIvEudAOBvrIXRB+/TaA7CY6rCeij6LTajIeuOCCHwYZiHT9oAgWJd2/iD4+lthDHuBz7NF9E2ztnmzmeExl2MdfTeEm+YKusHOpyGNa33l/bQmL567knRP8E1np8Flv5eLiCj2X8JSwIrPG7mIRuElQ+8go+FqGUFgQf7xJMxQaKnELRI1hMELCiF+A8C2ce73BIEhuUSOBG8WlcqPdEaOTx9h9zGKyrZPO///0/I+l0b0IuxkKOU5gFgdRYlAMZy1q768v9hWSD5QddUGk6gEDBC/iFQJnEyoPsmymZyqbRBxFj0Ph/OfOvLWeWOu8yM1EbSk7L5qCbkHePAeYmRlA27VhgFLtMMdLchjS65GAMOrd/0misodZGrQ+FvQYe3YNFQ6bx3GV82GdgZm2GM9SgGNH7mTE7GBiQxPi+CuxNo/jNRfeF2LKfHsJGqgI6zuWuGo9GyVxQPH7oB/DEMKjzJJTNxu3MymOpEt4a3TEj+Y8MZLPozV1NubrNDlHHZ59JmLizxacSNUZJLvKHWCOZDjAqUL9HIQVQY9uBA3KiM5+c9nmQ1zL++xeXh/9iyvFUPa7VHBClBceODl2JGozUexqi6bPm1LkaBzhu0k+H12GwpN3EFz5HeN92yOw4UoEJJfuMTnFw53x4CDcHKwx63L8meNfwBuUQBhxOWQcFEcggjZxzFkg3jXjwD07ylKYt0Kimlc7H88pK9tAGN6oVE0MhQdbZ4pNgeD/0usc3zgVv5oJkazDQfUT3rUtbS4NuT6cddjqFj7pvl5BThOEP7ttpudaDrLcvFBz2y9sZ7isrPUBE4DqdNK9Be3ZmiGO4fgtsA8szCwCcSq51lkr0TWGMUme/mSqnZsA6vZ6iXx9+TRa0mePZrOkkN/mzRIj9YOmtZ79fsl8yWSNFFMw9/xaqoGz1LZVHbfo+coXyCBhescsYt4X46VW0Bp++yLqnJ51bUwWeJ0YUXiIC3QoanVUBPoHpKlraN5+erm3+xDdbEX6zJeZvtsLpm63QNLk/H9+MDnLyEm6CJAfUaiylxmD76WSLzqlvvnBbE+FEtv4KWLXQ6u6EV7HIMquVV+ZjhmGnFAi0OfzhuWwMJn8Ug8pZayZz3QxqK0JyRSef9tFJUXLUabBa8akSnVQaYR4qUuARTsOLuvYph2+9L1fCdqF2pshuDLbHmmKsMYDk/wAAAP//AQAA//+b4TRWLBYCAA==){height=\"60px\" width=\"240px\"}"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Barlow Twins Tutorial\n", " :card_description: This notebook describes the self-supervised learning method Barlow Twins. Barlow Twins differs from other recently proposed algorithms as it doesn't fall under the category of...\n", " :tags: Image,Self-Supervised,GPU/TPU,Lightning-Examples"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab_type,id,colab,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7"}, "papermill": {"default_parameters": {}, "duration": 9.405997, "end_time": "2021-10-25T20:00:51.627704", "environment_variables": {}, "exception": null, "input_path": "lightning_examples/barlow-twins/barlow_twins.ipynb", "output_path": ".notebooks/lightning_examples/barlow-twins.ipynb", "parameters": {}, "start_time": "2021-10-25T20:00:42.221707", "version": "2.3.3"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"3365e6cc23b14f7480580fbdbd5bc019": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "3882c1fab7874567af01f5ca5b6b5529": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "431aeabf10b2446bb97010d31753e44e": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "4bb27c7671944b6e97113a02c0341e2c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "7ddd35cbfb3845b1a206ac0fcbdbfc85": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_3365e6cc23b14f7480580fbdbd5bc019", "placeholder": "\u200b", "style": "IPY_MODEL_4bb27c7671944b6e97113a02c0341e2c", "value": ""}}, "7f3c22706f004367af15c8839b531c8a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "a75cd1c0f8504f44a015affaa4e54a48": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_431aeabf10b2446bb97010d31753e44e", "placeholder": "\u200b", "style": "IPY_MODEL_d8f730207f83419cb86c5c93f4f7584e", "value": " 170499072/? [00:01<00:00, 116151155.58it/s]"}}, "b13dfae7160c4c8586c5780f597cd8e4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_7ddd35cbfb3845b1a206ac0fcbdbfc85", "IPY_MODEL_eca693d1c480402d8c204656916344ad", "IPY_MODEL_a75cd1c0f8504f44a015affaa4e54a48"], "layout": "IPY_MODEL_3882c1fab7874567af01f5ca5b6b5529"}}, "bd170e01fdeb46d4aff4afb77af815fd": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "d8f730207f83419cb86c5c93f4f7584e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "eca693d1c480402d8c204656916344ad": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_bd170e01fdeb46d4aff4afb77af815fd", "max": 170498071.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_7f3c22706f004367af15c8839b531c8a", "value": 170498071.0}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}