{"cells": [{"cell_type": "markdown", "id": "48a2fe14", "metadata": {"papermill": {"duration": 0.005641, "end_time": "2022-05-03T01:20:56.252312", "exception": false, "start_time": "2022-05-03T01:20:56.246671", "status": "completed"}, "tags": []}, "source": ["\n", "# Barlow Twins Tutorial\n", "\n", "* **Author:** Ananya Harsh Jha (ananya@pytorchlightning.ai)\n", "* **License:** CC BY-SA\n", "* **Generated:** 2022-05-03T02:43:23.580918\n", "\n", "This notebook describes the self-supervised learning method Barlow Twins.\n", "Barlow Twins differs from other recently proposed algorithms as it doesn't\n", "fall under the category of either contrastive learning, or methods like knowledge\n", "distillation or clustering. The simplicity of the loss function and its effectiveness\n", "in comparison to the current state of the art makes Barlow Twins an interesting\n", "case study.\n", "\n", "\n", "---\n", "Open in [![Open In Colab](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHUAAAAUCAYAAACzrHJDAAAIuUlEQVRoQ+1ZaVRURxb+qhdolmbTUVSURpZgmLhHbQVFZIlGQBEXcMvJhKiTEzfigjQg7oNEJ9GMGidnjnNMBs2czIzajksEFRE1xklCTKJiQLRFsUGkoUWw+82pamn79etGYoKek1B/4NW99/tu3e/dquJBAGD27NkHALxKf39WY39gyrOi+i3xqGtUoePJrFmznrmgtModorbTu8YRNZk5cybXTvCtwh7o6NR2KzuZMWNGh6jtVt7nA0ymT5/eJlF9POrh7PAQl6s8bGYa3PUum//htmebVtLRqW0q01M5keTk5FZFzU0oRle3+zxwg5Hgtb+PZiL/ZVohxCI+hL5JgjmfjPxZ26+33BG3dA+ealHPM4gQAo5rU59gsI8bRvl54t3Ca62mvHyUAhtOlLd5WSQpKcluBjumnoCLs1EARkVd9E8l3p9y2i7RbQ1B6pFwu/YDgW8KbHJHMTQrwnjz2oZm9M4pavOCfo5jWrgCaaMVcMs6/pNhDr0+AMN93XlxV7R6DNpyzi7W/OE+yIrsjU6rTrbKV5cd/pNyItOmTbMp6sbBB+EqaYJY4cWE3VUciNt1TpgfcRFv71Fi54xT5kSoyLvOBEJMOMxWXkFlBeBSX4u6Zkcs+3KszYRtiapbNRqF31UgetVuc8z9vBXIv1qD+F1f83B6uDlCUyfsZGepGPpmg01OB7EITQbhS9ribKy+DmP1DUiClLz4bnIHVOqa7BY+Z1wg5g3zgUvyehiNpnJKxSLc/ts76LKm0BzX3c0RNy1yXjDcB5lWoro4iNHQxM+f1kWeWQARAWQS++trISJTp061Kep25X/MycwtjuctSC5rxo7ppi7VNUox5+PhPHtrsS2O1qJ6yx1QujQUzm9sh6hbkBlvvGcN8hYnwjUjH6kjfZEd5c/jitz5Jc5U3ENnFynKl4eB7nyEgP2UZ+Yz3/rVEbyYr27qELrtC4FIC0J7sc7xWnmccdHfRRTs0VB+cA4lt+oFcRR/wUeH8FG5w2Mbx8FQ8TXEvv1xYf4wBP3O2WyL3/UVjpXWgIqaFeUPr+wTmDvUB7njH6/bOv+HRg4SqioAg5GDe1aB3ZeMTJkyRSBqkLsWqSEm0fZVBEN94zEZnYvrdx1JL5cxe+a+AbhSJecRRHW/ikTFRTa38dtQlNZ5CRKwFvUtZU/kvBoEF9Uxni/XqIM+dwKbTw3rhcxIf7gmr2M+H6SMwx8iBzJbw5oxeG3Lv5FX9B3AGaHPS8e8z77H7v9VMpvPG5ug1enh7eGK8h0LBTwUb+GInqzInlRUK65DmTPQu4c3+uQKjwKK77zwUxBX4Tq7yR1RuiwUsqlrABCM6esHdXoy47fk4+prYKy8ZF574x4V5BnHQBuf4g9Z9ld8U36L2aktZNNplNfw7zotwWTy5MkCUft4aLEopJj5/OPHl1BQqeAVOnHgNSQOqmBzq9V9cfEm/yx5ubMGKS9cYPZ3vx2OS/c6PVHUuUO7Y1Pci3BO/1zgq18byebfGemLtNF+6JRtOvMk926ibussZqM+1mNz4TWkH7rCbM5phwGRGDAaoF8fY5OHFnlldAA8sgoEXKnDukA1NgSeNjqkJT9brbN4pC9WRweYXyLugR73c+MYvyWfu0yC6+mjzN1Isfw3FKJS98CU/zI1IHFkFPR52cHL2FJk0sB6kMTERIGo9GzcPkLNfA0cwdwi/hfEYO86ZMd9w+y1egfM2T2Eh/vesMNwljSzuZRT420SW3eqy8N6aHMmwmnFUZ7/PGVPbIoNZvNU1BURdHs0bT2+HjL8sDSM2e6vi4Lj5NW8WOLVA6RTT2azxLV+bglaFNqLieqemS/gWkw7NyoAHo+2dEsiivengjKsPFoqWOvbSh/kxPaxyW/JRzH2Fl3EzD9/xjAefJqB3usKUFn/0Gb+S/d/jy3FN2yLOmnSJJtn6oehByEiHPSeXnDxFGPRnoFoaBJjcdQlbDwcjL1zTNuQpoxD7R0OG0uUTMi0fkVwdzBdYIwcwZunxrVJVLplNm54BZp7jfDfYLoNyqQi1K6KxIdHzmN+QQ2WjFIwUT2zTGdlRXo4NFXVUO4sgX5dFC7f0aP/ZlNeUjFBuL8Xjl6uRuP6aMjSjpjzsH62FDU7JhBuGccEXIvDfJFFBc/gHw80dklfCVYnRaDfpiJcutPA4F7qJsfJeUPQI+1fqMlNhFx1FM0GDqkjFVg7NojlQ0Vt4aM5ReSqcbpaCg8nCW5lRsBvbT4T1TLfFptsfh7gItzuKTdJSEiwKSrt1vcmnEXXrsLbYnWDA1bu+z2WKy9Arq+1KRqdfKsoBo0GcdtEpS/B1bO4v0cFiUhkjskvKcMrWwtAPHuwQq8Z+4LZ1vTQANfXt4J0DwZX9gWa9qh4XDM/voC9JXfwYEMMHJcfNtusn82ihvliVUwg5KrPGVf6GH94ZJpEZBen6EC4qYTHA1dXhW0JIex8txzv//c8lhzXIi/BFxOH9jGbQhZsRalTIBZZ8KkGyZAxeRQvXkFF1TWz/Hm46jNYUnjPbt3JxIkT7f6dSj8qfJJyVvBxgaIlblOyjtysNHWN9fjjqWi7glJfW3/S0Hlj2XnA8PhKT9w6g3Qx3XiXhvuxQsuT1proxBKI/AaZqY1Xz5muvY8G8XkRRCaHsfQsRAFDH/tZPbcYuHotOG0FRIqB4HR3wNVoIPLtz8ycTguu+jpEigE218vd1YCr5m+HpHMvEI9u4LTXwNWaLjl0iPwGAmIpeHx1VeCqTJdPs1/vweweQPO3HC24NhOhnTphwoQnfv6QSY2ICbkNmdSA4h87oaLaiYfn5diIEd4att2erOwJXbPUHp953p6orQVSUVWRAXBT8c/dJ5L9xhzaJGp71GR/wFP8P5V2z10NSC9T93QM2xUg8fHxT+zU9ijeU4naHon8CjFJXFzc8/kn+dN06q9QgF98SYSo2Xen2NjYZy5sR6f+4nLSK5Iam2PH/x87a1YN/t5sBgAAAABJRU5ErkJggg==){height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/barlow-twins.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)"]}, {"cell_type": "markdown", "id": "dab2d7b1", "metadata": {"papermill": {"duration": 0.003838, "end_time": "2022-05-03T01:20:56.260406", "exception": false, "start_time": "2022-05-03T01:20:56.256568", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "3bcc2dc2", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2022-05-03T01:20:56.269677Z", "iopub.status.busy": "2022-05-03T01:20:56.269140Z", "iopub.status.idle": "2022-05-03T01:20:59.914616Z", "shell.execute_reply": "2022-05-03T01:20:59.913702Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 3.652597, "end_time": "2022-05-03T01:20:59.916987", "exception": false, "start_time": "2022-05-03T01:20:56.264390", "status": "completed"}, "tags": []}, "outputs": [], "source": ["! pip install --quiet \"matplotlib\" \"ipython[notebook]\" \"torchvision\" \"setuptools==59.5.0\" \"torch>=1.8\" \"torchmetrics>=0.7\" \"pytorch-lightning>=1.4\""]}, {"cell_type": "markdown", "id": "cefe847c", "metadata": {"papermill": {"duration": 0.004031, "end_time": "2022-05-03T01:20:59.925670", "exception": false, "start_time": "2022-05-03T01:20:59.921639", "status": "completed"}, "tags": []}, "source": ["## Barlow Twins\n", "\n", "Barlow Twins finds itself in unique place amongst the current state-of-the-art self-supervised learning methods. It does not fall under the existing categories of contrastive learning, knowledge distillation or clustering based methods. Instead, it creates its own category of redundancy reductionand achieves competitive performance with a simple yet effective loss function. In this tutorial, we look at coding up a small version of Barlow Twins algorithm using PyTorch Lightning."]}, {"cell_type": "code", "execution_count": 2, "id": "4addcae1", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:20:59.935004Z", "iopub.status.busy": "2022-05-03T01:20:59.934714Z", "iopub.status.idle": "2022-05-03T01:21:02.711834Z", "shell.execute_reply": "2022-05-03T01:21:02.711143Z"}, "papermill": {"duration": 2.784679, "end_time": "2022-05-03T01:21:02.714368", "exception": false, "start_time": "2022-05-03T01:20:59.929689", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/usr/lib/python3.8/site-packages/apex/pyprof/__init__.py:5: FutureWarning: pyprof will be removed by the end of June, 2022\n", " warnings.warn(\"pyprof will be removed by the end of June, 2022\", FutureWarning)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["WARNING:root:Bagua cannot detect bundled NCCL library, Bagua will try to use system NCCL instead. If you encounter any error, please run `import bagua_core; bagua_core.install_deps()` or the `bagua_install_deps.py` script to install bundled libraries.\n"]}], "source": ["from functools import partial\n", "from typing import Sequence, Tuple, Union\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pytorch_lightning as pl\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torchvision.transforms as transforms\n", "import torchvision.transforms.functional as VisionF\n", "from pytorch_lightning import Callback, LightningModule, Trainer\n", "from pytorch_lightning.callbacks import ModelCheckpoint\n", "from torch import Tensor\n", "from torch.utils.data import DataLoader\n", "from torchmetrics.functional import accuracy\n", "from torchvision.datasets import CIFAR10\n", "from torchvision.models.resnet import resnet18\n", "from torchvision.utils import make_grid\n", "\n", "batch_size = 32\n", "num_workers = 0 # to run notebook on CPU\n", "max_epochs = 200\n", "z_dim = 128"]}, {"cell_type": "markdown", "id": "acfe4f07", "metadata": {"papermill": {"duration": 0.004083, "end_time": "2022-05-03T01:21:02.723379", "exception": false, "start_time": "2022-05-03T01:21:02.719296", "status": "completed"}, "tags": []}, "source": ["### Transforms\n", "\n", "We first define the data augmentation pipeline used in Barlow Twins. Here, we use pipeline proposed in SimCLR, which generates two copies/views of an input image by applying the following transformations in a sequence.\n", "\n", "First it takes a random crop of the image and resizes it to a fixed pre-specified size. Then, it applies a left-to-right random flip with a probability of 0.5. This step is followed by a composition of color jitter, conversion to grayscale with a probability of 0.2 and the application of a Gaussian blur filter. Finally, we normalize the image and convert it to a tensor.\n", "\n", "Within this transform, we add a third view for our online finetuner, which we explain later on. But, to explain things quickly here, we add a another transform to perform perform test our encoder on a downstream classification task."]}, {"cell_type": "code", "execution_count": 3, "id": "0268ff9d", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:02.733060Z", "iopub.status.busy": "2022-05-03T01:21:02.732590Z", "iopub.status.idle": "2022-05-03T01:21:02.741112Z", "shell.execute_reply": "2022-05-03T01:21:02.740567Z"}, "papermill": {"duration": 0.015061, "end_time": "2022-05-03T01:21:02.742486", "exception": false, "start_time": "2022-05-03T01:21:02.727425", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class BarlowTwinsTransform:\n", " def __init__(self, train=True, input_height=224, gaussian_blur=True, jitter_strength=1.0, normalize=None):\n", "\n", " self.input_height = input_height\n", " self.gaussian_blur = gaussian_blur\n", " self.jitter_strength = jitter_strength\n", " self.normalize = normalize\n", " self.train = train\n", "\n", " color_jitter = transforms.ColorJitter(\n", " 0.8 * self.jitter_strength,\n", " 0.8 * self.jitter_strength,\n", " 0.8 * self.jitter_strength,\n", " 0.2 * self.jitter_strength,\n", " )\n", "\n", " color_transform = [transforms.RandomApply([color_jitter], p=0.8), transforms.RandomGrayscale(p=0.2)]\n", "\n", " if self.gaussian_blur:\n", " kernel_size = int(0.1 * self.input_height)\n", " if kernel_size % 2 == 0:\n", " kernel_size += 1\n", "\n", " color_transform.append(transforms.RandomApply([transforms.GaussianBlur(kernel_size=kernel_size)], p=0.5))\n", "\n", " self.color_transform = transforms.Compose(color_transform)\n", "\n", " if normalize is None:\n", " self.final_transform = transforms.ToTensor()\n", " else:\n", " self.final_transform = transforms.Compose([transforms.ToTensor(), normalize])\n", "\n", " self.transform = transforms.Compose(\n", " [\n", " transforms.RandomResizedCrop(self.input_height),\n", " transforms.RandomHorizontalFlip(p=0.5),\n", " self.color_transform,\n", " self.final_transform,\n", " ]\n", " )\n", "\n", " self.finetune_transform = None\n", " if self.train:\n", " self.finetune_transform = transforms.Compose(\n", " [\n", " transforms.RandomCrop(32, padding=4, padding_mode=\"reflect\"),\n", " transforms.RandomHorizontalFlip(),\n", " transforms.ToTensor(),\n", " ]\n", " )\n", " else:\n", " self.finetune_transform = transforms.ToTensor()\n", "\n", " def __call__(self, sample):\n", " return self.transform(sample), self.transform(sample), self.finetune_transform(sample)"]}, {"cell_type": "markdown", "id": "9ff849b7", "metadata": {"papermill": {"duration": 0.004046, "end_time": "2022-05-03T01:21:02.750613", "exception": false, "start_time": "2022-05-03T01:21:02.746567", "status": "completed"}, "tags": []}, "source": ["### Dataset\n", "\n", "We select CIFAR10 as the dataset to demonstrate the pre-training process for Barlow Twins. CIFAR10 images are 32x32 in size and we do not apply a Gaussian blur transformation on them. In this step, we create the training and validation dataloaders for CIFAR10."]}, {"cell_type": "code", "execution_count": 4, "id": "e3468fce", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:02.761326Z", "iopub.status.busy": "2022-05-03T01:21:02.760849Z", "iopub.status.idle": "2022-05-03T01:21:07.845668Z", "shell.execute_reply": "2022-05-03T01:21:07.844979Z"}, "papermill": {"duration": 5.091829, "end_time": "2022-05-03T01:21:07.847911", "exception": false, "start_time": "2022-05-03T01:21:02.756082", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./cifar-10-python.tar.gz\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "0d637445d6d8411297c8497dd704eb7f", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/170498071 [00:00"]}, "metadata": {}, "output_type": "display_data"}], "source": ["for batch in val_loader:\n", " (img1, img2, _), label = batch\n", " break\n", "\n", "img_grid = make_grid(img1, normalize=True)\n", "\n", "\n", "def show(imgs):\n", " if not isinstance(imgs, list):\n", " imgs = [imgs]\n", " fix, axs = plt.subplots(ncols=len(imgs), squeeze=False)\n", " for i, img in enumerate(imgs):\n", " img = img.detach()\n", " img = VisionF.to_pil_image(img)\n", " axs[0, i].imshow(np.asarray(img))\n", " axs[0, i].set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])\n", "\n", "\n", "show(img_grid)"]}, {"cell_type": "markdown", "id": "1ebdbd2b", "metadata": {"papermill": {"duration": 0.005454, "end_time": "2022-05-03T01:21:08.067902", "exception": false, "start_time": "2022-05-03T01:21:08.062448", "status": "completed"}, "tags": []}, "source": ["### Barlow Twins Loss\n", "\n", "Here we define the loss function for Barlow Twins. It first normalizes the D dimensinonal vectors from the projection head and then computes the DxD cross-correlation matrix between the normalized vectors of the 2 views of each image.\n", "\n", "Then it splits this cross-correlation matrix into two parts. The first part, the diagonal of this matrix is brought closer to 1, which pushes up the cosine similarity between the latent vectors of two views of each image, thus making the backbone invariant to the transformations applied to the views. The second part of the loss pushes the non-diagonal elements of the cross-corrlelation matrix closes to 0. This reduces the redundancy between the different dimensions of the latent vector."]}, {"cell_type": "code", "execution_count": 6, "id": "06e4e4c0", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.080401Z", "iopub.status.busy": "2022-05-03T01:21:08.080118Z", "iopub.status.idle": "2022-05-03T01:21:08.087147Z", "shell.execute_reply": "2022-05-03T01:21:08.086522Z"}, "papermill": {"duration": 0.015233, "end_time": "2022-05-03T01:21:08.088681", "exception": false, "start_time": "2022-05-03T01:21:08.073448", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class BarlowTwinsLoss(nn.Module):\n", " def __init__(self, batch_size, lambda_coeff=5e-3, z_dim=128):\n", " super().__init__()\n", "\n", " self.z_dim = z_dim\n", " self.batch_size = batch_size\n", " self.lambda_coeff = lambda_coeff\n", "\n", " def off_diagonal_ele(self, x):\n", " # taken from: https://github.com/facebookresearch/barlowtwins/blob/main/main.py\n", " # return a flattened view of the off-diagonal elements of a square matrix\n", " n, m = x.shape\n", " assert n == m\n", " return x.flatten()[:-1].view(n - 1, n + 1)[:, 1:].flatten()\n", "\n", " def forward(self, z1, z2):\n", " # N x D, where N is the batch size and D is output dim of projection head\n", " z1_norm = (z1 - torch.mean(z1, dim=0)) / torch.std(z1, dim=0)\n", " z2_norm = (z2 - torch.mean(z2, dim=0)) / torch.std(z2, dim=0)\n", "\n", " cross_corr = torch.matmul(z1_norm.T, z2_norm) / self.batch_size\n", "\n", " on_diag = torch.diagonal(cross_corr).add_(-1).pow_(2).sum()\n", " off_diag = self.off_diagonal_ele(cross_corr).pow_(2).sum()\n", "\n", " return on_diag + self.lambda_coeff * off_diag"]}, {"cell_type": "markdown", "id": "af086bbe", "metadata": {"papermill": {"duration": 0.005637, "end_time": "2022-05-03T01:21:08.100125", "exception": false, "start_time": "2022-05-03T01:21:08.094488", "status": "completed"}, "tags": []}, "source": ["### Backbone\n", "\n", "This is a standard Resnet backbone that we pre-train using the Barlow Twins method. To accommodate the 32x32 CIFAR10 images, we replace the first 7x7 convolution of the Resnet backbone by a 3x3 filter. We also remove the first Maxpool layer from the network for CIFAR10 images."]}, {"cell_type": "code", "execution_count": 7, "id": "123382f1", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.113314Z", "iopub.status.busy": "2022-05-03T01:21:08.112701Z", "iopub.status.idle": "2022-05-03T01:21:08.268899Z", "shell.execute_reply": "2022-05-03T01:21:08.268104Z"}, "papermill": {"duration": 0.16539, "end_time": "2022-05-03T01:21:08.271128", "exception": false, "start_time": "2022-05-03T01:21:08.105738", "status": "completed"}, "tags": []}, "outputs": [], "source": ["encoder = resnet18()\n", "\n", "# for CIFAR10, replace the first 7x7 conv with smaller 3x3 conv and remove the first maxpool\n", "encoder.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)\n", "encoder.maxpool = nn.MaxPool2d(kernel_size=1, stride=1)\n", "\n", "# replace classification fc layer of Resnet to obtain representations from the backbone\n", "encoder.fc = nn.Identity()"]}, {"cell_type": "markdown", "id": "f440f360", "metadata": {"papermill": {"duration": 0.005551, "end_time": "2022-05-03T01:21:08.283411", "exception": false, "start_time": "2022-05-03T01:21:08.277860", "status": "completed"}, "tags": []}, "source": ["### Projection head\n", "\n", "Unlike SimCLR and BYOL, the downstream performance of Barlow Twins greatly benefits from having a larger projection head after the backbone network. The paper utilizes a 3 layer MLP with 8192 hidden dimensions and 8192 as the output dimenion of the projection head. For the purposes of the tutorial, we use a smaller projection head. But, it is imperative to mention here that in practice, Barlow Twins needs to be trained using a bigger projection head as it is highly sensitive to its architecture and output dimensionality."]}, {"cell_type": "code", "execution_count": 8, "id": "2975b668", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.296007Z", "iopub.status.busy": "2022-05-03T01:21:08.295469Z", "iopub.status.idle": "2022-05-03T01:21:08.300257Z", "shell.execute_reply": "2022-05-03T01:21:08.299653Z"}, "papermill": {"duration": 0.012837, "end_time": "2022-05-03T01:21:08.301778", "exception": false, "start_time": "2022-05-03T01:21:08.288941", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ProjectionHead(nn.Module):\n", " def __init__(self, input_dim=2048, hidden_dim=2048, output_dim=128):\n", " super().__init__()\n", "\n", " self.projection_head = nn.Sequential(\n", " nn.Linear(input_dim, hidden_dim, bias=True),\n", " nn.BatchNorm1d(hidden_dim),\n", " nn.ReLU(),\n", " nn.Linear(hidden_dim, output_dim, bias=False),\n", " )\n", "\n", " def forward(self, x):\n", " return self.projection_head(x)"]}, {"cell_type": "markdown", "id": "86a6d540", "metadata": {"papermill": {"duration": 0.005677, "end_time": "2022-05-03T01:21:08.313096", "exception": false, "start_time": "2022-05-03T01:21:08.307419", "status": "completed"}, "tags": []}, "source": ["### Learning rate warmup\n", "\n", "For the purposes of this tutorial, we keep things simple and use a linear warmup schedule with Adam optimizer. In our previous experiments we have found that linear warmup part is much more important for the final performance of a model than the cosine decay component of the schedule."]}, {"cell_type": "code", "execution_count": 9, "id": "56a7b0b7", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.325323Z", "iopub.status.busy": "2022-05-03T01:21:08.324922Z", "iopub.status.idle": "2022-05-03T01:21:08.328636Z", "shell.execute_reply": "2022-05-03T01:21:08.328021Z"}, "papermill": {"duration": 0.011487, "end_time": "2022-05-03T01:21:08.330091", "exception": false, "start_time": "2022-05-03T01:21:08.318604", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def fn(warmup_steps, step):\n", " if step < warmup_steps:\n", " return float(step) / float(max(1, warmup_steps))\n", " else:\n", " return 1.0\n", "\n", "\n", "def linear_warmup_decay(warmup_steps):\n", " return partial(fn, warmup_steps)"]}, {"cell_type": "markdown", "id": "f2c9250b", "metadata": {"papermill": {"duration": 0.005561, "end_time": "2022-05-03T01:21:08.341357", "exception": false, "start_time": "2022-05-03T01:21:08.335796", "status": "completed"}, "tags": []}, "source": ["### Barlow Twins Lightning Module\n", "\n", "We keep the LightningModule for Barlow Twins neat and simple. It takes in an backbone encoder and initializes the projection head and the loss function. We configure the optimizer and the learning rate scheduler in the ``configure_optimizers`` method."]}, {"cell_type": "code", "execution_count": 10, "id": "cab91fad", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.354107Z", "iopub.status.busy": "2022-05-03T01:21:08.353556Z", "iopub.status.idle": "2022-05-03T01:21:08.362090Z", "shell.execute_reply": "2022-05-03T01:21:08.361493Z"}, "papermill": {"duration": 0.016575, "end_time": "2022-05-03T01:21:08.363538", "exception": false, "start_time": "2022-05-03T01:21:08.346963", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class BarlowTwins(LightningModule):\n", " def __init__(\n", " self,\n", " encoder,\n", " encoder_out_dim,\n", " num_training_samples,\n", " batch_size,\n", " lambda_coeff=5e-3,\n", " z_dim=128,\n", " learning_rate=1e-4,\n", " warmup_epochs=10,\n", " max_epochs=200,\n", " ):\n", " super().__init__()\n", "\n", " self.encoder = encoder\n", " self.projection_head = ProjectionHead(input_dim=encoder_out_dim, hidden_dim=encoder_out_dim, output_dim=z_dim)\n", " self.loss_fn = BarlowTwinsLoss(batch_size=batch_size, lambda_coeff=lambda_coeff, z_dim=z_dim)\n", "\n", " self.learning_rate = learning_rate\n", " self.warmup_epochs = warmup_epochs\n", " self.max_epochs = max_epochs\n", "\n", " self.train_iters_per_epoch = num_training_samples // batch_size\n", "\n", " def forward(self, x):\n", " return self.encoder(x)\n", "\n", " def shared_step(self, batch):\n", " (x1, x2, _), _ = batch\n", "\n", " z1 = self.projection_head(self.encoder(x1))\n", " z2 = self.projection_head(self.encoder(x2))\n", "\n", " return self.loss_fn(z1, z2)\n", "\n", " def training_step(self, batch, batch_idx):\n", " loss = self.shared_step(batch)\n", " self.log(\"train_loss\", loss, on_step=True, on_epoch=False)\n", " return loss\n", "\n", " def validation_step(self, batch, batch_idx):\n", " loss = self.shared_step(batch)\n", " self.log(\"val_loss\", loss, on_step=False, on_epoch=True)\n", "\n", " def configure_optimizers(self):\n", " optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n", "\n", " warmup_steps = self.train_iters_per_epoch * self.warmup_epochs\n", "\n", " scheduler = {\n", " \"scheduler\": torch.optim.lr_scheduler.LambdaLR(\n", " optimizer,\n", " linear_warmup_decay(warmup_steps),\n", " ),\n", " \"interval\": \"step\",\n", " \"frequency\": 1,\n", " }\n", "\n", " return [optimizer], [scheduler]"]}, {"cell_type": "markdown", "id": "c3035885", "metadata": {"papermill": {"duration": 0.005617, "end_time": "2022-05-03T01:21:08.374836", "exception": false, "start_time": "2022-05-03T01:21:08.369219", "status": "completed"}, "tags": []}, "source": ["### Evaluation\n", "\n", "We define a callback which appends a linear layer on top of the encoder and trains the classification evaluation head in an online manner. We make sure not to backpropagate the gradients back to the encoder while tuning the linear layer. This technique was used in SimCLR as well and they showed that the final downstream classification peformance is pretty much similar to the results on online finetuning as the training progresses."]}, {"cell_type": "code", "execution_count": 11, "id": "e1321f73", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.387481Z", "iopub.status.busy": "2022-05-03T01:21:08.387178Z", "iopub.status.idle": "2022-05-03T01:21:08.397504Z", "shell.execute_reply": "2022-05-03T01:21:08.396907Z"}, "papermill": {"duration": 0.018391, "end_time": "2022-05-03T01:21:08.398964", "exception": false, "start_time": "2022-05-03T01:21:08.380573", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class OnlineFineTuner(Callback):\n", " def __init__(\n", " self,\n", " encoder_output_dim: int,\n", " num_classes: int,\n", " ) -> None:\n", " super().__init__()\n", "\n", " self.optimizer: torch.optim.Optimizer\n", "\n", " self.encoder_output_dim = encoder_output_dim\n", " self.num_classes = num_classes\n", "\n", " def on_fit_start(self, trainer: pl.Trainer, pl_module: pl.LightningModule) -> None:\n", "\n", " # add linear_eval layer and optimizer\n", " pl_module.online_finetuner = nn.Linear(self.encoder_output_dim, self.num_classes).to(pl_module.device)\n", " self.optimizer = torch.optim.Adam(pl_module.online_finetuner.parameters(), lr=1e-4)\n", "\n", " def extract_online_finetuning_view(\n", " self, batch: Sequence, device: Union[str, torch.device]\n", " ) -> Tuple[Tensor, Tensor]:\n", " (_, _, finetune_view), y = batch\n", " finetune_view = finetune_view.to(device)\n", " y = y.to(device)\n", "\n", " return finetune_view, y\n", "\n", " def on_train_batch_end(\n", " self,\n", " trainer: pl.Trainer,\n", " pl_module: pl.LightningModule,\n", " outputs: Sequence,\n", " batch: Sequence,\n", " batch_idx: int,\n", " dataloader_idx: int,\n", " ) -> None:\n", " x, y = self.extract_online_finetuning_view(batch, pl_module.device)\n", "\n", " with torch.no_grad():\n", " feats = pl_module(x)\n", "\n", " feats = feats.detach()\n", " preds = pl_module.online_finetuner(feats)\n", " loss = F.cross_entropy(preds, y)\n", "\n", " loss.backward()\n", " self.optimizer.step()\n", " self.optimizer.zero_grad()\n", "\n", " acc = accuracy(F.softmax(preds, dim=1), y)\n", " pl_module.log(\"online_train_acc\", acc, on_step=True, on_epoch=False)\n", " pl_module.log(\"online_train_loss\", loss, on_step=True, on_epoch=False)\n", "\n", " def on_validation_batch_end(\n", " self,\n", " trainer: pl.Trainer,\n", " pl_module: pl.LightningModule,\n", " outputs: Sequence,\n", " batch: Sequence,\n", " batch_idx: int,\n", " dataloader_idx: int,\n", " ) -> None:\n", " x, y = self.extract_online_finetuning_view(batch, pl_module.device)\n", "\n", " with torch.no_grad():\n", " feats = pl_module(x)\n", "\n", " feats = feats.detach()\n", " preds = pl_module.online_finetuner(feats)\n", " loss = F.cross_entropy(preds, y)\n", "\n", " acc = accuracy(F.softmax(preds, dim=1), y)\n", " pl_module.log(\"online_val_acc\", acc, on_step=False, on_epoch=True, sync_dist=True)\n", " pl_module.log(\"online_val_loss\", loss, on_step=False, on_epoch=True, sync_dist=True)"]}, {"cell_type": "markdown", "id": "f6325fed", "metadata": {"papermill": {"duration": 0.005584, "end_time": "2022-05-03T01:21:08.410318", "exception": false, "start_time": "2022-05-03T01:21:08.404734", "status": "completed"}, "tags": []}, "source": ["Finally, we define the trainer for training the model. We pass in the ``train_loader`` and ``val_loader`` we had initialized earlier to the ``fit`` function."]}, {"cell_type": "code", "execution_count": 12, "id": "3d172029", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.422975Z", "iopub.status.busy": "2022-05-03T01:21:08.422470Z", "iopub.status.idle": "2022-05-03T01:21:08.657523Z", "shell.execute_reply": "2022-05-03T01:21:08.656922Z"}, "papermill": {"duration": 0.243001, "end_time": "2022-05-03T01:21:08.659057", "exception": false, "start_time": "2022-05-03T01:21:08.416056", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["ModelCheckpoint(save_last=True, save_top_k=-1, monitor=None) will duplicate the last checkpoint saved.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}], "source": ["encoder_out_dim = 512\n", "\n", "model = BarlowTwins(\n", " encoder=encoder,\n", " encoder_out_dim=encoder_out_dim,\n", " num_training_samples=len(train_dataset),\n", " batch_size=batch_size,\n", " z_dim=z_dim,\n", ")\n", "\n", "online_finetuner = OnlineFineTuner(encoder_output_dim=encoder_out_dim, num_classes=10)\n", "checkpoint_callback = ModelCheckpoint(every_n_epochs=100, save_top_k=-1, save_last=True)\n", "\n", "trainer = Trainer(\n", " max_epochs=max_epochs,\n", " accelerator=\"auto\",\n", " devices=1 if torch.cuda.is_available() else None, # limiting got iPython runs\n", " callbacks=[online_finetuner, checkpoint_callback],\n", ")\n", "\n", "# uncomment this to train the model\n", "# this is done for the tutorial so that the notebook compiles\n", "# trainer.fit(model, train_loader, val_loader)"]}, {"cell_type": "markdown", "id": "8bd09d4d", "metadata": {"papermill": {"duration": 0.005851, "end_time": "2022-05-03T01:21:08.671789", "exception": false, "start_time": "2022-05-03T01:21:08.665938", "status": "completed"}, "tags": []}, "source": ["### Using the trained encoder for downstream tasks\n", "\n", "Once the encoder is pretrained on CIFAR10, we can use it to get image embeddings and use them further downstream on tasks like classification, detection, segmentation etc.\n", "\n", "In this tutorial, we did not completely train our encoder for 100s of epochs using the Barlow Twins pretraining method. So, we will load the pretrained encoder weights from a checkpoint and show the image embeddings obtained from that.\n", "\n", "To create this checkpoint, the encoder was pretrained for 200 epochs, and obtained a online finetune accuracy of x% on CIFAR-10."]}, {"cell_type": "code", "execution_count": 13, "id": "f2877d1f", "metadata": {"execution": {"iopub.execute_input": "2022-05-03T01:21:08.685209Z", "iopub.status.busy": "2022-05-03T01:21:08.684742Z", "iopub.status.idle": "2022-05-03T01:21:09.080222Z", "shell.execute_reply": "2022-05-03T01:21:09.079567Z"}, "papermill": {"duration": 0.404237, "end_time": "2022-05-03T01:21:09.081976", "exception": false, "start_time": "2022-05-03T01:21:08.677739", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["torch.Size([4, 512])\n"]}], "source": ["# ckpt_model = torch.load('') # upload checkpoint to aws\n", "# encoder = ckpt_model.encoder\n", "encoder = model.encoder\n", "\n", "downstream_dataset = CIFAR10(root=\".\", train=False, transform=transforms.ToTensor())\n", "dataloader = DataLoader(downstream_dataset, batch_size=4, shuffle=False)\n", "\n", "for batch in dataloader:\n", " img, label = batch\n", " print(encoder(img).shape)\n", " break"]}, {"cell_type": "markdown", "id": "d35cf819", "metadata": {"papermill": {"duration": 0.005993, "end_time": "2022-05-03T01:21:09.094796", "exception": false, "start_time": "2022-05-03T01:21:09.088803", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://www.pytorchlightning.ai/community)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "[![Pytorch Lightning](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAByMAAAE8CAYAAABjFR0gAAAACXBIWXMAAAsSAAALEgHS3X78AAAgAElEQVR4nOzdT2ydV3rn+eP3FQLYoVJsZFEgPU4pjYDCAO5QobxLai6dypKKlFQ2Mroj2inImwnEkrwZIl2kqgOtJJWcrGTUlKikoZpFqkr+s0zGZLuzk1xiV23kDiZyKkVNLYKRIkWFALqvB7/X5zBH9+Wfe8lz3r/fD0BI5qXFe+/73vc95zzneZ7nPvvsMwNsZ3F6Y9YYM2sfvmeMWb2wPnlvmx8HAAAAAAAAAAAANhGMRMHi9Ma4MWbBfn2h8APGvGuMWb6wPnmn8AgAAAAAAAAAAABgEYzEMxanN+YVaDTGfGmId+a6ApYX1icfFB4BAAAAAAAAAABA5xGMRG5xeuOIMeaKMaY34jvyUP/fhfXJ5cIjAAAAAAAAAAAA6DSCkR1nS7IqCHlqn+/Ep8aY+Qvrk6uFRwAAAAAAAAAAANBJBCM7bHF6Y3mHvpB7tWaDkve6/v4CAAAAAAAAAAB0HcHIDlqc3pg1xqwM2Rdyr95W70n6SQIAAAAAAAAAAHQXwcgOWZzeOGSDkKP2hdwr9ZNcuLA+udL19x4AAAAAAAAAAKCLCEZ2gO0LqZKsZyp6tes2KEk/SQAAAAAAAAAAgA4hGNlyi9MbCzYQGbIv5F69a4OS9JMEAAAAAAAAAADoAIKRLWX7Ql4xxkzX7BU+tM/rCv0kAQAAAAAAAAAA2o1gZMvYvpAK9h2v+Sv7VBmb9JMEAAAAAAAAAABoL4KRLWH7Qqok61LDXtGaLd16p/AIAAAAAAAAAAAAGo1gZAssTm/M276QX2rwq7lug5KUbgUAAAAAAAAAAGgJgpENZvtCKgjZa8lLemh7SS4XHgEAAAAAAAAAAEDjEIxsIFuSVX0hT7X0Jaqf5PyF9cnVwiMAAAAAAAAAAABoDIKRDbM4vbFse0N+oQMvd80GJe8VHgEAAAAAAAAAAEDtEYxsiMXpjRM2G7LJfSH36m2Vo6WfJAAAAAAAAAAAQLMQjKy5xemNIzYI2Za+kHv10AYkrzTz6QMAAAAAAAAAAHQPwciasn0hVZL1TNffiwHrKlNLP0kAAAAAAAAAAID6IxhZQ4vTGws2ENmFvpB79a4NStJPEgAAAAAAAAAAoKYIRtbI4vTGrDFmpaN9IffqvMrY0k8SAAAAAAAAAACgfghG1sDi9MYh2xfyeNffiz361PaTXGnkswcAAAAAAAAAAGgpgpEVsn0hVZJ1qbNvQlhrNihJP0kAAAAAAAAAAIAaIBhZkcXpjXmbDUlfyPCu236SlG4FAAAAAAAAAACoEMHIktm+kApCTnfqhZfvoe0ludy1Fw4AAAAAAAAAAFAXBCNLYvtCKjB2qhMvuD4+tVmSN7v+RgAAAAAAAAAAAJSNYGRkXl/IBUqyVmrNBiXvdPg9AAAAAAAAAAAAKBXByIhsX0hlQ36ptS+yed7WMaGfJAAAAAAAAAAAQHwEIyNYnN44YvtC9lr34trhoQ1IXun6GwEAAAAAAAAAABATwciAbEnWK/SFbAz1k5y/sD652vU3AgAAAAAAAAAAIAaCkYEsTm8s0xeysd61/STvdf2NAAAAAAAAAAAACIlg5D4tTm+csNmQ9IVsvvM6lvSTBAAAAAAAAAAACINg5B7RF7K1HtosyZWuvxEAAAAAAAAAAAD7RTByRLYvpEqynmnUE8eo1m1Qkn6SAAAAAAAAAAAAe0QwcgSL0xsLNhBJX8juuK5jTj9JAAAAAAAAAACA0RGMHMLi9MasMWaFvpCd9dCW5KWfJAAAAAAAAAAAwAgIRu5gcXrjkA1C0hcS8qnNkqSfJAAAAAAAAAAAwBAIRm6BvpDYxZrtJ3ln5x8DAAAAAAAAAADoNoKRAxanN+ZtSU76QmI3121QktKtAAAAAAAAAAAAWyAYadm+kApCThceBLb30PaSXN72JwAAAAAAAAAAADqq88FI2xdSQcjjhQeB4X1qsyRv8p4BAAAAAAAAAAB8rrPBSNsXcsEYs1R4ENg79ZOcv7A+eY/3EAAAAAAAAAAAdF0ng5H0hUQJ3jbGLNNPEgAAAAAAAAAAdFmngpG2L6R6+/UKDwLhPbQBySu8twAAAAAAAAAAoIs6EYy0fSEVhDxVeBCI71NbunWV9xoAAAAAAAAAAHRJq4ORXl/IBUqyogbe1blIP0kAAAAAAAAAANAVrQ1G2r6Qyob8UuFBoFrn1bOUfpIAAAAAAAAAAKDtWheMpC8kGuKhzZJc4YABAAAAAAAAAIC2ak0wkr6QaKh1G5SknyQAAAAAAAAAAGidVgQjF6c3lukLiYajnyQAAAAAAAAAAGidRgcjF6c3Tqj3Hn0h0RIP7flMP0kAAAAAAAAAANAKjQxGLk5vHLFBG/pCoo0+Vclh+kkCAAAAAAAAAICma1QwcnF6Y9wGIekLiS5Ys0FJ+kkCAAAAAAAAAIBGakwwkr6Q6LDrtp8kpVsBAAAAAAAAAECj1D4YSV9IIPfQ9pJc5u0AAAAAAAAAAABNUdtgJH0hgS19arMkb271IAAAAAAAAAAAQJ3ULhhJX0hgKGs2KHmHtwsAAAAAAAAAANRVrYKRi9Mb6gm5TF9IYGj0kwQAAAAAAAAAALVVi2Dk4vTGrDFmhb6QwJ6on+TyhfXJK7x9AAAAAAAAAACgTioNRi5ObxyyQUj6QgL7p36S8xfWJ1d5LwEAAAAAAAAAQB1UEoy0fSFVjvVM4UEA+7Vmg5L3eCcBAAAAAAAAAECVSg9G0hcSKM3btnwr/SQBAAAAAAAAAEAlSgtG0hcSqAT9JAEAAAAAAAAAQGWiByPpCwnUAv0kAQAAAAAAAABA6aIFI+kLCdTSu8aYBfpJAgAAAAAAAACAMkQJRtIXEqi988aYK/STBAAAAAAAAAAAMQUNRtIXEmiUhzZLcoXDBgAAAAAAAAAAYggSjKQvJNBo6zYoST9JAAAAAAAAAAAQ1L6CkfSFBFqFfpIAAAAAAAAAACCoPQcjF6c35tVzjr6QQOvQTxIAAAAAAAAAAAQxcjDS9oVUEHK68CCAtvhUWc/0kwQAAAAAAAAAAPsxdDDS9oVUEPJ44UEAbbVmg5L0kwQAAAAAAAAAACPbNRhp+0IuGGOWCg8C6IrrNihJP0kAAAAAAAAAADC0HYOR9IUE4Hlorwf0kwQAAAAAAAAAAEPZMhhJX0gAO6CfJAAAAAAAAAAAGMozwUj6QgIYgfpJLlxYn7zDmwYAAAAAAAAAALayGYxcnN44YYxZoSQrgBG9TpYkAAAAAAAAAADYSh6MtGVZP9zicQAYBgFJAAAAAAAAAABQ8Nz/8es/HTfGqMzilwqPAuVYpz9p4z00xhy6sD75oOtvBAAAAAAAAAAA+DeJMeYEgUhU6PqF9ckjxphPOQiN9gV7LQEAAAAAAAAAANiUEEBAhRSInF+c3jhEQLwVZrv+BgAAAAAAAAAAgGcpGDle+C4QXx6ItL/lCO93Kxzq+hsAAAAAAAAAAACepWBkr/BdIC4/EGnIqAMAAAAAAAAAAGinhOOKkg0GIg3BSAAAAAAAAAAAgHYiGIkybRWIlOnCdwAAAAAAAAAAANB4BCNRli0DkYvTG2RFAgAAAAAAAAAAtBTBSJRhy0CkRTASAAAAAAAAAACgpQhGIradApFypPAdAAAAAAAAAAAAtALBSMS0WyDSkBkJAAAAAAAAAADQXgQjEcuugcjF6Q1lRX6h8AAAAAAAAAAAAABagWAkYhgmI9JQohUAAAAAAAAAAKDdCEYitPUhA5GGEq0AAAAAAAAAAADtRjASIa2PGGAkGAkAAAAAAAAAANBiBCMRSh6IvLA++WCYf29xemPcGPOlwgMAAAAAAAAAAABoDYKRCGGkQKRFViQAAAAAAAAAAEDLEYzEfu0lEClHCt8BAAAAAAAAAABAqxCMxH7sNRBpyIwEAAAAAAAAAABoP4KR2Kv9BCKlV/gOAAAAAAAAAAAAWoVgJPZiX4HIxekNsiIBAAAAAAAAAAA6gGAkRrXfjEhDv0gAAAAAAAAAAIBuUDByjWONIYUIRBr6RQIAAAAAAAAAAHTDAY4zhhQqEGkIRgIAAAAAdnHIfsWyygEY2nKkf3fFGHOv8F10gSpmjUd6nXeMMSHWrgA0R8wxA9cUIBCCkRhGsEDk4vSGBpxfKDwAYC/0eTphA/y9hr6DO2XnuwWiB3bwd4/Fik6Yt19ongX7WQXQzWt3rAUgghXdpfNqKeKrf67wHWwn1nFY5fPdWVcizmFfZbMB0DkxxwxcU4BACEZiN58GzIg09IsEgjrhBlv379/Pv5pkbGzMTE1N7TQBLTyWZdnjJElu24HgKgPCVjq01bFHI8Ta3Q6g/uYjXrsJVnTcBx98YDY2NoK9CceOHTMTExOF72Nnn3zyibl06dKOPzPKMZibmyt8H93zzjvvBHvNr7zyipmZmSl8H0B3hBwzcE0BwiMYiZ08VLAjYCDSUKIVCO/NN980t2/fbtU7e/DgQQUq878fPnw4D1xqIDgxMTE2MTHRswuebtfbu8aYm/aL0hkt0cbzuq1Onz6dfwGArt2hEKyA8/777wcdE9gxZeH72Nnjx4+DHYejR48WvoduChmMFAIHQLeFHjNwTQHCIhiJ7Ty0GZGhy60RjASwq0ePHm0OIN2fbqI6OTmZL2C4r4mJiePGGH1dM8Zct+XcyJgEAKBkIRd/CFYAAAAAQHsQjMRWogQiF6c3VHrvS4UHAGAEKrmhL+14E2VQKntCX2NjY6eMMaf6/f6P0zS9ZAOTAAAAAAAAAICKJLzxGBArI9LQLxIILv9MNa1XZGiuf83s7Kw5f/68+fjjj02api8rUzLLsn+wvTUBAAAAAAAAABUgGAlfzECkoUQrENy4sZmC+JyyJdW7Tj2rFJRMkuQlY8wPbNlWNkQAAAAAAAAAQMkIRsKJHYg0BCMBlEU9q/ygpDGmZ4z5oTFmmYMAAAAAAAAAAOUhGAlTRiBycXpDGVzThQcAICIXlLx8+bJ5/PixftFSv9//EVmSAAAAAABUapy2KgDQHQQjUUZGpGHhH0CVbty4YY4dO2bW1tbyfpJZln1kjJnnoAAAAAAAUCrNxW8aY/4/Y8wCbz0AdAPByG4rKxBpKNEKoGqPHj0y586dy7MkkyQZM8ZcM8ascGAAAAAAAIhKGZArWZY9snPx47zdANAtBCO7q8xApCEYCaAulCX52muvubKtp+yOzHEOEAAAAAAAwahK2pV+v/+PxpgfaP795MmTsQ8++CCfkwMAuuUAx7uTyg5ESq/wHQCoyCeffJKXbb169aqZmpo63u/3P0rT9MvGmAccEwAAAAAA9uSQsiCzLDubJMlL+gfSNDUKQK6uruZfAIBuIhjZPaUHIhenN8iKBFA7Ktv65ptvuoDkywQkAQAAAADYk3lbijUvv5okiVlbW9sMQGr+DQDoNoKR3bNQckaksWUZAKB2BgOSxphVrlkAAAAAAOxKWZDzNgvyoH74/v375rvf/W4egNzY2OAdBABsIhjZLa9fWJ9cqeAVkxkJoLYGApLTaqpvd3UCAAAAAIBnjasXpHpAGpsFqTKs77//vrl9+3bhhwEAMAQjO6WqQKQhGAmg7gYCkppQ3TPGLHPgAAAAAADYNG5bnLzssiAVhKQMKwBgN8kuj6MdKgtELk5vqNzhFwoPAEDNaPK0vLxsHj9+rCe2xEYKAAAAAACesaBApPpBHjt2zNy4cYNAJABgKAQj26/KjEjDYj6AJvnkk0/ygKRkWfaeLT8DAAAAAEDnqT+kNvC6eTMAAMMiGNluVQci5UjhOwBQY2q0r1IztgF/1ddQAAAAAADqYFbzZM2ZyYYEAIyKnpHtVYdApCEzEkATXbp0yRw9elT9I48bY04YY25yIFGmqakpc/DgwUa855OTk4XvAQAAAGidPOHg9u3bHFnU0uHDh4M9Lea5QHgEI9upFoHIxemNQ8aYLxUeAFB76v0wMTGx7dNUkGS3QZ7+/+3+jY8//rjwvbt3727urnSTmyonOQpIXr16VWVo/iJJkl8xxjwo/BCiOXfunOvfWXs6d3W+hPTWW2+ZmZkZTjAAAFALGpfcunWLgwF0W97GZGNjo+vvA2rq7NmzHBqgxghGtk9dMiINWZFAcykYGSgQsjb4jX6//8szMzMvD35/u9+nPo4KSupL5WDKot+ncq0nT55UetqCMYamGCVSZiBy5xv0NtwrfAcAALRBYUwfCJv9AAAhxFwsYp4LBEIwsl3qFIg0BCOBVnh1hxexp8FemqaF71numjFuy78ou/rQ1NRUT4GpkydPmvv375t33nnHvP/++4X/OQb9LgVmn3/++bfSNF1hEFqK5YYFfj8rfCcsguAAAKBqzO0BAHW2GjkgCSCAhDexNeoWiDRMWIBWWN3hKzT3u27aAMy8vY78O2PM7xlj3lXZ16WlpTwYOTsb/xKjsrE3btxQAPUXCQoBAAAAAAAAwOgIRrZD7QKR9IsEENADG6A8YYz5VWPMdQUlL168aJaXl/P+lTGpVKt6F/b7/T9wPTIAAAAAAAAAAMMhGNl852uYEWnIigQQyT2bMflqlmU/mZubM1evXo0akBzIjlwo/AAAAAAAAAAAYFsEI5vt+oX1ybqWDSQYCSCm1SRJft0Ys6Z+krEDkh988EH+Z7/f/1rhQQAAAAAAAADAtghGNpcCkfM1fvYEIwHE9sBea667gGQsGxsbZm1tTdmRL9pysQAAAAAAAACAIRCMbKZaByLpFwmgZPMuQ/LcuXPRfvP777/v/kowEgAAAAAAAACGRDCyeeqeEWnIigRQgRPqIXny5Elz9OjRKL99dXXVPH782GRZ9vuFBwEAAAAAAAAAWyIY2SxNCEQagpEAKvAgSZI/1K9dXo7XSlcBySRJDnKdAwAAAAAAAIDhEIxsjqYEIg2L9AAqsqpr5cTEhDl9+nSUZ3D79m33V65zAAAAAAAAADAEgpHN0KRApKFfJIAK5WmRr732WpRnQDASAAAAAAAAAEZDMLL+mhaIBIAq3dN1c2xszBw7diz409jY2DD379/XX3uFBwEAAAAAAAAABQQj641AJACM7or+j9nZOMmLn3zyifvrkcKDAAAAAAAAAIBnEIysryYHIh8WvgMA5bmTZdlPer2eOXjwYPBfevfuXffX8cKDAAAAAIA2GbdtOga/2JwKAMAIDvBm1VLTMyJvGmNOFb4LACVJkuT7xpgzyo58//33g/5SLzNSE9DVwg8AKItLfx5Mg/b/e/AzurrN94FQZu2ipVugHN9msfKOMebBwN85L+M5Yr8OuWtEv9//5TRNXx74jev2WDywx8V93WvRewEgnnHvejPuXW9+LU3TF7f5re66c89+rQ7cI1AuF3A8kmXZTJIkL+3227Mse5wkye2B+8adwg8CALC73eaS/pyxceMFgpH1824LSrMuE4wEUDFtijgzNTUV/Fk8evSo8D0AUbkBeL44tMuC3qDB/q5L7i/9fv+naZresgP41ZYEgma9YOxWE5f9WNjHwpoLArnnc8h+bWdwUvXAXtfrFhDyz8tXRjgvzRbnZs47L1e9BWmMTufXCfu15Xudpqn5+OOPN/9b/aanpqamvR857v6SZdkju9Fp1Z6LBAnQJaHvJ757DQ/2H/LuA7+z3X3g5z//ub+hcdPAdcddq/KxSr/f/3Gapn/jXXdGeU473WP3o+nHaztagzuRZdlXkiQZcz+TJMnmfeLWrVuF//Xw4cN5JZ6JiYmxiYmJnn+/8e4bN0c8fjGEPCdinFtHShiHx+nhEs7KHj5bux3X7TZltnWzQ8xrX9Pes+XCd3Z3ZIfqX1td+1d3eAzDc+MIt5Fpy3nLgKXCd4xZs8fhjre+sVezEa+Z9whG1su6HQQ12oX1yXuL0xtvKxDQ9QMKoDL5jVcTxNAGMiMHaSDx94Xv7oNdmP47+y/cswGBNkwe9F4tZ1n2+0mShK+na8yvMihutG0DCW5BTyWTtTlAf/c3Cbj/1uKQvyFhcnJSi0X5n/qamZnRguGLNuDgBvTveot+TTx/ZreZnISw3eR0K7P22OnP6S0e381Wk7DxPU6sQzthFyyfuXa5wNbGxkb+5Z+Xg+eoHD16NP/Tnaf6U/esgfPSZVt8jyDYUHSOzPf7/T/yMx7v379vbt++vXnd0H/rGO3EXS90nHRcjh49enBsbOyU3XB5zV4rrpDNio7Q4tiHkV7q+Zpc20eRj1EGrzXuPqDrjK43g/eC3eh64+4Jr7zyiu4H+rf1dcYGtr5jrzu7jU8WIq7FNPF4bUfHcSHLsjfc/fxnP/tZfr9wX7vdKwZ594xn7hvevXylovvGfMTxYQhf2Gbs1yUxEjq23ZTpZfOueoGLpo8xY57nrzZszBf6fdjq8zn4O/xgGBsqd6Zx1bydTz6Tfa95ir78DTC6Hzn+GoebP9qNTW5DjH8tWbPzx1GPx4mI44g1gpH1oUDk7IX1yVYsMFxYn1xYnN4YJ0MSQFXUN3JmZmbXsjqj2mVB4Z5doDyuxYfHjx8XfmBUU1NTL46Njbld1r1+v380TdMvN3yyoLJHH2nn8ZMnT7bcKb4XWjjWlzcQRrNsG0hYXV3dDCYMuzCkz6o/cPf/7ijgYBeM8q+JiYnjNgj0LTs2u9LEANDly5fzxdCtXvOoTp8+nX8NobCoZ2wgbqtF2Z2emwvUGbup5OzZs4WfKdkR+9q+6jImdO1yi5WaLI56HfNfv85vnyaYbjFzdnZ2zFvM/POKFzPr6pBfmUUBAR2PDz74IH9vR11MFhdU9o+TjsuxY8d0TDavFRprJEnyDXtMgFZToG2r7LC90P13bm6uaW/XCRvoyxdl3bXGjVF2uq8Nw/3/+vfeeeedfJHR3gf0pcDWGbs4uGavedvdB/IsVv0boTT0eG3nmXuG7ud6z2/cuLHvOYk7D/RviT12z9zLbcbrpSruG7ov7uWeuBWNz0NQ4D0mfQ5mZmai/o4A8s/s2tra5rkzDLcpcyuDmzIHN79tlc3rBS5WmrzWEPI8txtDCt9vAp0bly5dGvqZ7rRZz507jn9+6T3aKhhmA95/42WHd31D5bjdyHTOrXW47HuNrUYZRwzOHR23IUbHxq5tbH7G7Zzl+/bzvVtgMr8mvfnmm4UH9uPq1av5/00wsh5aFYh0VG52cTq/kBGQBFC6JEn+H2NM8GDkEG66YOTyctiNw/r35ubmXrYDiBOFH2iGQy4QqclhyIUS+/4YG0BCcxyxi3vBAgnDcgEH11vWZSPoPLJl0655AaDlpgS5QwUizbOLTdv1yd1yUc8tzu6ygWNLoZ57APP+hNF/bdtNAkNwAVz/vFQQ7NixY35gkiDYwLmnDUBaxAu5COVzizr60uLya6+9ZuymJ10nvpkkyR9u8xkBWkGLZaHGbVoka1Bwa95+xvN5he6L3/3ud6OPUXT/dPcbLfzquqPNQXZx8UMbONi2hHrIMXbDjtd2XIWFPNvDHUfda2O14fDHC/Y+7jJer/X7/T9N0/RPyryP67XWaIyFf5NXHwk5fh/clGm2CGC4DQ9u04MXuNCmzOtN3fwW+jxvajBS4+JQ78NWQW93Pvn3GjeX1p92E4bb6HutyefUPo3bTa1ntWHXVVHQear3MOT9ZzCgqY1E+mzrM97r9V6y978zdlPM/7nbxoNY9wuCkYiKgCSACilo0NONt+RJ10qWZX+mHcwa4IccXGgBVAO7qamp4zsEBupuOUYg0u3YtmVtq+7LguHM2kWhfLeeFoU0KI8VSBiGCwIpoLFVAMhOohoTlAxhh2NRWNTTZzr0pKoimwvP/oTRBQfLtlMQrIrFzBrY8twr8/i4BWbde7QRxh6PD211hPmdJvYAGkMluf9M9wJlL7ispSqCObqvuvuQ5jbKVpiZmdH46YfGmLftNZHrzvZ0LP9CC8FaoNc9Y5QMtBDc8dP4Uvfxubm5F9nMgir5Gx40xnTzac1/JiYm3Nxnt0xsYJObSzsDVUUqzw6vwLILQrpNrbr/7DC/Dkq/R/c6fblNTd6mmG95Gw+u+Bub+v3+r6k1TixJyw5yU023+UOogKQdIANAmSoLFqini0pV6GYfkiYMXqmNJvZrURbLKbdwHJLbrZ2m6cVqXyKGcMhOaLVw39P5cP78+XxgXObgfDcuAKTnpednMwRP2b6wKyP2UGwblVr+HwoGaVHPHb+Y2QUlme33+z/S4qAWnxUY14KvMlGqCkQO0iRWz+d3f/d388B9mqZuMfMfGpwxP4oT9rUWzr0q6Hql46HzxC6+5KVbO3IsgLZy45QfuHuBrrnnzp2rRVaZnoOuO97Y5Iy9LrqJR2+rLJaOGrdVa3QsDyoTUveMsgORPlc9R+eUAtw24/ZD+zy7PLZExTSm0VzMzX20Gc9uGv3QXhOPcIwwCn8+/dZbb+XnlK04c83OucIumNXHrL0vL+ne4z5XuvZXtdbhNjW5eaSek20rdcpubFp1x0Pzy5jjCIKR9XF8cXqjzQFJlQ95vfAAALRTfj0/efJk8Bfn+ubZicGhwg/Umzan5CWRQtMOY4u+XfU1bnfd/f1gELIugZ6tuIH7YFDSBhwWtvhfWsULLrrrjT7HP3QLtHU/fkMat9eODzVB1kRZ1xRNGOtazkwT2S0WM39gJ5JNuzcMY8sF5bqcezpPdM5oYm/7iv6AkuFAIy27cYruBdpoUOXi4U50/dN1R9dD26v5Q3fdCdG33uctSjZp4XjWbpw6rrGbjqUWxeuyaUrnlALcel52bHncPl8CPqicC1p4m61cJnbXN2Rij9yGSp1TXlCybRsx3HrHh5qbaY7mAn912rDrNh4oeULrG95n/MOnT5/+98L/EBjByHo51fKA5AoBSQAdoRIHa65xdGheMK9R2ZH9fv9rxk5uQtKCtDJRbYkJSlTV0+x2mXRN4hb+NHh/8uSJTrpv2V2drV048hYgD9kFiGvuGGqBtgUlWV2mXZ61rV27mqg9p3AAACAASURBVCg3JavEX8x0E8ksy37UskC5O0a1XFD26dqg52cDAWfIdAEa45BdgFvS5/fy5cv5vaDu/fVc1ZSB606U39MwK24xWBunNHar67F0m1n0PO3Goh+6DZzojNpuInPn58CGzK5U40AELrt/YCPGP7QgS/JIv9//yK13aE6pOVodNzP53PqGCxIfOHDgN83OrVr2jWBk/RCQBIB2yK/lCriEpl1lGuBkWfbVBi1ynlC5B020Qy9oeO8xWSj1dMVfEGp6Jp3OXwUcNGj3dnX+sANZktoteUrXnjqVLd2nK36mnY6prq9NtEVm3rdaFAg74x+jugcH9Px0nXNlW+3CBAFJoL60YepHWoBzmfFVlvHci4HrTu0XP0vwJbfBqAkbp/T89DwV8LFB5WsNbcmBvcmDkXUe32yRif0DsiSxH27uMpDd39Tr3nyWZR9pXUDZkLofN21O6QeJNRYiGNk9BCQBYP/ynVUVZres9Pv9n6qXoZrBh+SazdsF56bsSsx3+IYOYCj7dGZmRlmXP/abbqMWDtmswXx3oCt11oJMupzrFafsCbtw1Kbgz5ZcILIFvajG7fVi89ysa6bdqFygfCAQ1ujMXRccaNIx0vN02apamCAgCdTWvN0wNabrp+5xTQ3k6brjMuy6HIz02wA0bTHYlca048ol2k+gTvxMbJcl2YZxJqqzRXb/UgPn0wqgXtM4QusCyoZs8pzSBSVjtFZyCEbWV1cCkg8LDwJAQKEHAl7Z1XuFBwekafptfUcBydC0YGI+L316rgHnyyFXXi/0jk/XKzJN00uFB1GlPMvA3x1Y92ymvVL2hBega0XwZ5BeW4sCkUdsyeBpvZY2npt6XZrU67Onz6B26jbxnNQ9w5VKbOJ5NxiQtIsrAOrjmdLjbmzddNr4pYBkF7WhAofuGV5A8hQlW1E3LqOt6eNM1Ic7pxpYVWTFlXfXmL9pVRV2EjOgSjCy3roQkJwlIAkgkp7dsRfLrsFIVzbUBcxC0o5nr0Rk3Qf/+SQ69O6qgwcP5oHeLMsesXO4Vp7JMmj67sBhuOCPFsHaOCnXtbRFgciP/B5SbT039br02bOlj8aaeE5qMbnpk3o/IGlLHXOvAuphpYWlxzd1NTOyLa97ICB5jf58qBs3zvTaA9DrFPui63fDqoo8M45o68brGAhG1l/bA5J3CEgCiCBfcI2xcH748GH312GCkQ+MMdfHxsai9I70Fk5q3auu3+9/zUQo0ere0yRJLhceRFXm7aJJq7IMhuH6/bhJuQ3+NL0Rf06vq0WByDEFIvWaukClj/RZbGpAsg1cQNKVNGOxDqjcMwuILbi/oYV0XirYI1mW/aXrKwjUieY9Gmda1xjjYD/cmNlt8K1xQDIfR+g67WV0YkgEI5uBgCQAjCYPAMTYnTQxMeH+Okww0rjsyJMnTxYe2C8F97wSPnXdNTafpumLGlCGzkDy3lMyTeph3pU706C8bVkGw3KTcrtL+MM2TMpbkD24GYjUselKINLRZ5GAZLVcBoH5fFH5z1lUBipDIHIPRmlTgXA0l3Wb3J4+ffpfeWtbK1+7aOr1yI0zrWtt2YyJargNvq4McA0DkpuBSAVOu9yjea8IRjYHAUkAGF4+AL5161bwt8zLjLxTeHBr+rm1qakpfyIfjBfwqWvAI39eoQNTs7OzLjD8LgsjtbAZiGRx79lJuQ08EPypzqEsy/6bFvLUf7CrQXK9br1+G5B8ryF9WFpF10WvnBmbaIDyzROI3DfG3CXTfUOtOQ4cOPCbda+Gg/1p8ua/gbnP+8x9sF+uMo9tS3SlJm/oghtHvPXWW61vRRMLwchmISAJALtTtsFxlUOLscigoGK/3/+pLcE6rPzaHaNUq+vDmGXZ2cKD1dOx6Ok4hM5S9fpw1mVg2mUEIrdANloYk5OT7t8Z5ZrrjPf7/feTJDmo7Oym9x/cL71+vQ/qmWmMudnsV9NMWlT2+kdSygwozwnGKmgqV9Ehy7L/wmYi1NXA3IeNb9iXgb7rqgRWdWkbjSO+5cYRZETuHcHI5iEgCQA7yxf3XJAuJAUi1f8xTdNRUy5XFMCcm5vzF9aD0CBIJSzs4nLdSqLku3dDHwu9hzMzM5qQ/8QYs1r4AZTpiOsRyeJekZ+N9tlnn60xKR+dVxp72Gx034p203apR+RuvP6fvRpM6jvJnYv9fv9PuSYApThke+61pQcyOkbzPS+znns3aktzH2/jG1UgsC+uZKttTbRU4XrXETeOUNsFxhGj89ZB7xGMbCYFJO8sTm+0cvJKQBLAPowrQ1CDlRil+F555RX315EDYGmaXtSfCkiGVtNSrToWb+hYrK6GjRe6XpFJknyj8CDKlJe/1O/TLlgG5Vtz2WjPPffcL9meFyiHFuuO67y8dOkSb7lHZYVqMKnvLJ2TuiaonzIl94D4bIb8mII5ocekQFm0udPeu8+wkQV15m36OM44B/ulc8nLDv+LCq5/qrTzl67lR+iKX13hbTAmGNlg01oMJyAJAM9YUDk+Lf7HqN/u9Xzcy0qGsiP/xSsvGowWVlSW1pavqMt94YSOhZ5byGNx8ODBvNxtlmWPKDNYLVf+UosjXe3DN6ya9ryoPX3WvWvmKGVaNYZcop/H1pRhUfGkvvMUFDGfX0e/1vX3Aojsiu696rnnPndAE2ks45WbJ8DTIv1+/9dsoLk1BkoLH+r6Mcb+aE1Jaw4247bs7PBljSNUjazrLT9COdCOl9FZLiA5e2F9ci99dGpNAUm9Nrvo/4WuH2wAuzriFp9jlGg1Nhipcqtpmu6lXOCDNE3/amxs7JQW2EMHb/TvqUymnZxWXr6n3++fS9M0+MLP7OxsXirXGPOdPfaQQxjLbnGvTllnKqWsgLWjIH1d+jmo54U+p2NjY2dsIJ30jG3oWqvPusuCNsa8O0KZVmVl/0WSJPlCSB2O/+B56VOQuopgqSb1mlT3er2X6nLfiEVlgbzduPn7XXUmt85LZUfOzc29aKsaUMoMCE9rCWc0N1BZtaro+q/7gNvUqP8+fPhw/nddj+7evZv/XdclfdGHqho6Rjouul/oz8H79q1bt/LMRB2vqjJjNMfVfE+VgJIkGfm+rTmot7l2aK7qUMzxiu7V3jwvGP/+X1eqlNC2CjN6PVoHOH36tCstTJ9s7IvOJ10jJiYmypxLb44jqmz54cYRujfpGtn0cQTByOYjIAkAtnSCgl8aJMSYKHmTo78uPDg8jWBOaRIZOhipRU39u8qySNO06kXlWReoCj0QsgFXQ3ZZpTYD/1Ut7mkArs+kG5Srh+hO9Fw1MNdCkv6sokybrkt6v65evarP6X+1WZKtDqhrYUmLXu44ub67I3p7xEDZsnbN6ppYxXHWa1RJb/3pvnaj81MLm3q+obPJd6L7pQ2QL9lg2L0dfrwR3GKmO+92WoT0rwt6/8teXNbualu6fYFgJBCevzGl7E0fuv64wM9u94Fer/fMf7sWB7omUXkiLh0fHadhgmCDY03Nc3R8yrxv6/fYjSyKlJ4YtUrMftqFaA4Wqz+8jsHS0lLh+x3R2uoUCh7p2E5MTJyy4xw2YpajleeUrn/aBH3x4kXz9OnTPz1w4MBvFX4osKaPI9xX3RCMbIeuBCRX7GsFgEErWtiPufisSaq1n9KgWuhdm5iY6GlQEXLhcyDLYuTJaWD5zsfQCyh6z+zC8lobFs2bKnbgfyduwWhwwG0zlv/OBvf8DLr8g/vCCy8cnZmZGfMXkpQV5haRyqLPvHa1nzx58kUbYGtdmS1XSlkLXttMnNYK3ylyx/HmCBmRxt+9WmbGrgJgyuK0u3UHH17b4rx0tFhwROdnr9cb03mtxThdy3Vuxg6O6fOrhaKzZ88au8HjROGHGkLnnI7BFuecO9/8D7o2VIy/8MILMzMzMwfddUHnjQKE+oyWcW1zu5enpqambQkz7mtAOPnGFN3ry7zP61qkgM3AvWDN3gPueJ9z96SO2HvBuP277glfmZubG9N9VJuYdD/QdYmMyXA0p1DFCnf9z7JMNTL/xh4Xd5z8a7I7PrpWz2ZZ9tszMzMv6f/XMSrz3qHz2QYVR5nv7TcQNKuNSy64FPJ1agylsY+OQZIk/3mb8VKbHWnzi9M5YwPNy/QpL01rzyld/7QRZGZm5jdLqCxSp3HE+sD9yXjX9EP2a8txhMtsr9M4gmBke3QpQ5KAJACfBiDHtaAXa/FZi+u6kduAx36DfFrw7akPWuiFZg0y7OR0vsJgpAZCp1QeM3Qw0usdR1ZkdTZ7JpQ1KNfnT0EGHX+3az3Lsp8kSfJ9V6JFwdGdaEejHaBrLHGi3+//Tq/Xe1HBH52rmiiXlX1QUYmZ6LSYpInTwK77de813omdCdrv9/+8zED54GKm9a59vavDLqjZ81OTx/l+v/8H2lSi91FBSd3XYr4WLaDq8zUxMXHcfj4adT5uMWFf9xZdd3z/7ft+yL5uLbTmlQt0rdH7UkZ/OR1jGww+wb0NCOZQv99/6+c//3lpG1O2uBat2WvRzV3uff51Kh+722uTrkvzL7zwwldPnjw5puu0FhJ1XaIP8t5pTKkxgrepLT9OSZLstqD9wLs/rvj37eeff/5rp0+f/kWdA/q3Y28k0vhbi8s6N5IkGbb05WBwdeRfq3G0Wg1oDBtyzOxK8idJ8scNqRLwWeE7AWgsqUoNo9AcRl+DXNnGOpRsdK1ktBmbjVdb0rX+w60e2K+9nFMK9G1F55POqyoqiQxSlSF9ZVn2zSGu3Xulth/nnjx50oRxhH99b8Q4gmBku7Q9IPmAgCQAz7i92fY0yNaCcKybqgYG5vN+Dn9VeHB0NxVIUZ8uLd6HnBxoYKgJiV1UrmqwHyUrUu+VFg5sEKrKrM8uq2RQrh3nXums61qwT5JkxyDDNh7Ya8ZNG7zMB+gqHaQduxr8l7GIVEWJmdgUvNH7p+OUZdmjJEkuV1D2c96Vh44dKNdips5LF3i116VvDDFZ3InO6YU0TZUtO68J9tzc3Eta9NN5GfM1NXHXuo6BPkdeIPi6ff6jnnP37Lm6YjOVF1544YVz6nHk3vuYfZx0XG0wcp5gJBCMNk79ojYVxF6E1/hU14kA16JB+aaWJEny65J6BJ48efKgC3jVsexa3SlzXvcNLfTa+/Yf7nMDjrtv63hf0XhSC+S6p8bezGKzI8dK3kSkMc6ZLSpA7Ivre1ZxVZ+RuDLvIWjsbKs6ZMaY/2mM+X+H/We/+MUvHp2YmNiyvrBfRUbP1VVnqSIwOTDOpHfkFrYLLO/FwDn1o2HnJlmW/XtlfBce2KZMta5DsXvJbkXnss2OfCniNVDrDWOMI+IgGNk+LiB54sL6ZOt2nBCQBGDN2p5rL8YORBpvx2aohUK7UP8t/buhgzqaZHgLm6X3jrQ9K/PnEZJb9LfvHapR9aB8IXBmnctec4tIx7WIpKzP2Jl1JZeYicbPMOj3+/9ijLmYJMmVKnph2t2xeZA3JmVD6rptA695EDLCrtwVu+liQWXR9JouX76cZ+vFMLBrvfbZkf6Ccr/f/3Gapv8pUFm3B7Ykk87hlampqeNaQIvVI8vYEuteqdbxtveRBUqQV+hQsCD0WHSQ27Cg+8HTp0//9sCBA/97hBKT/nVp855QRuZ8m+i+oeu53dx23S7OhrrePnBVabIs+0ttZnHj2Fi0GG/nRo2raLCDxtz/dN/W2CAEjSs1/zDGfDTqhjCb/bQVV1JYf57QGEOfAa09aP6h31dmdpvGmdrEZ7N5Q8/nWkHvUahNDN459T9HKdm6w/m0bZlqrTvpfqTnXmag22VH2vWB0NfA0sYR/sbrksYRy8pwd+OImPepnWx7pqHRNJlVWdNW1om2WZ+ztgwTgG45ZBftP1QgUoMDZeTEnIjbpuvGBkNCbfJYUV8M/dtazA/JZSQqKBj0Hx6OMpNe1MAm9DHRcbbBjkYGbVqgtEG5JlAKumiCo0CDMeY37CJPrInrPVsi8VWbtZz//i16zwVlJ1B5EC3qL4pE1y69BhuIVEDoZRvYrWKBYV49PXTtiZnFpmu2XrNdzHw7SZJfj3hNemDfT52XjzXZjzlh9BZAat3H1C0ou/tymqb/IdKkXdeEr+tY6/fFvB54C4L0UgL2L79Q6j4ec36gcakW8+z94LytchCz1527J/yGxioKROl+FHoe0UZ6j7xA5PmIY8qbSZJ8+bPPPvtnHZ9QwaqtcN9opZABFdd3fdkGkf6dMeZ1XTs0v3IZvAqal0VrFNrU2uT+5A30fwV6yq5M9Yqdc/2KMeZXdT1VRRxd79577708qFbWPclVBFO7JhssDSmfC8UeR+geoYxhe2/6eknjiAV/HKHXWMU4gmBke33BZkgSkATQBrN28PP3rh+hsiHLKBXpTSRDLjg/SJLkexp4aFd1SBowaUFeQcEKyqDkvy909o4CAHqvbJlcdlJWo5TFvYFgz/lIgYbtrNrg0nUFOsoIQGh3soJoTStZ5AKR9v1RQOjLVfaAcQHdmGXRdG5qwqjAoDHm9yJk6m5H5+WX9Xs1aXRlw0PTIpE3qT9Uwusa2TYLyjFpB/HrLiAZa7LuLSq3ct4GlEjl5L8ae+OUNoZog4i9H7xaciWSO3assqZ7MAHJ3blqBl7pu5juPPfccz2dG5pDhp7nOcpA0nmeZdnRwoNA0QNbdUNBpFe1iVBBSc3rYo0rB3ktXAhGtsM9m233SzbQ/UiZt2Vs6HW8cyrkfEDjiDfKGEfoHuGNI8ps1VD5OIJgZLsRkATQVEdc6cJ+v/+Ptql3HoQ8f/58Pmguo7SIlxW5FqH8Qz4ZjrFrNtLAbDdHXP/O0JlJocvkYmTjZWRFDgR7Xq+izLBXaivPiNKEKtZCkvGyI5sWjNQEygUiI2etDuOEAroqrxurPJB/biowWEFfozsuIKnnEeuc9D7ftcyOLHlB2dFGpLf1e2NtgPLumWS4APujjI2xmD2slBGpjSHe/aCKEpluHeQ6Acmd6f7tVdooa6yle3Ye4dF4Kdax0b3DZpqFzgpCu63azZ75XEfjyjJKNep8tQH0rxQeRNO5QPfbWjsrK8itDfhWyPGz5pUHY44jtP7X9XEEwcj2cwHJVu4+UUDywvqkFsDfLjwIIITVIb+W9/B1c/Df1o4qY8xnxpgfGmOuKQCiDD8tMr/11lv5oMYLtEWlm7FKTVgxJq/aTbamAZvKUobkla3olZjhki9ehw5W6b2xQY+1EjPk8KzNjNdYg/Itgj1Vl+O9YgOifuAtOH1WbSCi15SsKC3EutKsNQlabZbSiUHXoIFzs6rrkBY31Rsx2uKm7q92oeiNwoMVq2hB2VlwWQQxAsEuiN7v93+58CCAofX7/T8yEcaijj7/LiOy4vuBM+8vJOJZ/lzO9hYu06rbyBKrXOutW7fcX8mqx15orvMbrqxwGQFJ9cy3AXQ2X7WPKwH6uts8GTsgqfGzV9UliH6/n980Yo0j9J64jMi6jSPKqDrnEIzsBgUkf7A4vdGoXfejuLA+uWBTmwGE1Rvya2kPX8cH/+2f/exnYypbqOCjSqKpFOsrr7ySTyQ1eC2TBgmuL1jE8oP5qF+L+6GVnOGymTkXOljsDWLpFVmRLMvOmmd3Hwalwa8mLObzxvl1GJQ7K36Jxlh9VeqejebTwp6ujerfmqbpsRqUTdZmi54mojGy5fV63cQsSZI/rsG5edMtbsZYNNJmA7tQdLBu2bpuMbeCBWX3e3X8/U1KQWnsY/uuAtibI/oM6bMUI0teYwB33bVZb3UZq8xrs4TGUrGuT02lyipeNn0Vx2vZlS6M3JuvlqXV0QgqK/wlXUPKCEhSlr4TVvxqLrEDkt4aYYgA9yGNI7RRONY4wt2n6ziO0KbLmL2OfQcK30GbXVuc3lDgrnULuovTG+OU7wPCGeUmpMXavWQNafE4Vkm9/VImjCaOmkAmSRJzVK5s0J/0er2XNDgJ+X4oKKjj+MILL7yRJEnsIEeUzCSdW5oY9fv9n6ZpSjCyGrMxS2C6HnDW6zXMftV5d2RsbOzMxYsXo2wc0GdVE5MXXnjhq0mS1HrjmJ6nXdi7WGWPSE9e+SPW7lXv9b5dow0RytL7Sq/Xe1nnjj4/ITeB6Dqu6659b2vxmr2S6VUtKBub5fLuxMTEcWVHlb1BCsCu8vtnrAoqWqT3+tXW6gKgvs1Zlv3DyZMnD+raVEYriybwFsGrKPsvD5IkuawNu5pXhs468Y4zwUjsxwNdQz777LNP5+bmfknnVazr6N27d91fZ1m/bTXXXuKjpaWlMR330G18nIFWB/u9N+fzylgbsL1xxNfrOo44ffr0QV0DYo8jCEZ2T+sCkjYQqQ/ydOFBANEpk6JNk14FR7ydz38YO/MnSZJv6NocepLqMlzm5uZchku0636/3/9amqbBB26uV2Sapt8uPIiy5It7sRbevUF5nYI9gxRsPzI1NdVTgN8LngZjP6tjdhJUdj/CobjNAXaTRi0WEFSST9eeGOenNqXY1/uTyJtSRmazA29OTEx8SbuOXWZxYMdt1nvV2a9+3+Cqj4PO++Na4A59zmmhRjuSAy2mAJ2TZdnvJ0kS5X7gl4lO07RW9wPrgZ2z/EDjqjJ6ddWdNo14m1iq3Dyl+8aSjkmZJfCAET147rnnVKnqh9qIp7WdGJtQvcARAfT2u2OrylzTtU8bemO0e/EC3PvOto05r3TjCNt6qI6B+FLHEZRp7aZrbSnZSiASQGi6+XqT1zKCAje1uK8bfugeYF6mYsxr/rz6eioQGXrSQonW6vX7/d8xkTINtFBUs96DO5lXuRlNpGKU2vImPbXt8e0+j3aXf+UBqtildFQm3PzbhpE6vF7fHTvp1s7ad+3ENsZX5T199Hmz1RfWa5CN66oZBL9fx+rHC3TEIVVxUInWGJ+lqstED+mmzd6OUsWhaWo0h9D44bo23oXuP+6d65S8RAgaW349VisAxwYkWb/thhV3X/I2FgblBbjH9/nvRptXujYnVp3XPDbHEQQjEcsVG8hrLAKRAELT5L2C4Ih2IX1HA//QN30NpuwArRdxB2KUslgDZfnqUA6yi9R/6UWVaI3B9Uxwvdhq7l6SJJf0OY3RS0HBSPVcVWZH4cGasKU7TY02B0QrpaOsSJcFU+PNEA/sztoTNmgY46vyLF1tWrBqcRySJPm+secIgNrILxSxshlqUCZ6WPncpayeT3Vm53M/rUmmef4cvPtZEAEX4QHniusdF2uco/kOOkUbeh/pvhSrd6633rUf0cYRA5n6nR9HGIKRnfaFyJkyURGIBBCaFhvOnj2rYMBju/O5zEyYvFRDjB1jXi+1GMFVBTh7GgCGLtVLVmQt5MGeyIt77zaoJOGysqIUlIsxmdJnKEmSg3Xc4V6z7DQnnzTeunWr8MB+uetPmqbUVKuYl0lSl+tE/jxCZ7gM9LwBMJr8c+OVawvGW5CrY3nWQfdcFl6XS7W6IEqapn9deLAa+X3j8OHDNXk6wPbcJlFXISS0gb6RaD/XOzdagCtQgDs/H2O0n2riOCJ2diTByG6rbSmwnRCIBBCaFhVdllaSJMcq2LF0z5VECL1r1su4eiPCztl8QOUFPIPQ8fCykuidVZ0yBuV1L8/6DFuyM1p2pFW7ybm3O7o2/SyzLPuKrm1eECcI1xvTZlSwGaJiXuC/LjuJ8+fxyiuvFB7YD8q0AnuXZdlvmwjjFY1H7captQZV6cjH5rFK4jVBDccs+bljzyWg7vKS9JqLx9h8yXink64oO1LrXKHbHAzYcyWwfr+f3zhCjyN0P/I2YDOOsAhGdlvjassTiAQQmhYa3nnnHaNdxMaY1ysMfuXZkaH7vGjAryCHzbgKuQllPMuyryoYELpEq3sPyEqqVpZlR+/fvx+8b4I3KG/S4p6zEmsyVeedwt4CWl0CQkeSJBmLESh3G0LSNP2rwoMonbcxpS5YVAZqxvWLDK2hVTp0jVrX/CZWSby68zaL1KYcnu5joTPqgVhcJluXNzUgqM22RKE33ptnq+TsORjp+kWGxjhiawQj0RgEIgGEpsHQQCCyykFCtF2IN27cyP/s9/vnCg/u3byCAaEDkQrw6Lgo4ENWUqUO6fhGHpRfKTzYALEmU3qvFdzv9/thU64CqOHCXr6hLkZJvrr1KES+QPBPNXsb1iIGI+n9BYwmWolW735Qm6oAQ8rvXzEWfZvAzutMnTa81fA+Buwkv4bQHxsB1fm+lD+pGOsedawuNKSox4tgZLfVvXHqoBUCkQBCUUDk4sWLdQlE5mKVgNTASl/a8RUqKz7LsrMmQolWHRcdEwV8Cg+iTFGDPerN2sBBuRNtcG4/py/WLSBRw4W9fOdrjMzIXq/nSrQ2bZyMBvMWQBpXuQaoWH4/UCWHkLQx0KviUGYf+RDy8VVXg5E2A3Gt8ACAYT1wmVERy2qy+apb7mizeU0D3Pm5GLoalFfq/d3Cg/UXdRxBMLLbGlMabXF6Qwt/xwsPAMAeqD/k0tKSC4jUIhBp3YxVAtILGobo0XdCJbHW1taCD9q8cjCNzJprkXxBPPQOQQ3KbbD5bwoPNke0yZQX/K1VQMIu7K0XHqhOlB2s7pimafrXhQeBiOihBOxZHowMvXnKu8c3sXf5PW2qUbUVANij/NoXer7jbSRk81XHJEnyf2sdoIYlq/NzMfQm18OHD7u/NnGDa9RxBMHIbmvEwNoGIk8VHgCAEWmXs0qWKuClYEKSJF+uWSk+1dO/rEGaV8oyCJVTVQlI9XkMsBNx3njlX0NRELaBDb7bKl/cC71A7pX7bGpWZC7WZMp7v/fc8yKi2mSG9Pv9XzYRzk9v0tjExefW6Wq/MwAjiZJd45VibuT9IE3TvIkW11Gg3rys7roF5/JrH71OEVAelPPmW63GOGJ7BCO7rfYfCAKRAEJ57bXX8uCZK92TJMmv1HSXUh4cjdEw8VfO0wAAIABJREFUXgFJ9QFUZmPhweEpSHJcE6fQu8ea3kuwZaKUwfQmtE0vgZmPoUJPprz3u47ByNpQyekYfT1adH62QtMn8QBKETujoamb4/L7WMT+tgAC8KoM1a1sab4JkQ0NCCgfz9fwvhSl4o63CZtxxACCkd316YX1yVp/IAhEAghBi8vvvPOOOXv2rOt7dt4OOOra/0XX5uu66Yeu0e5Ktfb7/XOFB4eXl3nVexqSJjrq1ZZl2U9YeG4vb0Lb9GAPi3wVU6Z3aC06PwEA++C1S2jqImL+vLuSgQIguHw+TjASodX1nIrYMoFxxACCkd1V64VeApEA9kuDnOXl5Twb0tY6XzPG/IYxZrkBb26eHalszpC08/Ljjz/Os4r2WApmPMuyNxQEWF0NextxmaBJknyj8CBKl2XZUa9sUDD6LKr/QAuOaD4493Y8BuG953G6xbdDtKxRe37+uPAAUBJXghgA9ikfp9iNmJ3hLXLXddMpAHQVAe5miTaOIBjZXbXd8U0gEsB+KBNSQcj33nvPzM3NucDH63ZxvynZLqtaENfCeOg+DSrVai0UHtzdiSRJDurfCLlzTLvPVaI1y7LHTe8l2BYq5xsjGGk+L7H5d4VvNk+UHY5eqSZsLw9G3rp1a9sf2I80Tf+J9x5V8DYLAaiYxt/qL89xaBavYgUVDgAAlWnRJuzgCEZ2Vy0zIwlEAtgLF8xS6VBlQvpByDRN/xeXadgkaZpeMhGyIxVIVGZjlmVfHbU3RZZl3zReuddQVI5WO66SJPkeO5mBevFK1QEA0BlPnz79R442AAAYldbc+v3+P/PGFRGM7KaHF9Yna7dTjEAkgFG4AOSlS5fMhx9+aJaWlvwSe40NQnpWtCNbgdXQwQAFJJX5pkzHwoPbm02S5KW1tbXg2VunT592f21CCV2gU7zsbHq5AgA64ZNPPjG/8Au/8L9ytAEAwKgYR2zvwLaPoM1qt5hEIBLAbrQgrubJ+vPo0aPPlC9VFmSapn+lAF6apq0py5MkyWVjzJL6KSrrMxRlNurfVKZjkiTDBmzzsq5emdcgdCxtSaW1Bjf3BgAAAOpipOonAAAAnmjjCIKR3VSrYCSBSKC7lPE32BNRwSljG1vrS4Eqr/+Hb91ez1oVgByg6+OSK0EbijIb1ZtqZmbmJdtLc7f7gnq0HVcPwdXVsLcQvTbrSuFBAAAAoDrjDW0hcMTYzAwAAFCZRo8jbt++XXhgvwhGdlNtgpEEIoF6OnfuXJ6FuB319xsMIkbmsubu2WtYV8oF6vVen5iYOKWgXcisRP1bKmtrjJkf4v3MsyJD94pUsNn190zT9GbhB1Apfc5DU++E559//tfSNOXgYq/yydxO9ygAQLdoTBmyjcCtW7fcOPlIQ+cd2khoHj16VHgAAKribZCY5SCgJvK5JeOIgkOF7wRCMLJ7atMvkkAkUF9a5LU3zp2s7fDYKAZvzHe8nUP0KPs8OzJKMFJB57GxsVM22Ljdbq3xLMveePLkSfASrQpESpqmFwsPomprU1NTvdDPQRPQmZmZFwsPNE8+gb57927QJ65JkLXd5xGf3yOC99I19vyMcd4DAKLRXKGnKiohFxFVDcRq9CJijIwGANgrNkighjS3PM44ooDMSARTiw8AgUigMZ7jUFVu9enTp387MzPzm8pGDVlu6caNG+b06dPGZkduVyb1RJIkBxWIDD15eO2119xfh+1biYbTAN9udBimPHCdRck48EpSt7X0dK0pc9c6RA9bAGiO0BtUvM1GszuMkeusR4lWAACGwzjiGSotOx1rHJEUvoO2q3xxi0AkAIzmwIED3zbPBu+C+OCDD/J/Jsuys9v9e1mWfdNEKNGqTE9bBvQ6WWC1tFmuJCRvQHuk4e9PlJ2CMbL92kilnYfI3h+ZyulYTT8/AaAr8o1NodtHaLyiDSpZln2l8GD9nTD0iwQAYBiMI4ryKlCxqisQjOyeSrMQCEQCwJ6saPFdZU1DBoeUpba2tmaSJHlpm74Ns3rs448/DlqyQk6ePOn+2sTd5l2Qb17yMvWC8II9W51vjZFl2e+bCAN0bxJEZuQO0jT9u+0f3bsWBcsBoCui9RFeXV3VGHnMBfca5IR7/gAAYEeMI4ry50swEkFcWJ+sbERKIBIA9i5N0zw70vVZDMXrAzm/xT+pXpJ5OdeQFHCxQZc1gi61lZeoDD0ob/gOQeeQC9KHRs/IoeXXjaNHjwb9R70JV9MmjK3kHV/uEwC2k18fQmc0mGeDeY26J2jDlMZaBCMBANhVlE3YprnjiPEsy74acxxBMLJb1qp6tQQiAWDf8gzC0KVaNcCwzbVP2drwjnqmHddjoQch9IpshDwYGWtxz+4Q3CoA3gTzJlLGgReMZAVxZ1GC5eoBarMjp11fUNQCwXkAO1nTImLoUue6z9tewoNj5DqbV693ApEAAAxtTesejCNyJ7RWE3McQTCyWyoZkRKIBIAgtBh7XX0W1W8xJC87csH7Z/O/v/POO0F/lwZ4yu7MsuwRwchai9I7weyejVt7/X7/awOvIxj1Qez3+z9u4vtSsmiZMK6XLtmRANAYUbLlzbPVQRYKD9aQ6/UeuqoJAAAtxjjCcuOI0OuAPoKR3VJ6MJJAJAAElWdHev0Wg3CL7y7IYkszvBGjNIN77kmSXC48iFpRUCxGsEelMG02bq+B2WfzaZq+qM+MsuhCcpOfNE3jNGdol2jBcnfNy7LsbOFBAEAd5RfuGIuI3/3ud115+XMNyGqYd2XkvR7IAABgZ4wjPpePI7TWsbGxUXgwFIKRHVJ2v0gCkQAQ3B1XQiLkQEkDDQ04FGSx2WoqzXBQmV+hAy5eVidZkTWXpunf6BnOzs4Gf6LeTrvlwoM15nYKxsiK9EqOUlttOFHK6eh6qIVcTcR0+hd+AABQN/l9M8Z4ReNgZTXY8vJXCj9QH+P9fv9P9WwuXrzICYoY8pLpoUvkA0ANRAtGNmkckWXZn5nIWZGGYGSnrJf5YglEAkA0+QAmdKlWN+BQsMUFXLSLKyQtEtnG4NddzzfUWrRBuYJ5Xq/SpgR8FlzGgbI7Q/MWUQlGDifa4vPVq1fdXxsVLAeAjlKQ5F2NMb3ey8FoPNyAMcuyNhXquZIViUjyMoahN4F5jhS+AwDleOA2unZ4HHFFCQl6rjGzIg3ByE4pbWGLQCQARHUzy7KfqO9iyIGSnw2kr7W1teCDkNdee839lazIZogW7JFLly7lf/b7/T8vPFg/4y5Iv7wcPj6lhR31i9Rnm0D90G6aSOengs26HtpSwk0Jls8G/mJREECT5GMWjY9DU1aDu/dnWfZeDcusqarJGS10xs5mAEK7deuW+xfrXr4QQLtFm1s2ZBxxqqxxBMHI7iglGEkgEgDic/0WQ/eO9LKB/EbbQShwqmCL+hCS+dUYUTMN1JtPAZ80TV9uQAbainYKanAeY6egm/QkSfL9woPYzp0syx7FyNw13vUwy7K/KDxYH8t6D4wxnxljPgz8Nc+ZB6BB8o1uoSuHONqkomwBjQX6/f5HNVpIPJJlWb6pSwudodsrACUiGAmgSnkwMsamJrP1OKIuNscR586dK2UccaDwHbRV9IVfApEAUJqVLMv+y7Fjx8YUHAk1YHDZQAo+hS5Defr06fzPNE0vFR5EnWlQflyD8hi75LRwpsD32NjYkh2r1DFQvaD3QGXPYu0U9HZgkjU8AgVvx8bGTmnxOXQfT10DlSHe6/VessHyugXM83H3kydP8pJ8CpKHCJTrvbTltDkXATSJ20B1XJtUYpRTV0UH9cubmZl52bZNqHrThhYQP1IfqvPnz0d5zUBstnShsRUZbvKGA6jIPVuqtadyrTFKnmscoTHK1NTUy3auVatxRFll3glGdsP6hfXJBzFfKYFIACjVgyRJvqdFeAUxQi7Ch17QN7YEpZ6nMniSJGGS2Sw6XtcUoIgRiFPwRAHJixcv6vx4P0mSL7ueNDVxwhjzrcePH0cpz2ps1nCv18tLtCZJUqfX3gT5+DP0ddDRMde/W8Ngef66dV5qo0fIiaPKadtrNecigKbJN1BpzBIrMKesAWXOT01NubWPBRsILdvmAuIHH3wQ5R4IlOHu3bvut1AeHkDVNMfqaT4Ua+7/5ptvDo4jqgpIbo4jlLFZ5jiCMq3dEHUxYXF6Y4FAJACULh8duYzDUDQICT0Q0aLQ2NiYspi+U9GCDfZOx+u6MqVi9Y5UuVYFOjUQrlkPBQ3Q/9LYoFSsnYKuFIwrv4yRrCqIq2BujFLCW/T3OFT4oXId6ff7P4oViPSu1ZQLBtBEK/1+/6eh+6r7dF/QQqK99p6qqGTrCT8QGWvBFCiDPksa02RZ9tu84QAqpgpkjzSO0Ib6GLYYR/yo6nGEMjbLRDCyG6Lt4l6c3lAE/1uFBwAAseVlJBQkitUzLRSvt+UVzopGitqHSRSM1EA4SZKXsiz7HzXYHT3rlyxRwDQW7by0KIu5By6IG6u/h46919/j/QqD5fM6J9VjVeVj9XkMHSCnXDCApkvT9NsmQl91n1tIdH2vsyz7B1tJIbZxO5b+gRufEIhEGyiTWeMsDUU4oACq5OaWZYwjNJfzxhFlXP9qMY4gGNkNUVbQbCDyWuEBAEBZ8pGDF8yoHQVKbf+xd20AFc0TNfvM0UDYC0h+VOGChMY3H7oBesySJS4TTdmnZA3vmTJh/kXXwVg7WLVbVOemJos2C6bMDMlxGxy89uTJk7HLly/nZQJD9Qp2/HLBNe3d2nozMzOm3+//uOvvA7BPV5TVoPtrrHuCsQuJyk53m1W0sGfLxMa6P8zb68MZ9djTIialWVE2ZR6rl1po3rlMdB1A1a7EnlsaO47Q7/DGER+WNY5QNnqV4wgFI3uF76JNPr2wPhl88ZdAJADUQilBov0g66sdkiT5holQFniQApJ2QD5mB+RlLkpsBn00QI8diDTPvp8svuzdgzRN/0pB3Zg7WF2w3O5eLSt794gNfp7S4rPOlxs3bhR+KASvXPB3Snhd2Eaapv+09SMAhqS+6pd1T9DGjdi0WUULerpGq1+lMebv7Vgi1GLivM2YuJam6YsaI2lsHasnJrCTNE3/zm6iC0pVKGy1h15JWcYAsB3NLS/Gnls6ZY8jNJ+N2Vt7GGRGtl/wnc0EIgGgPsoKEu3FQKbNTU6bRoveh8nRgPytt97K+8cYY5ZsH4XYgZ8TdqfgZtAndiBSi4k2a/g6WcP7tpklHnMHqxcs1y/5YeQg8oJfllWvLVbfUvPsxhHKaQNouiuu51MZm/W0oKeFPZWct2OXU3Yx8Y6u5SMuKI7bYEzet0rrLqoYoZKwuk5rjBQ6Mx4YQV7FI8bnyvUss/3aq27XAKDb8nGE1gQaPo74Z38coaCn5rNVjyMOFL6DtrkT8vUQiASA2rmZZdmfzc7OHtQifJ0WKNxOMld3H82WpumfaAygAWzs4Ld2SGvRTb9rZmbmZRv4uW6DPyEDd7P23+ylaZoHmjQJiP050meVrMigdE5cHxsbO6Xrjo5hLFosU1BQGTdjY2NLWZa9YTeFhMr+PmIDgr0nT57kryVWNqRDuWB0WFdKEi+EXheouQf2uvytMsYsjq7XGkeo/67uRVNTU9N6DvrSomaSJB/b47DVdVa9qv+9FgzdN3QP0HhI94CYm1GAEej8Pa5SrRsbG0HfNy3GqyrJ0tLSWL/f/+923sEGqZbxyvyyERN1tjmO0JyvjEoLZvdxxOMkSW63YRxBMLL9gk2wCEQCQC1poPSdsbGxM1pQjr1oPQo9HztookRrO6w8ffr0azMzM7+pAbIGtjFpkUMLiDqP9OfExMQpu0twzQZ+bm4zEN/NuM2EPKesM/2sdgpevXq1tHIlej02+PM2k/FglEn41ddee21M5WdCL5L5lDWrc0UT016vpwnftSzLvmk3Xqzs8byctf1KdY7n5+TFixdLmTgSGEeHdaVlzXjhO+2nnk9/pA1NZYxZHG1m0j1CX8qm0O9W//SjR48eHBsb6+10ziVJkl/77969m99jynrOwAjyTQ0KKMU4P11VknPnzv3i2NjYt/r9/ltpmn7bjvm7tKGitbwKJsx/UHf5OKLX69VlHDHWlnEEwch2e3hhfTLIDZtAJADUmnaNntHuqboEI71Mm++RadMeBw4c0C7lDxWE0QC3jExcNxj3gpJuEH7NBiZX7de9bSa2h+zXrP3KB/DKhNRgXZ+ZMgfqmkzos2qzJAj+hKONGZeUrVjGDlYFO/U7dDxV8mZmZuYlt3PVGLPuLZw9GNjBOu6VH9P5eCTLst+2pV/zfiHaFRu7TLCjz1UDygV3MZCCiOpY2j4Gvc6uvNatpGn6n1RZQdmRutaVXT1E9wmNMdzYXIvwXlbQM7TxhPKraIB8ffGVV16JVoXCbfjStWtubu5FtWywX8aO+80OSRduvLXd4wAwtDRN/1hrH3UZRxi7lrAVzSFjbsYNiWBkuwW5AROIBIDay0sUKnOszF1bO/GafRNsaRedXG9PTEyc0SKB6+9SBheU1EKeJgM6173A5NKwT0EDdX1G9G+VXa5EC5GazJjPdy7+IYH64JZVNlXZimVdC92CmXau6rqnCaItqTNd+OFtqISO/h2dk2Vev+teLtj7fB6h7zCAPVBg4m1VD9G9t6wya9vRImZZFRiASO5lWfYTbcCK2R5EC+r6zCrg6bKCNP63436zU2aQ+bzvpDb8/S5BSQD7tFqncYSxc8+mIxjZbvvOiiQQCQCNodKAp9Rnr+pgpJsw2t2rlGBpn+V+v/8HJ0+efLGK8h8KUCgIqi8FgHSu6evw4cN+6Z9NWihRqRL9f/qqcsegzezUX98luBKHDfKWvoNV55ULzrvsF7dzVRkEvlu3buX/5c7Hqnp4KHha56xIsoQABLDQ7/e/ojJrGiPXqZ0B0ERJknxfFXkUJIxdxWGrrCCNr7Ya7xs7B9XcYG5u7mCWZe8lSfJLhR8CgNEsM44Ii2Bku+1rdZBAJAA0ymq/3/+xeuNoklZlg2oFACx6RbbTgzRN/6ML+GhQXlWAT79XX03oq6TPhYI//X7/p2mazhd+AKFUvoPVZb+4nauxSpnthxbrFBy3fX0XavcEASAQlWvNsuyjs2fPjmkzSJVjZKAFNL87o3FtWSXlfTt9fv2MIQUkbTl8siMB7McDxhFhJW16MXjWhfXJPd90CUQCQPOkaZqn5Sg4VBW7GzUPuBCMbDWNMc6rL+jFixe33aGMz2mDgAuKpWk6R3nW6LSD9ce9Xq/S62GdeeWC/zPnI4CWu5Mkifo+5ZtDtuvbCGAod+wG2HzeV0feJslDHFIAATCOCIhgZHut7fWVEYgEgMZaUY8MBQOrCg7pd5vPAy7fLjyItlE0410/0IYifRY1aVHg1hjzeogy+tiV28H6+OzZs0wYBygjUouIT58+/VtjzJXCDwBA+6zYrPl8MwabqIC9cxtgvb7TteJlSBKMBBAK44hACEa2156yIglEAkCzJUly2dheYFXwSrSywN0N89odrSC0y7TCv9Ek5erVqy4QeZ1s4VKxg3ULeh9cedYDBw78x+JPoApedglZqkA8Kkl9XddB3ZtZSAT2bEVVcDT+Z3wFoEMYRwRAMLK9Rt51TyASAFohDwJWUZpQgciJiQljgy4sqHaDMtC+7DJy67pDugouEGkXafSZoE9k+djB6nFZuubzjSsK1N4r/BCGkc+zjh49GuzNsvdOQ+Y0EN08C4mju3//vvt/jjToaSOiNE3/RP/6W2+9xdsMoEsYR+yBN444RDCyvUbKjCQQCQCtoSDgdS2+e1mKpfB+H9lf3fIgSZL/TZlWCkZSsrUQiHyXQGSl2MFazNJ9m+v0vrDZBmi2ZxYSyezandeDb7zGTxPlWlG5d5V9pz83gI5hHDEiv5cvwch2Wr+wPjn0JJlAJAC0Tp4dWWapVg3ANBlVyc69lgpHo6kkpjIkH+u863LJVpVbJCOydjq/g/XSpUt+cHyh8AMA0C2b9wVljIfMdAa6QuXe6c8NoKMYR+wRwch2GnoRmEAkALSSyrytaWBU1qDI7YhN0/RS4UF0hQtI/kQlWxX86FrQR5+5GzduDAYiyaKqh87uYNXmAG+zCMFxAPjcvCvlrftCW7K7NPby+tACMd3z+3Nz3gHomNaOI2Ku4xCMbKeheo0QiASAVsuzI8so1aqByuzsrFHfQEr/dZ4Ckr+uoEev1+tU0EeTDwUivTKYBH3qp1M7WHVt1jmpzQH6TKq/K8FxAHiGMsVfd9ldTQ+o6L7mrvtASVZci5CLFy/SPw1A1xTGEU2+Dmoc8f7770ddwyEY2U67ZkYSiASA1rvpMtRiL6oo4KkJaJIk3yk8iC56YIMem0GfsvuXlkmTDWWBavKhSYgmI5TBrDWNgc+3bQfrIL9cMIFIANjRiio76FqpLHIF85p2b9BYRD27dd2fmJgoPA5EtrnZy6sQAgBd8cw4QsG8Jo8jNE8+fPhw4WdCIRjZPp9eWJ+8t9OrIhAJAN2QJMllU0LvSO/fv1J4EF31wC5MfP3555//l6WlpVaWbVVGsCYbygLV5EOTELKDG2HZ38HatnNT56VfLphAJADs6k6apv/BbVbRvUHX0SZk0GvDl56rxuO2SgnlMlGFPCCpYLg2ImosAgAdUhhHNKUSj8YRWtPw1w1ttacoCEa2z44lWglEAkCn5EGRmJNB/dt2B/a76htS+AF03ZU0TX/LlW1t4i7BrWiRTwEslaNyZVltwGfHcRhqZXMHqzs3m57B63a0euflefqWAsBItFnlV13vdWUI1HUxUc9Jz00bvuxY/O0kSX7FEIxEdfKNiK5kq3pWU7YVQMdoHPEbGkcoS7Ip4wiv1cyr+guZkRjFtiVaCUQCQOdoAXpNCxSxFiW8xXuyIrGdzV2CypLULkEFfprYr08LKqdPnzbvvffeZjakHbAvEPBppGd2sGoipglZE8uLuWxILzPmVTsZBgCMRpvrtJPvVVdyTYuJusbWYdOKWzzUc9Jze/r06d8yFkGNaE74qmsX0paNiAAwgjt2HPF7uhb6Qck6jiO0ZuiNI/K4UsyNJAQj22fLYCSBSADorPy+EKN/jAKcCshogLXd/QfwLKdp+rKyaHU+1nmX4CAXhNSCiv60wZ6v20AW537z5TtYtaDr+oVpN38TMkvcZFIZCPY6f91mxnBeRkbmEdB6q/Y+/3suU1KbVlZXV/N7RJllKDUOUUBH4xC3eGg3RP3egQMHfotrPmpmNUmSX1eWjStX6MbQ3DsBdMhNOy973c0z3ThC1WzK3AC7wzjiVRs4LW0ccaDwHTTZwwvrk4XyYAQigdbKd77qpvLo0SOOMkrnasq73pTAEJRtcMIOeJdnZmZ6Ggzfv38/D6hoYF6n65kmCBq0a2e3qMegMeZSkiRXyD5onTt2QXc+y7Jvzs3NvaTj/sEHH+STttu3b9fq9SoI+eabb7rdrPlkMk3TP27TgrRXHqhun7Wbxpglt0EBQOvdtF9HlDXwwgsvfHVubm5M94jHjx/nYxfdI/S1sbER7L1Q0EbXegU9tfnPs2Y3eBGARJ09sFk2GjMvT0xMnNJ9U19ra2v550WfnZCfGQCoqZUDBw6s2DWQeY0jTp48Oab1NK2DuDFEV8YRBCPbhUAk0C26cRzXgF69y4AyKQiuEhMKziRJssKbjxGt2sG4vhYmJiaOa5egvhT80eKEvqqgQbsG7JocuIxiZUIq6E4QshNW7DXtmaDkJ598snluVrVwts25+ZMkSb6RpmnrrsNe1lHdFtzv/Ou//uvPJiYmvqj7IAHJ+GL2rWkzMpCC03rLfJIk83Zj1Qk/MClaVNT94u7du/mf2mA1zGYWbX7S2FqLhjrf9d9+VRN7rf+O7QdPj3b4n++6j0vv2V6Sy3Zs9Uav13tJC+PKmPQ/M/qs6DOj/waAFtKcRpnjC24c8cUvfvEroccRLghZ13EEwch2eSYYuTi9cYRAJNBquoksnzx58gu73aB0Q9tu8VQ3qsESnrZ5Mdph1p0DIWmR2J4n3+t6cOb/b+8OYuO6z8SA//2egMCKhAjoIRDdrBWgUC5pydC6tVvS2T0astLsxQK2ktvAuuzWiuUTDyuqB56sgEZPNnYbCwXknrqyrWsRCW1vllaD7iWCgUgbhEYOxcorI7l4xsU3+j9mxCEpinqPfDPv9wPGkvmGFN/jzPCb//f/vi+eP7EpoO6v2RE38u1YJCX7/f6fvfLKKy9U1QbVrum6dwmOqoL26rahXcrNkeTUxIhqU57ZelIybsePH1+IRbO4VYnJTz/9tPEFs3g8njhxYlidu8ljc7Uoimtjn7TP4jn1NG2HNsYvVUVyNQvtwIEDYxsu99s3vvGN/55SejOSkZu9Nm0Xd42q2lRLtm0vnnfsv6qqiaFruf3a2ZHNVYvf/va3Xzp69OihDRUI6+J3RsQ3Kb/WPeE9183878TC5U5fB4+MfaTl4ndpVzR0rq37HbmFWPxeLopiOVcZV8+ZP9nsORPPkyrGGr1u1eJ8kLhsXrxG1TlSYy/inUl8H1Qlj+rQtZgy3idlk9Qt4EFez433m2k0jjh69OhCrAVtFUfcvn17/e8NxRGNi2Tk/ZTSi235hngm65ntnIjUtgOm24O8m+YXFgbYQixILOx0UfRpVC1a8y7XTmsiGdlBEcOcL8tyfZfg888//6cLCwsvVIF4tSgRCxLVY3p0QWInNu4YjP/fJGlyc6Ql26RVH9zL339TX7uLPsi3SJif6vf7//H48ePfr5Ijo4/L+LN6XO5G9biM15R4Y73xDWZuxfo3bX5s1rnQGhXJBw4c+IuxA+2w3O/3/2R+fv77kv+N8n62Hs/6evHA75ZtVRurUl5UPJaTLXM5Fp+rPjk2tlR/7/f7v0kpfZb/90FOKt3Lf+520XD4b0VFRZ0idmpAkwujbXuEY2AnAAAWmUlEQVRcdelcd6J6jK9u8pwZPm8OHjz40vz8/DAIqlrSP0mV7H/C4vwzi++nK0n0iEUnKM5p8n1Qo5uvY9NhVRk37Zp4/uSqv7GPT5AbG2LeuZHXxfU4ot/v/7N471Hdqck4ou7ROSNrLg8O5G9SMnI6DB9oI4nIb3X9gkAHxHP9u7lyYztz2+yUvdP1yrYpFomd2lvIjVSP3dQqajjwm3oNE4FlWabRndPPP//8ifn5+Re2WpAY3SU4KhI7T6g07Y28Abgx4a+HVeKM+sVr3WpZlqtVYjIelwcPHvzh/Pz84c0el1s9JkfFgtkmCfGh/Abz02pHa1mWbX693fiGd6dJpMcW6kdcy5WpbX0+PijL8o9z/LVZfLVd3DXq3obfo5Jvj9u4OMP+uFN12mBHquf1tpXrOc5pRIOLiHU+H8+PfWR6delcd2PsOTOSWBiNEzbGDMfybWg02d+gppJdbaxwvTT2kfo08bt9Et8H3WvwOrcxfmrk+dOmqr+aVMnEx+KIjXFDQ3HE8P1Lg5ua7hzIJ7cXL9g0745EJPvIQsH+uac6jc30+/2fRIAS7QTrFG3pMkkPr31NW985nYPtIxuqDYaLoxt3CY6KuabRCTJ/6N6Gm58fuzFMTG6xo3+4MNbv9/9FJM93+LWrN+bV5qB4XN5refJxoy4utD7IjwOAthjGRVpXMkUebIjXt03yZ9X7hSbiqC5tzLDG1LyureXZ2NR+ww0fdY95GnUgv5C/OXaESfNFXvyQiGS/TNtOFJh0Z8uyfCESkXW2aI0WgtFCJCp2yrKUjGSvPdisUma7XYET3rKFyTC2o3+7xyQANGS4iFj3eIaRGWRd74jCZNiYwARgZxqJI0Zmot4pVnozNxosM2fvPJCIZB/dX+nN7GSHGrBHvvrqq5+kBlq0VrMMyrL867GDAADAvhgMBj+sZhnXaWQGn2QkAEypfr//p3Fmt27dqvUER9q0DmdGptxWRyJrspn7yX4yAwHaZfHAgQP/OuaV1R1EjLRo1ZoOAIA2GauGGgwGLxVFcehJ32O/3//7siz/39iBR1XvbZ5hW5kriuJw3bF/OHHiRPVXyUgAptXcdutcT4gntiv0q9rxtv136Fx0V7t5c7tT2Z2RDgvDmZFppTcTswYXJSSBXXhdVSS0ztnUQFVkJCKPHj0af70yAQsyAAB0yGYLhZ999lmKasEn2Wr2dEppYTAY/OeiKP79DufV7ZfhBuEbN8bysc/s+PHjwxncRVFIRgIwrWL82MJW5/bb3/5201mK0T3g+PHjW35ejiN+XBTFX+bNTW01XEdsMI54WBTFemXkaEIyLsrs2GcBPO5+vFDlVs9Ae8T84DMRJDWRjMzMigQAoFWKooiywIWTJ0/WMu8oYt8LFy7EQmP0F/vb2Ijb0jj4SCx0/u53v6s9/o958blNa/0llwDQIrlLwvdHOgI8s9OnT6e33norfpH+PFdHtnEdPeKI/xBxRN3JyJE44nb8pxg9GAnJld5MlKT+NKX0xdhnAzx6bbi00ps5JhEJrdRIVWTsZJqfnx8GZy0NngAA6LZhjBpxax0ino6E5Icffjj8aoPB4L/kNm5tcz4qQq9evVr7tzVyLcX/AEy1siyHG29eeuml2k4zfjdfunRp+PfBYPBJLiBom4gjDsf3+vDhw1q/tZFrOYwjirF7PEpKruYL8+7YQaDLojVjJCGXPQqgnQaDwVvRiqpaNKlL7OZKj4Kzy370AAC0ULRYq3URMRblLl++nH72s59F5eWhwWDwcVQQjN1x/xzr9/tvNxH/p8ev5Z2xgwAwXW6kmuOIlDc3vf/++8M4ot/vf9K2OCKldHEP4oitk5HpUULywUpvJnrOf/cJQziB6RevAT9Y6c1EW1Zz4qC9zsZupmirUOdupsOHD6fFxcVhj3ctWgEAaKlGFhFTrmy4fv16LCR+p9/v/6+2LCTGomZZlt+MRc66qxlCvAfIVEYCMO2Gv+tGfvfVJn5P3759Ozb4x4zq1bZcx5wcTU3FETtORlZWejP3Vnoz8RN4Oc+IA7ojnvM/iteAaOPs5w7t1u/3L6QcRNQp2lNFj/eiKP6rhwAAAC0VG2d70Vo0ZhTVbXl5eZiQjIXEliQkP4jvJRY3m2jRGtfw6NGjKW9OtikZgGl3L0YTRRwRm/LrFnOo7969G1/1TN7oP9VxRFzHHEd8VH3sicnISsyGixlx5klCJ4zOhbzmRw4TYbEKItbW1mr9fl977bXqr63ZvQUAAJsYdvFooqohbUhIDgaDf9jHGZJxnmc+//zz4eJmE1555ZXqq1oTAKATyrL8n6mhOCIqD8+dO7eekNznjU3DOCK+l6biiChsyNbjiB0nIysj8ySvjB0EpoG5kDCZzqbcQqpO0VJhZCfTPY8NAABabLjgNZJIq10kJGOuUoxHSCn9XXxoDy/HkXyOZ2K+UywgNtFWLW2xiAgAU67RTU1VQrJq2ToYDH6dUjo1dsfmHMktU4eJyPhemoojRq7h7pOR6Q/zJGPR8wfmScLUMBcSJtexamd0zIus0+nTp6uvZlYkAABtt95irYlWrZXLly+nS5cupUgIppQu9vv9/xvrbmN3rNfZXI35aiwgRrIwV1fUzoZEADrqTiQIFxYWGosjIvn3xhtvDEcsFUVxKKX0tzlBeGzszvWq4oiFphORG+KI9TzDrpKRlZghl+dJ/sg8SZhY5kLC5BtWRX7yySe1nkgEXhGA5Z1adkQDANB6ZVn+TWq4OjLl2Ds27lXVDSmlX+TFxLqTkme/+uqr/51S+nlUY0ZVZpMLiOnxqkgbEgHolKIofpb2II6IZORI29aFlNKv8u/dxuOIiF+ajCO2Kmx47uuvvx67824tza5Fa4rzKaVvNXYmQF1iLuSqdqww+QaDwT9FQPHyyy/XGkxE26c8L/Kn5kUCADAhogXZP0bVYlNt1jaKCoCInaMiMz2Kz39dFMX/yItwu9n0G9/4qX6//2dlWb4QH4ik53vvvZdu3bo1duc6xYbEjz/+uDqHP2r0HwOA9tnzOCI2AUW1ZK4mTNHlIW+uulFnHPHOO+801lWhsl0cUWsyMj1KSB7JC5Znxg4CbRFzIc9rxwpTIaoif379+vXh/Jo6RcvXgwcPflkUxXdG2yoAAEDLRRLwTLRSrbt7yHYiKRnVANFdpDIYDB4WRXE7Lyje26Lt6Vxe/FzM1RHrIs6Pc2g6CVmJ9xS5GuR1lZEAdNS+xBGRlIzb/Pz8+scGg0Gsy93KccSDLZKTizmOmGtzHFF7MrKyNLs2l5OSC2MHgf0ScyGXV3oz9Q6VA/ZNzKeJtlCx6FHn7qYIfi5evJjy5oWzY3cAAID2irlLv4qZ6iMtR/fM4cOHh9UUcYsE5aFDh3b8T8f3HAuGcYvNgU22Uduoqmbo9/u/Kcvyn4/dAQC6YV/jiPh9XMUQkxRHRIeIq1evbhlHNJaMrCzNrp3KSckXxw4Ce+V+TkLa1QjTJTb+/F20Woh2DnWK4CG3mfrBFruuAACgzYZVDTGTKW77KRYVo/Xa9773vU0XFGPhcG1tbbi5cC8XDTeK65SrMcaqGQCgY1oXR1R/bjQpcUTjyciKeZKwL77ImwFWtWSFqdRI24hqJ1Oupt6bBvkAAFCvYzFz6fe///03o4tILNKxtajAiFlSeU7Vv9zyjgDQDeKIp1B1WPvqq6/+z4EDB/7NZp9ZjH2kISu9meVc3nqlFVcHpl881+biuScRCVMpesGfiYHadfevjyArG9vFBAAAE+JeWZbvRCXihQsX/My2EW1lq/nzZVn++db3BIDOEEfsUFRsVtfowIEDf7HVZ+1ZMjI9Skg+WOnNnM0t326O3QGoQzy3Xo7n2kpvZrPB+MB0iG4DVQVjbWIhIgZNR393yUgAACbcclQ1LCwsjG64Y4PLly9X7WMvGdEAAOvEETsQG5p2EkfsaTKystKbubPSm4m2bz/Ks+yAZxfPpdfjubXSm7nhesJ06/f7P4kTvH79eq3n+dprrw3/LMvyr8cOAgDAhIlKv8Fg8OVbb71VzURnRFQyxHynWGyN9UTXBgD+oOoY8MYbb4gjNvE0ccS+JCMrK72Zayu9mWM5Y/rF2B2AnfgiP4eiJasqJuiGU2VZvhCJyLp71keP98zrCQAA0+BOURR/Gefx/vvvD1uJ8UjE/rEZcTAYPCzL8o9dFgAYE5V+r0flX8QR0VGMR542jtjXZGTFPEnYNXMhoZuGLVrrnhW5uLiYjh49mvJrizbPAABMi9hodyUWEt955x0LiXkB8eLFi7GA+GVRFP82pWRNAQA2tx5HvPfee+KIXcYRz3399ddjH9xPS7Nri7mcc6FV3xi0S8yFXNaOFTopNu/86vPPPx+tYqxF7PCK1goxdzal5PUFAIBpE4uJZ+7evZvefvvt2ruMTIpqATF7XVcUANiR9Tji3Llz6eHDh528ahviiBjFeG3sTptoXTKysjS7djYnJV8cOwjdFS1Zz2vHCp02DHwuXbpUa2VktKv6+OOPY0fTr4ui+KOxOwAAwHQYxtNffvnlcP5TLCh2iUQkADwTccQu44hWtGndTE62zJknCeviuXBMIhI67chgMPhxBDw3btRbuBg93kNRFH81dhAAAKbH2arV2tWrV9Pp06c786NdXl5eb6mWKxmsLwDA01mPI6LDWN1dy9rsWeOI1lZGjlqaXTuWqyTPjB2E6fdRroY0vw2IgOfnH374Ybp8+XJtFyN63UeV5cGDB6PP+3fMiwEAoAOGsXWc5s2bN4fx9bS2bY0uKDEr8/jx4+nrr7/+p+eeey5GI90ZuyMAsFOPxRGRqJvWtq11xRETkYysmCdJx/RyEtLcNmBoMBj8QyQLT548WetCyUiLhSs5mAIAgC6Y6/f7/60sy+9H95GolIyNf9O0mBiVn9FGLio4Yr00pXTK5kMAqMXcYDD4ONbqIo6ISsmIJaZJnXHERCUjK+ZJMuXMhQQ2ExtyfnH79u1hEFCnCJRid1NK6bspJVXYAAB0zXK/33+7LMtvfv7558OEZHQOmeSk5EsvvZQuXLhQxfnhpyml1bE7AgDP4kis5aeUhrv8I46IpGSMV5qyOOJSzsnt2kQmI9OjhGT1Q47bt8buAJMpntSrK70ZuxSBjYYDst9+++1a50VGcPHee++lvLtpcewOAADQDY+NCIoKh0hIRmJyktq3Rnx/7ty5ND8/X33oZu5+YtMhADRHHPEEE5uMrOR5krGz69WxgzA5zIUEthMbcP4xdlfVPRh7pCry5ZSSttAAAHTdcDFxMBj8uCiKYU+yu3fvpuvXr6dPP/10+Pc2WlxcHLZSqxYPB4PBr4ui+E8ppWtd/4ECwB56LCmZxBHrJj4ZWcnzJCMpOTt2ENrrfuwsMBcSeILhUOzYTXX58uXt7/kUYrj2K6+8klRFAgDAmCN5LtL50bWm2CAYnUriduvWrbFP2kuxqTA2K8YC4tGjR4f/cr/f//uyLC/nzioAwP6YyDgiJyH/qok4YmqSkZU8T3JV61ZaLuZCLq/0ZsxrAHbiTgQuJ0+eHLZ2iJYJOxVBzsZ2EKO7nQaDwcOiKP6Vtk0AALClY3nz3qnRzlzRgi0WEuO2F9UOMzMzw/cC1a1aOMyu5IVDm50BoF22jSMiMfnLX/5y6uOIqUtGpsfnSV4cOwj7792ciDQXEtiJCFh+1cSVyrum/zwnOwEAgCerKh0WB4PBvyuK4nD1GbGoGAuJkZiMv8fC4sOHD596cTEWC2OBsPrzxIkTw+qFQ4cObbzrR7l9WtysMQBA+z1VHBGetoJyszgi/tyQfEx7HUdMZTKyYp4kLXMzt2RVfQQ8jbP5NurOUwQJcznQGXVvJNgAAAB2by5XOyz2+/0TZVm+sN1XikXGWGDcaItk47rBYPBlURS3csXCDRWQADAV1uOIwWDww9Hk5GaiA1rcNpqEOGKqk5EV8yTZZ+ZCAgAAQDccyQuL1abAY/lWWdjsKuTxCbdHPnRj5M97xioAQCfsKo7o9/u/Kcvys5EPtS6O6EQysrI0uxatW5fNk2SPmAsJAAAAAAB0WqeSkekP8yQjIfnm2EGoTwx8PW8uJAAAAAAA0GWdS0ZW8jzJD7Yqa4VdupmTkHdcQAAAAAAAoOs6m4ys5HmSkZR8cewg7Nz9nIS85poBAAAAAAA80vlkZMU8SXYp5kKurvRmll1AAAAAAACAx0lGjsjzJFdTSmfGDsI4cyEBAAAAAAC2IRm5iaXZtbmclDRPks3EXMjlld7MjU2OAQAAAAAAkElGbmNpdu1UTkqaJ0nKcyEjCfmBqwEAAAAAAPBkkpE7sDS7FvMAz5sn2Vlf5KT0qpasAAAAAAAAOycZuUPmSXbWR3ku5L2uXwgAAAAAAICnJRn5lJZm1xajVad5klOvl5OQ5kICAAAAAADskmTkLi3Nrp3NSUnzJKfLFzkJaS4kAAAAAADAM5KMfAa5det58ySnxiVzIQEAAAAAAOojGVmDpdm1Y3me5KsTfzLddDOldNZcSAAAAAAAgHpJRtYoz5OMpOTs1JzUdLufk5DmQgIAAAAAADRAMrIBeZ7kqtatrRVzIZdXejOrXb8QAAAAAAAATSpc3fqt9GY+SCkdyzMIaZd342cjEQkAAAAAANA8lZENy/MkIzm5MNUn2n4xF/L8Sm/mTtcvBAAAAAAAwF6RjNwjeZ5kJCVf7MQJt8f9nIS81vULAQAAAAAAsNckI/fY0uza+ZhXaJ5k42Iu5OpKb2Z5ys8TAAAAAACgtSQj98HS7NqRnJB8s3Mnvzeu5GrIB104WQAAAAAAgLaSjNxHS7Nrc1G9Z55kbWIu5PJKb+bGlJwPAAAAAADARJOMbIGl2bVTOSlpnuTu3M9JyA8m8ZsHAAAAAACYVpKRLbI0uxatW8+bJ/lULuXZkFqyAgAAAAAAtIxkZMvkeZJRJXmm69fiCT7KcyHvbX83AAAAAAAA9otkZEstza4tRutR8yTH9HIS0lxIAAAAAACAlpOMbLml2bWzOSnZ9XmSX+S5kKtjRwAAAAAAAGglycgJkFu3xizJix29BO/mRKS5kAAAAAAAABNEMnKCLM2uHcvzJF/tyCnfTCmdNRcSAAAAAABgMklGTqA8TzKSkrNTeor381zIa2NHAAAAAAAAmBiSkRNsaXbtfJ4n+a0pOaWYC7m60ptZHjsCAAAAAADAxJGMnHB5nmQk796c8FO5kqshzYUEAAAAAACYEpKRUyLPk/wgpbQwYWcUcyGXV3ozN8aOAAAAAAAAMNEkI6dMnicZSckXW35m93MS8oOxIwAAAAAAAEwFycgptTS7Fq1bz7d0nuSlPBtSS1YAAAAAAIApJhk5xfI8ydWU0pmWnOVHeS7kvbEjAAAAAAAATB3JyA5Yml2by0nJ/Zon2ctJSHMhAQAAAAAAOkQyskOWZtfOxpzGPZwn+UWeC7k6dgQAAAAAAICpJxnZMbl16/k9mCf5bk5EmgsJAAAAAADQUZKRHbU0u3Yst259teYrcDOldNZcSAAAAAAAACQjO25pdm0xJyVnn/FK3M9zIa+NHQEAAAAAAKCTJCMZyvMkV3fRujXmQq6u9GaWx44AAAAAAADQaUXXLwCPrPRmPkgpHcuzHnfqSkppTiISAAAAAACAzaiMZEyeJxkJxlNbVEpeydWQd8aOAAAAAAAAQCYZybaWZtfmUkpHqvus9GZubHd/AAAAAAAAGEop/X/v3BJVuTshzwAAAABJRU5ErkJggg==){height=\"60px\" width=\"240px\"}](https://pytorchlightning.ai)"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Barlow Twins Tutorial\n", " :card_description: This notebook describes the self-supervised learning method Barlow Twins. Barlow Twins differs from other recently proposed algorithms as it doesn't fall under the category of...\n", " :tags: Image,Self-Supervised,GPU/TPU,Lightning-Examples"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab_type,id,colab,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10"}, "papermill": {"default_parameters": {}, "duration": 15.139577, "end_time": "2022-05-03T01:21:09.919938", "environment_variables": {}, "exception": null, "input_path": "lightning_examples/barlow-twins/barlow_twins.ipynb", "output_path": ".notebooks/lightning_examples/barlow-twins.ipynb", "parameters": {}, "start_time": "2022-05-03T01:20:54.780361", "version": "2.3.4"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"0d637445d6d8411297c8497dd704eb7f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_dd30a8ab063447708f22b4847426cf80", "IPY_MODEL_d49d112ca0ac4352af35bb4ad446cca1", "IPY_MODEL_c8e5e7fd020941f889a28247ba77f3f6"], "layout": "IPY_MODEL_fb2becc997ff43098ee238f48c6216d5"}}, "579c0a2f47ea40eca302d66d457e869d": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "57e0ec6141f84b6f898aa10e4062a6c7": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "587932cf61a747b3900695e6bb3b35d9": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "73d30dde3bc04b06b0e18d4a24601b20": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "78828cc33b144d349ad8cacbd45fc6c8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "7fc4fbf0e8924419975a975195a47674": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "c8e5e7fd020941f889a28247ba77f3f6": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_579c0a2f47ea40eca302d66d457e869d", "placeholder": "\u200b", "style": "IPY_MODEL_73d30dde3bc04b06b0e18d4a24601b20", "value": " 170499072/? [00:01<00:00, 114866769.73it/s]"}}, "d49d112ca0ac4352af35bb4ad446cca1": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_587932cf61a747b3900695e6bb3b35d9", "max": 170498071.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_7fc4fbf0e8924419975a975195a47674", "value": 170498071.0}}, "dd30a8ab063447708f22b4847426cf80": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_57e0ec6141f84b6f898aa10e4062a6c7", "placeholder": "\u200b", "style": "IPY_MODEL_78828cc33b144d349ad8cacbd45fc6c8", "value": ""}}, "fb2becc997ff43098ee238f48c6216d5": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}