{"cells": [{"cell_type": "markdown", "id": "efb6c4fd", "metadata": {"papermill": {"duration": 0.031879, "end_time": "2021-09-16T12:37:55.184431", "exception": false, "start_time": "2021-09-16T12:37:55.152552", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 4: Inception, ResNet and DenseNet\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2021-09-16T14:32:23.232366\n", "\n", "In this tutorial, we will implement and discuss variants of modern CNN architectures.\n", "There have been many different architectures been proposed over the past few years.\n", "Some of the most impactful ones, and still relevant today, are the following: [GoogleNet](https://arxiv.org/abs/1409.4842)/Inception architecture (winner of ILSVRC 2014), [ResNet](https://arxiv.org/abs/1512.03385) (winner of ILSVRC 2015), and [DenseNet](https://arxiv.org/abs/1608.06993) (best paper award CVPR 2017).\n", "All of them were state-of-the-art models when being proposed, and the core ideas of these networks are the foundations for most current state-of-the-art architectures.\n", "Thus, it is important to understand these architectures in detail and learn how to implement them.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [![Open In Colab](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHUAAAAUCAYAAACzrHJDAAAIuUlEQVRoQ+1ZaVRURxb+qhdolmbTUVSURpZgmLhHbQVFZIlGQBEXcMvJhKiTEzfigjQg7oNEJ9GMGidnjnNMBs2czIzajksEFRE1xklCTKJiQLRFsUGkoUWw+82pamn79etGYoKek1B/4NW99/tu3e/dquJBAGD27NkHALxKf39WY39gyrOi+i3xqGtUoePJrFmznrmgtModorbTu8YRNZk5cybXTvCtwh7o6NR2KzuZMWNGh6jtVt7nA0ymT5/eJlF9POrh7PAQl6s8bGYa3PUum//htmebVtLRqW0q01M5keTk5FZFzU0oRle3+zxwg5Hgtb+PZiL/ZVohxCI+hL5JgjmfjPxZ26+33BG3dA+ealHPM4gQAo5rU59gsI8bRvl54t3Ca62mvHyUAhtOlLd5WSQpKcluBjumnoCLs1EARkVd9E8l3p9y2i7RbQ1B6pFwu/YDgW8KbHJHMTQrwnjz2oZm9M4pavOCfo5jWrgCaaMVcMs6/pNhDr0+AMN93XlxV7R6DNpyzi7W/OE+yIrsjU6rTrbKV5cd/pNyItOmTbMp6sbBB+EqaYJY4cWE3VUciNt1TpgfcRFv71Fi54xT5kSoyLvOBEJMOMxWXkFlBeBSX4u6Zkcs+3KszYRtiapbNRqF31UgetVuc8z9vBXIv1qD+F1f83B6uDlCUyfsZGepGPpmg01OB7EITQbhS9ribKy+DmP1DUiClLz4bnIHVOqa7BY+Z1wg5g3zgUvyehiNpnJKxSLc/ts76LKm0BzX3c0RNy1yXjDcB5lWoro4iNHQxM+f1kWeWQARAWQS++trISJTp061Kep25X/MycwtjuctSC5rxo7ppi7VNUox5+PhPHtrsS2O1qJ6yx1QujQUzm9sh6hbkBlvvGcN8hYnwjUjH6kjfZEd5c/jitz5Jc5U3ENnFynKl4eB7nyEgP2UZ+Yz3/rVEbyYr27qELrtC4FIC0J7sc7xWnmccdHfRRTs0VB+cA4lt+oFcRR/wUeH8FG5w2Mbx8FQ8TXEvv1xYf4wBP3O2WyL3/UVjpXWgIqaFeUPr+wTmDvUB7njH6/bOv+HRg4SqioAg5GDe1aB3ZeMTJkyRSBqkLsWqSEm0fZVBEN94zEZnYvrdx1JL5cxe+a+AbhSJecRRHW/ikTFRTa38dtQlNZ5CRKwFvUtZU/kvBoEF9Uxni/XqIM+dwKbTw3rhcxIf7gmr2M+H6SMwx8iBzJbw5oxeG3Lv5FX9B3AGaHPS8e8z77H7v9VMpvPG5ug1enh7eGK8h0LBTwUb+GInqzInlRUK65DmTPQu4c3+uQKjwKK77zwUxBX4Tq7yR1RuiwUsqlrABCM6esHdXoy47fk4+prYKy8ZF574x4V5BnHQBuf4g9Z9ld8U36L2aktZNNplNfw7zotwWTy5MkCUft4aLEopJj5/OPHl1BQqeAVOnHgNSQOqmBzq9V9cfEm/yx5ubMGKS9cYPZ3vx2OS/c6PVHUuUO7Y1Pci3BO/1zgq18byebfGemLtNF+6JRtOvMk926ibussZqM+1mNz4TWkH7rCbM5phwGRGDAaoF8fY5OHFnlldAA8sgoEXKnDukA1NgSeNjqkJT9brbN4pC9WRweYXyLugR73c+MYvyWfu0yC6+mjzN1Isfw3FKJS98CU/zI1IHFkFPR52cHL2FJk0sB6kMTERIGo9GzcPkLNfA0cwdwi/hfEYO86ZMd9w+y1egfM2T2Eh/vesMNwljSzuZRT420SW3eqy8N6aHMmwmnFUZ7/PGVPbIoNZvNU1BURdHs0bT2+HjL8sDSM2e6vi4Lj5NW8WOLVA6RTT2azxLV+bglaFNqLieqemS/gWkw7NyoAHo+2dEsiivengjKsPFoqWOvbSh/kxPaxyW/JRzH2Fl3EzD9/xjAefJqB3usKUFn/0Gb+S/d/jy3FN2yLOmnSJJtn6oehByEiHPSeXnDxFGPRnoFoaBJjcdQlbDwcjL1zTNuQpoxD7R0OG0uUTMi0fkVwdzBdYIwcwZunxrVJVLplNm54BZp7jfDfYLoNyqQi1K6KxIdHzmN+QQ2WjFIwUT2zTGdlRXo4NFXVUO4sgX5dFC7f0aP/ZlNeUjFBuL8Xjl6uRuP6aMjSjpjzsH62FDU7JhBuGccEXIvDfJFFBc/gHw80dklfCVYnRaDfpiJcutPA4F7qJsfJeUPQI+1fqMlNhFx1FM0GDqkjFVg7NojlQ0Vt4aM5ReSqcbpaCg8nCW5lRsBvbT4T1TLfFptsfh7gItzuKTdJSEiwKSrt1vcmnEXXrsLbYnWDA1bu+z2WKy9Arq+1KRqdfKsoBo0GcdtEpS/B1bO4v0cFiUhkjskvKcMrWwtAPHuwQq8Z+4LZ1vTQANfXt4J0DwZX9gWa9qh4XDM/voC9JXfwYEMMHJcfNtusn82ihvliVUwg5KrPGVf6GH94ZJpEZBen6EC4qYTHA1dXhW0JIex8txzv//c8lhzXIi/BFxOH9jGbQhZsRalTIBZZ8KkGyZAxeRQvXkFF1TWz/Hm46jNYUnjPbt3JxIkT7f6dSj8qfJJyVvBxgaIlblOyjtysNHWN9fjjqWi7glJfW3/S0Hlj2XnA8PhKT9w6g3Qx3XiXhvuxQsuT1proxBKI/AaZqY1Xz5muvY8G8XkRRCaHsfQsRAFDH/tZPbcYuHotOG0FRIqB4HR3wNVoIPLtz8ycTguu+jpEigE218vd1YCr5m+HpHMvEI9u4LTXwNWaLjl0iPwGAmIpeHx1VeCqTJdPs1/vweweQPO3HC24NhOhnTphwoQnfv6QSY2ICbkNmdSA4h87oaLaiYfn5diIEd4att2erOwJXbPUHp953p6orQVSUVWRAXBT8c/dJ5L9xhzaJGp71GR/wFP8P5V2z10NSC9T93QM2xUg8fHxT+zU9ijeU4naHon8CjFJXFzc8/kn+dN06q9QgF98SYSo2Xen2NjYZy5sR6f+4nLSK5Iam2PH/x87a1YN/t5sBgAAAABJRU5ErkJggg==){height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/04-inception-resnet-densenet.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/latest/)\n", "| Join us [on Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)"]}, {"cell_type": "markdown", "id": "273b3c0d", "metadata": {"papermill": {"duration": 0.031054, "end_time": "2021-09-16T12:37:55.245887", "exception": false, "start_time": "2021-09-16T12:37:55.214833", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "df9f9791", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2021-09-16T12:37:55.309229Z", "iopub.status.busy": "2021-09-16T12:37:55.308754Z", "iopub.status.idle": "2021-09-16T12:37:55.311310Z", "shell.execute_reply": "2021-09-16T12:37:55.310848Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 0.035577, "end_time": "2021-09-16T12:37:55.311419", "exception": false, "start_time": "2021-09-16T12:37:55.275842", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# ! pip install --quiet \"pytorch-lightning>=1.3\" \"tabulate\" \"torch>=1.6, <1.9\" \"torchmetrics>=0.3\" \"torchvision\" \"matplotlib\" \"seaborn\""]}, {"cell_type": "markdown", "id": "919ff7b6", "metadata": {"papermill": {"duration": 0.03039, "end_time": "2021-09-16T12:37:55.372930", "exception": false, "start_time": "2021-09-16T12:37:55.342540", "status": "completed"}, "tags": []}, "source": ["<div class=\"center-wrapper\"><div class=\"video-wrapper\"><iframe src=\"https://www.youtube.com/embed/vjSSyGxlczs\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></div></div>\n", "Let's start with importing our standard libraries here."]}, {"cell_type": "code", "execution_count": 2, "id": "f68d5759", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:37:55.441162Z", "iopub.status.busy": "2021-09-16T12:37:55.440683Z", "iopub.status.idle": "2021-09-16T12:37:57.080634Z", "shell.execute_reply": "2021-09-16T12:37:57.080146Z"}, "papermill": {"duration": 1.677515, "end_time": "2021-09-16T12:37:57.080748", "exception": false, "start_time": "2021-09-16T12:37:55.403233", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/tmp/ipykernel_1007/1951796227.py:25: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n", "  set_matplotlib_formats(\"svg\", \"pdf\")  # For export\n"]}], "source": ["import os\n", "import urllib.request\n", "from types import SimpleNamespace\n", "from urllib.error import HTTPError\n", "\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pytorch_lightning as pl\n", "import seaborn as sns\n", "import tabulate\n", "import torch\n", "import torch.nn as nn\n", "import torch.optim as optim\n", "import torch.utils.data as data\n", "import torchvision\n", "\n", "# %matplotlib inline\n", "from IPython.display import HTML, display, set_matplotlib_formats\n", "from PIL import Image\n", "from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint\n", "from torchvision import transforms\n", "from torchvision.datasets import CIFAR10\n", "\n", "set_matplotlib_formats(\"svg\", \"pdf\")  # For export\n", "matplotlib.rcParams[\"lines.linewidth\"] = 2.0\n", "sns.reset_orig()\n", "\n", "# PyTorch\n", "# Torchvision"]}, {"cell_type": "markdown", "id": "f4e16e1a", "metadata": {"papermill": {"duration": 0.030742, "end_time": "2021-09-16T12:37:57.143330", "exception": false, "start_time": "2021-09-16T12:37:57.112588", "status": "completed"}, "tags": []}, "source": ["We will use the same `set_seed` function as in the previous tutorials, as well as the path variables `DATASET_PATH` and `CHECKPOINT_PATH`.\n", "Adjust the paths if necessary."]}, {"cell_type": "code", "execution_count": 3, "id": "66494ff2", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:37:57.208554Z", "iopub.status.busy": "2021-09-16T12:37:57.206287Z", "iopub.status.idle": "2021-09-16T12:37:57.278994Z", "shell.execute_reply": "2021-09-16T12:37:57.278491Z"}, "papermill": {"duration": 0.105281, "end_time": "2021-09-16T12:37:57.279113", "exception": false, "start_time": "2021-09-16T12:37:57.173832", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}], "source": ["# Path to the folder where the datasets are/should be downloaded (e.g. CIFAR10)\n", "DATASET_PATH = os.environ.get(\"PATH_DATASETS\", \"data/\")\n", "# Path to the folder where the pretrained models are saved\n", "CHECKPOINT_PATH = os.environ.get(\"PATH_CHECKPOINT\", \"saved_models/ConvNets\")\n", "\n", "\n", "# Function for setting the seed\n", "pl.seed_everything(42)\n", "\n", "# Ensure that all operations are deterministic on GPU (if used) for reproducibility\n", "torch.backends.cudnn.determinstic = True\n", "torch.backends.cudnn.benchmark = False\n", "\n", "device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device(\"cpu\")"]}, {"cell_type": "markdown", "id": "66d340c9", "metadata": {"papermill": {"duration": 0.030784, "end_time": "2021-09-16T12:37:57.343999", "exception": false, "start_time": "2021-09-16T12:37:57.313215", "status": "completed"}, "tags": []}, "source": ["We also have pretrained models and Tensorboards (more on this later) for this tutorial, and download them below."]}, {"cell_type": "code", "execution_count": 4, "id": "d1d2e773", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:37:57.410780Z", "iopub.status.busy": "2021-09-16T12:37:57.410299Z", "iopub.status.idle": "2021-09-16T12:37:58.351216Z", "shell.execute_reply": "2021-09-16T12:37:58.350722Z"}, "papermill": {"duration": 0.976609, "end_time": "2021-09-16T12:37:58.351334", "exception": false, "start_time": "2021-09-16T12:37:57.374725", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/GoogleNet.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/ResNet.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/ResNetPreAct.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/DenseNet.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/tensorboards/GoogleNet/events.out.tfevents.googlenet...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/tensorboards/ResNet/events.out.tfevents.resnet...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/tensorboards/ResNetPreAct/events.out.tfevents.resnetpreact...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/tensorboards/DenseNet/events.out.tfevents.densenet...\n"]}], "source": ["# Github URL where saved models are stored for this tutorial\n", "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial5/\"\n", "# Files to download\n", "pretrained_files = [\n", "    \"GoogleNet.ckpt\",\n", "    \"ResNet.ckpt\",\n", "    \"ResNetPreAct.ckpt\",\n", "    \"DenseNet.ckpt\",\n", "    \"tensorboards/GoogleNet/events.out.tfevents.googlenet\",\n", "    \"tensorboards/ResNet/events.out.tfevents.resnet\",\n", "    \"tensorboards/ResNetPreAct/events.out.tfevents.resnetpreact\",\n", "    \"tensorboards/DenseNet/events.out.tfevents.densenet\",\n", "]\n", "# Create checkpoint path if it doesn't exist yet\n", "os.makedirs(CHECKPOINT_PATH, exist_ok=True)\n", "\n", "# For each file, check whether it already exists. If not, try downloading it.\n", "for file_name in pretrained_files:\n", "    file_path = os.path.join(CHECKPOINT_PATH, file_name)\n", "    if \"/\" in file_name:\n", "        os.makedirs(file_path.rsplit(\"/\", 1)[0], exist_ok=True)\n", "    if not os.path.isfile(file_path):\n", "        file_url = base_url + file_name\n", "        print(f\"Downloading {file_url}...\")\n", "        try:\n", "            urllib.request.urlretrieve(file_url, file_path)\n", "        except HTTPError as e:\n", "            print(\n", "                \"Something went wrong. Please try to download the file from the GDrive folder, or contact the author with the full output including the following error:\\n\",\n", "                e,\n", "            )"]}, {"cell_type": "markdown", "id": "98953f51", "metadata": {"papermill": {"duration": 0.032879, "end_time": "2021-09-16T12:37:58.416719", "exception": false, "start_time": "2021-09-16T12:37:58.383840", "status": "completed"}, "tags": []}, "source": ["Throughout this tutorial, we will train and evaluate the models on the CIFAR10 dataset.\n", "This allows you to compare the results obtained here with the model you have implemented in the first assignment.\n", "As we have learned from the previous tutorial about initialization, it is important to have the data preprocessed with a zero mean.\n", "Therefore, as a first step, we will calculate the mean and standard deviation of the CIFAR dataset:"]}, {"cell_type": "code", "execution_count": 5, "id": "ccec76a0", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:37:58.484521Z", "iopub.status.busy": "2021-09-16T12:37:58.484053Z", "iopub.status.idle": "2021-09-16T12:38:01.516073Z", "shell.execute_reply": "2021-09-16T12:38:01.515592Z"}, "papermill": {"duration": 3.067595, "end_time": "2021-09-16T12:38:01.516189", "exception": false, "start_time": "2021-09-16T12:37:58.448594", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Files already downloaded and verified\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Data mean [0.49139968 0.48215841 0.44653091]\n", "Data std [0.24703223 0.24348513 0.26158784]\n"]}], "source": ["train_dataset = CIFAR10(root=DATASET_PATH, train=True, download=True)\n", "DATA_MEANS = (train_dataset.data / 255.0).mean(axis=(0, 1, 2))\n", "DATA_STD = (train_dataset.data / 255.0).std(axis=(0, 1, 2))\n", "print(\"Data mean\", DATA_MEANS)\n", "print(\"Data std\", DATA_STD)"]}, {"cell_type": "markdown", "id": "e32a02e1", "metadata": {"papermill": {"duration": 0.03235, "end_time": "2021-09-16T12:38:01.582422", "exception": false, "start_time": "2021-09-16T12:38:01.550072", "status": "completed"}, "tags": []}, "source": ["We will use this information to define a `transforms.Normalize` module which will normalize our data accordingly.\n", "Additionally, we will use data augmentation during training.\n", "This reduces the risk of overfitting and helps CNNs to generalize better.\n", "Specifically, we will apply two random augmentations.\n", "\n", "First, we will flip each image horizontally by a chance of 50% (`transforms.RandomHorizontalFlip`).\n", "The object class usually does not change when flipping an image, and we don't expect any image information to be dependent on the horizontal orientation.\n", "This would be however different if we would try to detect digits or letters in an image, as those have a certain orientation.\n", "\n", "The second augmentation we use is called `transforms.RandomResizedCrop`.\n", "This transformation scales the image in a small range, while eventually changing the aspect ratio, and crops it afterward in the previous size.\n", "Therefore, the actual pixel values change while the content or overall semantics of the image stays the same.\n", "\n", "We will randomly split the training dataset into a training and a validation set.\n", "The validation set will be used for determining early stopping.\n", "After finishing the training, we test the models on the CIFAR test set."]}, {"cell_type": "code", "execution_count": 6, "id": "23070215", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:01.654133Z", "iopub.status.busy": "2021-09-16T12:38:01.653629Z", "iopub.status.idle": "2021-09-16T12:38:03.932272Z", "shell.execute_reply": "2021-09-16T12:38:03.931804Z"}, "papermill": {"duration": 2.317573, "end_time": "2021-09-16T12:38:03.932392", "exception": false, "start_time": "2021-09-16T12:38:01.614819", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Files already downloaded and verified\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Files already downloaded and verified\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Files already downloaded and verified\n"]}], "source": ["test_transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize(DATA_MEANS, DATA_STD)])\n", "# For training, we add some augmentation. Networks are too powerful and would overfit.\n", "train_transform = transforms.Compose(\n", "    [\n", "        transforms.RandomHorizontalFlip(),\n", "        transforms.RandomResizedCrop((32, 32), scale=(0.8, 1.0), ratio=(0.9, 1.1)),\n", "        transforms.ToTensor(),\n", "        transforms.Normalize(DATA_MEANS, DATA_STD),\n", "    ]\n", ")\n", "# Loading the training dataset. We need to split it into a training and validation part\n", "# We need to do a little trick because the validation set should not use the augmentation.\n", "train_dataset = CIFAR10(root=DATASET_PATH, train=True, transform=train_transform, download=True)\n", "val_dataset = CIFAR10(root=DATASET_PATH, train=True, transform=test_transform, download=True)\n", "pl.seed_everything(42)\n", "train_set, _ = torch.utils.data.random_split(train_dataset, [45000, 5000])\n", "pl.seed_everything(42)\n", "_, val_set = torch.utils.data.random_split(val_dataset, [45000, 5000])\n", "\n", "# Loading the test set\n", "test_set = CIFAR10(root=DATASET_PATH, train=False, transform=test_transform, download=True)\n", "\n", "# We define a set of data loaders that we can use for various purposes later.\n", "train_loader = data.DataLoader(train_set, batch_size=128, shuffle=True, drop_last=True, pin_memory=True, num_workers=4)\n", "val_loader = data.DataLoader(val_set, batch_size=128, shuffle=False, drop_last=False, num_workers=4)\n", "test_loader = data.DataLoader(test_set, batch_size=128, shuffle=False, drop_last=False, num_workers=4)"]}, {"cell_type": "markdown", "id": "98cc6aa1", "metadata": {"papermill": {"duration": 0.033181, "end_time": "2021-09-16T12:38:03.999425", "exception": false, "start_time": "2021-09-16T12:38:03.966244", "status": "completed"}, "tags": []}, "source": ["To verify that our normalization works, we can print out the mean and standard deviation of the single batch.\n", "The mean should be close to 0 and the standard deviation close to 1 for each channel:"]}, {"cell_type": "code", "execution_count": 7, "id": "32ca685e", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:04.069474Z", "iopub.status.busy": "2021-09-16T12:38:04.069009Z", "iopub.status.idle": "2021-09-16T12:38:06.979180Z", "shell.execute_reply": "2021-09-16T12:38:06.979562Z"}, "papermill": {"duration": 2.946994, "end_time": "2021-09-16T12:38:06.979707", "exception": false, "start_time": "2021-09-16T12:38:04.032713", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Batch mean tensor([-0.0088, -0.0180, -0.0446])\n", "Batch std tensor([0.9446, 0.9240, 0.9487])\n"]}], "source": ["imgs, _ = next(iter(train_loader))\n", "print(\"Batch mean\", imgs.mean(dim=[0, 2, 3]))\n", "print(\"Batch std\", imgs.std(dim=[0, 2, 3]))"]}, {"cell_type": "markdown", "id": "ee040290", "metadata": {"papermill": {"duration": 0.03343, "end_time": "2021-09-16T12:38:07.048437", "exception": false, "start_time": "2021-09-16T12:38:07.015007", "status": "completed"}, "tags": []}, "source": ["Finally, let's visualize a few images from the training set, and how they look like after random data augmentation:"]}, {"cell_type": "code", "execution_count": 8, "id": "06e3c206", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:07.121209Z", "iopub.status.busy": "2021-09-16T12:38:07.120734Z", "iopub.status.idle": "2021-09-16T12:38:07.258239Z", "shell.execute_reply": "2021-09-16T12:38:07.258625Z"}, "papermill": {"duration": 0.176795, "end_time": "2021-09-16T12:38:07.258770", "exception": false, "start_time": "2021-09-16T12:38:07.081975", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDQ2MC44IDI1NS45NTk3ODI2MDg3IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSCi9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTIgMCBSID4+CnN0cmVhbQp4nFVPu27DMAzc+RU3xkNlUZYtaXSaxki3BAY6FB0CV3UTxE5dp4/PL52+CZAi73Q6irGntGS0IzT2km9gVEgX8XXXxE01RzOSFrwjW2jlpTt8dSbPVciD85lA+v/4SNTTAKfMOa0tlIUxUjPrvCm0d3iOuEGPtDSTO4s7i7tGJUJrHSZXY9zPI02HdMVYHLGmNYZvoUb7VzzNNBDLeaEFZKNV0M6zyWGyoIrfBZqO5jXS5XQJ9cP5m/U93WJWvrRd7E/b0+7YIybgQgXjtM+mwOx92z0d4gghL1fLBIEV2/yTFbrcsE5wR/U1XdUky9IHrzFPQgplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjI0NQplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagoxOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDkxID4+CnN0cmVhbQp4nDWMuw3AMAhEe6a4Efg4gPeJohT2/m2ILRfcPemJ82xgZJ2HI7TjFrKmcFNMUk6odwxqpTcdO+glzf00yXouGvQPcfUVtpsDklEkkYdEl8uVZ+VffD4MbxxiCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzUgPj4Kc3RyZWFtCnicNVFJbgAxCLvnFf5ApbAn75mq6qH9/7WGUS8DA9jYJO/BRiQ+xJDuKFd8yuo0y/A7WeTFz0rh5L2ICqQqwgppB89yVjMMnhuZApcz8VlmPpkWOxZQTcRxduQ0g0GIaVxHy+kw0zzoCbk+GHFjp1muYkjr3VK9vtfynyrKR9bdLLdO2dRK3aJn7Elcdl5PbWlfGHUUNwWRDh87vAf5IuYsLjqRbvabKYeVpCE4LYAfiaFUzw6vESZ+ZiR4yp5O76M0vPZB0/W9e0FHbiZkKrdQRiqerDTGjKH6jWgmqe//gZ71vb7+AENNVLkKZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc2ID4+CnN0cmVhbQp4nDM1N1UwULC0ABKmhuYK5kaWCimGXEA+iJXLBRPLAbPMTMyALENLZJaJsSGQZWJhhsQyNrGAyiJYBkAabE0OzPQcrgyuNAA1FxkFCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA1MSA+PgpzdHJlYW0KeJwzsjRVMFCwtAAShpbmCuZGlgophlxAPoiVywUTywGzDIA0WGkOTEUOVwZXGgC/jA1WCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzIgPj4Kc3RyZWFtCnicPZBLcgQhDEP3nEJHAH/hPJ1KzaLn/tvI7plskKrA8hNxHBNn84gIpBz8rGFmUBO8h4VD1WA7oOvAZ0BO4BoudClwo9qEc3ydw5sKmriHx2y1SKyd5Uwh6jAmSWzoScg2zmhy45zcqlTeTGu9xuKbcne7ymvalsK9h8r6OONUOasqa5E2EZlFaxvBRh7ssM+jq2jLWSrcN4xNXROVw5vF7lndyeKK769c49Uswcz3w7e/HB9X3egqx9jKhNlSk+bSOfWvltH6cLSLhXrhR3smSHB1qyBVpdbO2lN6/VPcJPr9A/TBVx0KZW5kc3RyZWFtCmVuZG9iagoyMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMwNyA+PgpzdHJlYW0KeJw9kktuAzEMQ/c+hS4QwPrZnvOkKLqY3n/bJyXpihzZFkVqlrpMWVMekDSThH/p8HCxnfI7bM9mZuBaopeJ5ZTn0BVi7qJ82cxGXVknxeqEZjq36FE5Fwc2Taqfqyyl3S54Dtcmnlv2ET+80KAe1DUuCTd0V6NlKTRjqvt/0nv8jDLgakxdbFKrex88XkRV6OgHR4kiY5cX5+NBCelKwmhaiJV3RQNB7vK0ynsJ7tveasiyB6mYzjspZrDrdFIubheHIR7I8qjw5aPYa0LP+LArJfRI2IYzcifuaMbm1MjikP7ejQRLj65oIfPgr27WLmC8UzpFYmROcqxpi1VO91AU07nDvQwQ9WxFQylzkdXqX8POC2uWbBZ4SvoFHqPdJksOVtnbqE7vrTzZ0PcfWtd0HwplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM0MSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQ3ID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjU4ID4+CnN0cmVhbQp4nEWRS3IEIAhE956CI4D85DyTSmUxuf82Dc5kNnaXqP2ESiOmEiznFHkwfcnyzWS26Xc5VjsbBRRFKJjJVeixAqs7U8SZa4lq62Nl5LjTOwbFG85dOalkcaOMdVR1KnBMz5X1Ud35dlmUfUcOZQrYrHMcbODKbcMYJ0abre4O94kgTydTR8XtINnwByeNfZWrK3CdbPbRSzAOBP1CE5jki0DrDIHGzVP05BLs4+N254Fgb3kRSNkQyJEhGB2Cdp1c/+LW+b3/cYY7z7UZrhzv4neY1nbHX2KSFXMBi9wpqOdrLlrXGTrekzPH5Kb7hs65YJe7g0zv+T/Wz/r+Ax4pZvoKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE4ID4+CnN0cmVhbQp4nD1QuY0EMQzLXYUaWMB67alnFotLpv/0SPn2ItEWRVIqNZmSKS91lCVZU946fJbEDnmG5W5kNiUqRS+TsCX30ArxfYnmFPfd1ZazQzSXaDl+CzMqqhsd00s2mnAqE7qg3MMz+g1tdANWhx6xWyDQpGDXtiByxw8YDMGZE4siDEpNBv+uco+fXosbPsPxQxSRkg7mNf9Y/fJzDa9TjyeRbm++4l6cqQ4DERySmrwjXVixLhIRaTVBTc/AWi2Au7de/hu0I7oMQPaJxHGaUo6hv2twpc8v5SdT2AplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODMgPj4Kc3RyZWFtCnicRYy7DcAwCER7pmAEfib2PlGUwt6/DRAlbrgn3T1cHQmZKW4zw0MGngwshl1xgfSWMAtcR1COneyjYdW+6gSN9aZS8+8PlJ7srOKG6wECQhpmCmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzkgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzQgPj4Kc3RyZWFtCnicLVJLcsUgDNtzCl2gM/gH5DzpdLp4vf+2kpNFRg5g9DHlholKfFkgt6PWxLeNzECF4a+rzIXPSNvIOojLkIu4ki2Fe0Qs5DHEPMSC76vxHh75rMzJswfGL9l3Dyv21IRlIePFGdphFcdhFeRYsHUhqnt4U6TDqSTY44v/PsVzLQQtfEbQgF/kn6+O4PmSFmn3mG3TrnqwTDuqpLAcbE9zXiZfWme5Oh7PB8n2rtgRUrsCFIW5M85z4SjTVka0FnY2SGpcbG+O/VhK0IVuXEaKI5CfqSI8oKTJzCYK4o+cHnIqA2Hqmq50chtVcaeezDWbi7czSWbrvkixmcJ5XTiz/gxTZrV5J89yotSpCO+xZ0vQ0Dmunr2WWWh0mxO8pITPxk5PTr5XM+shORUJqWJaV8FpFJliCdsSX1NRU5p6Gf778u7xO37+ASxzfHMKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE4ID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzMyA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTc0ID4+CnN0cmVhbQp4nE2QSQ5DIQxD95zCF6iEM8DnPL+qumjvv61DB3WB/OQgcDw80HEkLnRk6IyOK5sc48CzIGPi0Tj/ybg+xDFB3aItWJd2x9nMEnPCMjECtkbJ2TyiwA/HXAgSZJcfvsAgIl2P+VbzWZP0z7c73Y+6tGZfPaLAiewIxbABV4D9useBS8L5XtPklyolYxOH8oHqIlI2O6EQtVTscqqKs92bK3AV9PzRQ+7tBbUjPN8KZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDg5ID4+CnN0cmVhbQp4nDVNuRGAMAzrPYVHwI9IvA/HUYT9W+yENJZOnxHKB2vkAYLhjS8h+KIvGYS1Cw8q+0h02EQNZxUkE8OvLPCqnBVtcyUT2VlMo7NBy/St7W+DHro/3Y4cCgplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE1ID4+CnN0cmVhbQp4nDVROQ4DIQzs9xX+QCSML3hPoijN/r/NjNFWHsFchrSUIZnyUpOoIeVTPnqZLpy63NfMajTnlrQtc4C4trwvrZLAiWaIg8FpmLgBmjwBQ9fRqFFDFx7Q1KVTKLDcBD6Kt24P3WO1gZe2IeeJIGIoGSxBzalFExZtzyekNb9eixvel+3dyFOlxpYYgQYBVjgc1+jX8JU9TybRdBUy1Ks1yxgJE0UiPPmOptUT61o00jIS1MYRrGoDvDv9ME4AABNxywJkn0qUs+TEb7H0swZX+v4Bn0dUlgplbmRzdHJlYW0KZW5kb2JqCjE2IDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2FucyAvQ2hhclByb2NzIDE3IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ4IC96ZXJvIC9vbmUgNjUgL0EgNjcgL0MgNzAgL0YgNzMgL0kgODIgL1IgOTcgL2EgMTAxIC9lIDEwMwovZyAxMDUgL2kgMTA4IC9sIC9tIC9uIC9vIC9wIDExNSAvcyAvdCAvdSAxMjAgL3ggXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDE1IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDE0IDAgUiA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxNCAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNyAwIG9iago8PCAvQSAxOCAwIFIgL0MgMTkgMCBSIC9GIDIwIDAgUiAvSSAyMSAwIFIgL1IgMjIgMCBSIC9hIDIzIDAgUiAvZSAyNCAwIFIKL2cgMjUgMCBSIC9pIDI2IDAgUiAvbCAyNyAwIFIgL20gMjggMCBSIC9uIDI5IDAgUiAvbyAzMCAwIFIgL29uZSAzMSAwIFIKL3AgMzIgMCBSIC9zIDMzIDAgUiAvc3BhY2UgMzQgMCBSIC90IDM1IDAgUiAvdSAzNiAwIFIgL3ggMzcgMCBSCi96ZXJvIDM4IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTYgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDggL0NvbG9yU3BhY2UgL0RldmljZVJHQgovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAzIC9Db2x1bW5zIDQ0NyAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMjI3IC9MZW5ndGggMzkgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggNDQ3ID4+CnN0cmVhbQp4nO29V7Nl23kdNlfYOZ4c+nROt2++SMQFSIAEKFIUlYNVtqtc5Qe/2P4DLpcNl/SgP+AHl23JdjlUSVW2RUkUBRki8kUGLnBjp9vh9Mlpn53DCn5YY3zfRJ+NY2gbJMWqOV569t5zrzDXXOusMeb3jc/70pe+ZBwcHBwc/i3h/1kfgIODg8OfS7inp4ODg8MscE9PBwcHh1ngnp4ODg4Os8A9PR0cHBxmQXj2o/m4rV97adbIF3L4KImzf700kW4lfjuZjLJGr9fPGp1uV7rFMX4Smwk3m8d3nj7HT1udrFEuVbNGwKf8cDCSbsViIWsUCmgkAQ7jpIVTyFvnNz/XkOPI/o1Gg6wxjiPpdtTCAR8c4xSeHqHbydiTbqMEh9RsNrPGX/itT5gz+MZb38b58aeVahGNSkW3NsKAHB+dZo2LG5eyRqlYlW6n7VbWqNVL+CjFKf/+7/9+1vjNL/y6bnaIU+gPcAqFHPrXKjXplvg4uIgH2R9inHNRoHs/PcoaW3vbWePwCFeqUGtIt+EEV/ne3UdZI5/DBSoFOmee3Hs3a6xcvpY1vFIde3/2vjmD/+K//q/Ofjgz0jMNz/DYOD2MMdF4nDXGIwyIxzl2etqRbs3mXNYoVUrmOXjPf/CniX/w9/7+2Q//l3/4j7KGx+uexHpdcpwhaZI8183z9Sb1OKGLvMdyvIV7nV7W6HP0jDG1BuZbvYL5XCzxRiiXpdvmU0ytcYTfFgrYfqWqN8JwMORx4N+5eczAWrmgOy1jF3ke5AHvr6OWXr5SqcR98Tk2xvaX5/U2ufbKZ83Pw717Ojg4OMwC9/R0cHBwmAXu6eng4OAwC6bontv335N2EkHuCaiGlIsigKpWksaQ7dIEAuKgD+3DD3QXBaqcgwiywlgkS9WazPHhcdZo+3nuFMrIaKS6Z8zfhCHEjyF1uyiGnJUL9M9De5fqG0/BS3HYXqDqlDfCefX3D7JGaxdK6OMj3fsJhZfG4kLWmKp7hmHII4eOIxJrkkykW60GgabdhmJ72j5BN6PKY64AgWYwxFYurF/AjnJQhe7ffSz9xxMcZa1OAWgOCl000RH/zk/eyRo/vf8wa1x74Tb2mOSlG6+VWVheyRpXVzd4ntptMqGoncO5bz/ayRonB7vSTcTQJMKAp9bFPQvf/1X+pZ+mewJxpNfl9AhS7/3372aNShWXIF8oSrdKEadf4re+6Ph/prrnVORCHJvHg6w16vLthPdyHGGGyHKFSWS0TJVnurg0j61xZh9wVaPqqQrcmG9mjTDBfJZbo1ZUpbJZx2JA4hW4I3xSrurW2hSdCxz5Wq3Mw9bLJ+srISfb3BwPI6czNuADJAhwgpMhzj2xbpOzcO+eDg4ODrPAPT0dHBwcZsEU5t5ta8RSpYI33ojvwzFfvKtWcEa3w29JVMIy3rc9KxSpyJdwM8Zm0xSvyuORBjeUywg+yOfRLRdSN6hplI/Q0jBH5n6EAJ0cX9TzoRKniCFW3T72VSS1LBaUOxT59r68BEo+SXEKidGRSQ5BM0tThlBRr4NQHx8jVCIiUW2dtqTbcn4pa8zPg0ONyM0jK5qqxDCjwMcBh3lchQ8/BOmu116R/rU66MzCAuhVKPzXV+byznsPssY//oN/lTX+wl8nuUt0ZHo9iAmrq82sMddAo1zXwI4KKe3GCgj+xtJy1th+rAzx3fcxDgG1HQmrmgqZKr8STNsWPkutiKVuq5U1th8h+spPOHmsgDMJnquRGKacsVP39GeLehWzQuIF5xfm5NsOYw1FdBgzfK2Q07le40aqjMDzGN1Y5C0ht629tZzJ8Vt0m0x0hq8vY6ImOVwFjUe0pEK/QYWEmltKFS607seA33Z6EN8GfXSzRAhT4aNM2PzYQysX2afwPNy7p4ODg8MscE9PBwcHh1kwhXae9nTps1rFG2ye+QaGy+Wjrnbz5aWa3cZ8Gy9YpDjmcl4+AP2fSApHpG/vlSK4wIAkIjZoFIv6Il0qcMWwhhfv9im6RVzOrlY0PyEeY1/tFroNT7mpvrAVE/g4lxxf7ldq2GkYNqVbb4TV2Il33qpcjmRAMov6ffDTfF5VhUDSORKmfzBjx0RW6kuKcVteBgX2Uxz53CLO9Mr1dd0sV1d9nwPOre1u6/L3yVEL22cKx/Eevp1fuCDdjvYxXuM+uh3m8Uni2xMJp7Cxvopja5IZ+XrKk/ICzx0HGQZYSNWpYG/U+1WuXosMkFLPkdy5aDSUbr2TFj4cIIakxAiIUUeTVQ52kCGzfu1q1sgHz498tpNfxbH//8Ucl78FnqWKNJvQzWQ5e9DFHCtZ69SSXjieYD5HXOAuV3D/Bta5J2M+EHjv+5wM3Y7efTJC8u14jOtiE3xJf5pws6EMuJVfOBjikEYjiXThIVnTaTQePNc/ZdBOV3UMc9E8D/fu6eDg4DAL3NPTwcHBYRZMYe5bh7r0eXQKwlIto2ezjDf2ZlUpuS8LalyVk/jhfE5XY/uMvS9xKT9mNn5qeY4MRmCXI8bedwf0DRkqd6iVJXQc79vyGi+uB2GqBgQm5Es7F+JLfMm3l+aFxchyXqWEjUS+coclahoDY1Oz5zEeP09DZYGvkFNKPiQXTmJSEvqG9FMlksUCmQXjDTyDjczPY7PlmkZxT8b4VjSNzc2trPHsyZZ0ixnwcOkCePrO9n7W2No5kG7Hu2hXOPLi+JALNQAjovhweoI1+kIZYxVYK9CTCD+5wNiGZhkrv51TPbY/NUhwePfkVD7c38aRxPRb8anADEYalS0yyM0uJmquKFfhvOnxZ4Icb4QJXTxGQ51jOQnzH+NiScaJZ616iw4WTbCRASn8gPpP2UooEKOfvsaf80azHkIxI0zG3LsX44exJe5JpL0s9MvWJKbFWFkzHjUECQMIrDwan7+VXBU5hYIV8H8W7t3TwcHBYRa4p6eDg4PDLHBPTwcHB4dZMEX3/GjzSNoBZYjLFxEi48XQMkIriUgSD0o1SFdhiv77u5pE9PTwGbo10D+XYvvtvqpIE9p/1Av4NqYq2jcqahjqIEEKCSalepnGNLUd9qR76tEchKFIuRy2VsyrnOrzL4rIQyNKPMWinvLqEiS/zuS8bIThcMB9oVuJprCjgeprgz520Wwi1yKYo0DjqXRb4G+blJWvb2DAlxcRaJJYWRR7e7iUDx88xSe7e2hsb0u3o32onNdv3Mwa24zw2N99JN3aXeiY+3RHFjGrVlR5qEzJ73CXqVwVHFspr90mCQTEJj2wL60hwqmjwVS/SkzNVhIP4AFtvLefbsq3O08wbl36bZ9yJaA9VBmuxuE6PMSAV+dwHX1L1fvVBl3NDJ83kRU/ZmmFtCWesFGkIfTPmenQHTzQuw7/jkXZjHQFRdTfIC/KI+1C8ioNT4bYSsi7L5W7Na8LLeI3xG2YiE+DwUivi0iged7ghVAeAnrKEgvFMzZphKtWaGpG2Vm4d08HBweHWeCeng4ODg6zYApzv3ptVdorddYO8umhwKgjz0qcKBbwAt0o4cW4e4ivgrG+b792++Ws8Z33384ap23QpSjVreWZvbO8gPdtsezb6isXGPtoVwvoL7EvDZpWeIm+xjdoYigFWCYD0LFR3/IoKQm7pFMG/8RI5pUx5oKPV/r2+DybkMFQMotI8BljcXysySqB7kLMDkKei7Ld6zcvZ43VErpFbdD/Ug6X4OecUmlNIiYsm09htfn4o4fS7Wi/lTU+dv2lrLGyCEFgbU0nw/EaGPX2U8TxtBncs9rUg1yeh9SwtYv+J3sIdTpRecYkZPH1JkZmbflPmNhaxNkjz4zHOKaDbY7Mh/el28kBmHiXmUVbh8LcNdfuEmfvIa1pL17FgOes/JxpbqK/zFe/YoiAIFE7tp1HZCkN+EeDglT6KIt5B5OOhKaHqcQL6k5TaoBFZiKJP6+tIFToa0NJTyPJ8pa8JvKXSbkRXoJ8Xt8I5e5P0+dfE+1STmOGSJqEhJ3KQKHsIpYcHBwcftVwT08HBweHWTCFdd64oNVlV+axHirZFCEZaL6gK2Vq3hFLBgKC9QddZWtei66U1AEkVaBkFfD49A34XPytj4Oobu2ALv23X35bum0P8L5dDvGCPuyBid+4jBXPtSVdMpNqyWIoUCaXtxdDewORJnBImr1jpZfEHohKaOX2nIVkcQwLMm4hD0PpkhDJDhniEgsebFycl243r8AG9OQuqqc8eA+NGx//QtZILA2k2QABH3GsnjzDmvv9x7rmnqMjScpFyiAEgVpb3pBu8zWw+HyAQ+rOYcCvLqtx5/oKyM7GJRD2+/cxebY296RbXAI5avdbWeO9n34va1xZaZqzsLLRnv/GCv84k3tiFeHQ70xCxtdtgZtv3r2Hg/xIwwyknvYWxY1HWwg8iCy+ubgBnp6QjSbJNO+Y9HlWnupRytlZfTgtVSX7/6D1v5SZaIcBHsJeiyW9l0UyEveWgKdSLip3rtOaR6p5D+l8Kkl65ZzOcNlaSmZ9yKonoVXWRdbQx1w6lxgYYyURpZrCx4QoruoH9hCRukf0bE2kkLLlVWqYHTfo4W6tMIhIqphMhXv3dHBwcJgF7unp4ODgMAumMPfLS7rMNKA9x/wcvS3I3O0Sh2Lv2GJByq02ArCfHOjKcnLCVWzutkZ3iZXmonS7WQc/LR6DJS2FIAX1or69B1wgS/k34KSN9/OPnuGHy6uXpL/4aZ6e4JBSvr3bnDDyGCRP7jAhH5IIXmNMkbG7pXnQ2KkFIQtFVingmrg0llivwhjToqFGnpHDt1+AU+SLd9RXsJLHCW4yvj1qg2v4kQQtK3WZnwdzPzkBA22Tm5yOlOWtiqUFydF4gAGPIosM8rIFIc69XEZsg++pj+poiMuxcRGnUKqC8tfrGgffJvNNUsgyD97+Vta4svIpcwbnFOZILMbq8fT9+HkOn1h+HcM+BmTnIWIPtj78IGsMuG5ujDlkTMjDfVyg9pD81GKIhQqI3ur6WtbI5c7LoRDCLuY4qawdW+Tco52r/dmZT4z13S/F3E/pzDukrFQYWOVdGxC78nTWqFBVy1Wte96TUj2MEuHBSnxO1Vr+LtBqZ7+FOI2A8erlnF6YMQuyFngj+CbPHerZhVLvk58EEl0f6404odmHzzo0Ph8CoR2AwX0VWeOnRhujUfy8y48N9+7p4ODgMAvc09PBwcFhFrinp4ODg8MsmKJ7pioYaW5/ocy6o/zEs9RCj2qFOgVQmEgD3VqVgmqZeSY5KmiLcxr3c9BDMNBXthHgElLvm6urZca1Cg5+wP5DxjH0RpDSPvxIzX3v3LyeNSpzEEHG4s2sjq3GF6dkRk9UaYjbrDelW0K5xxQZ1GWmYIFuEVIlpkiRMVdWrTCgcHztGsTQl6l71qoq0MQD1ncq4MOFeVahyXFwLfuDlCWCRAv1C9ipV9QYo3IT8qVEO0lGWZRYKlLMEtBMIqHvil5uY8xxi87NZUaG1XCQ86uWLwWlVS/FHHjSP09jkrNKz6h73s+pgFQ5JTuOiqmkFRljDmkO/egdyJ37W5CST7payubZbitrtFocEN4d6xdWpNvrH389aywsQr73NQLv/CAjnJZEOKWWbB1IxJL//FuON134/KXylMTCvN3mJbBE3EGPejQj6spl3HSJ5U8spX0Hcu/wskipYTsqSCqYnVKCl2LdoZXpJGWHq1XOzxi3YTzUOZZjOmKPim3ER5BdUFpsj0OeYCB5htGUyVanJ4iEFZ4cds92s7bv4ODg4PBvD/f0dHBwcJgFU5i7Z2VuSMGikOELPkmEFEUxxkwiSeNh3dEQn1xY0dfy27cQz3H10htZ4+F9EKjxSMNE4hgZLG2S0Xod9h/X60q+rrJqyr2H8HcY9vCaXSA7PjlsSf8+a+surGJrHl1CctZfEeEOEVlVQTKprJAXj7wm9aeMoSDPPAqpUyQ8qGkVhr15A5FV1zZAWPKs7zqgoaQxJvQK3DuPiMfmS8CIxVwSj7yboRg56gxhXnWDXImBR8ysKBWf5+bG8l0NtPorvRoti0b5sD1GN9IgczpWHaDTQeTKHL1X6nMatXYWh20Yz0itWmF85byV7sXZG6WShQJ0j3SOPX0HViBbd+HgeURG+cxybznghxFD8YqUn37tMxpW9bFPfDxr5Ao0ueBuf15lOCMm8GKdHLeyRq+nPjiNBhSPJhs+z86O37JCb34p5t5kdpyf4LoXS1aNMgpWKTPQwkAGXK+yFCCasJvPSVnk5SiXNNNvm0WxYlbuKnAqjq1ZEfFbmc8TRtH5uSmvepMEMkufpi32VKwyZShl7Jd4ieYtVUFkkJCFzjoM7Otb6YVn4d49HRwcHGaBe3o6ODg4zIJpa+4WK5C1uJjm9YZ+hamvpC5PphwIVYzAy5bntCLxq6/dyhqLS82ssbAEyvDoQ31DblawmrnJMhK1BTCXsHUo3eaYFLG+BlfKrXs/zhplEs+PtpXGphFYWI/JMNvPsMxat5a/pdxuQteDLteabZsJGbo0JuuZll3S7bCSLZeAm1Xs6+aVBem2wroaKaMFoj5+WK4qE4nIksaUTUIaEXok3bG15i5eGGKekWdeR8GyZcmTsFRrOJfGAgahWNeRGZPO5IroNuAqdnuk5CvnkxgynUlWSE8te9Yh4wcMmXtzUZOvzuK9B+9ga1QGykxfWajMSTcv4IDQliXkoG1bxp1P34EnyMkO6PzBKabH06OWdJPMogKDQ164g+Ilv/nF35Bua0wxEt0gEXNMM+1uIu3d3Qeffe89LP23jk6k/+UrcMl5+aU7WUPCP+yJKMv0nrwMncvgK/T3aa5jBhYLytxzbJ8eQ9dKGVAxsRJvpKKJeO/myaxjDniL9rLGmNYJtjYek5Izd2ho1TiR2INeD9NjxFX1Qk4PUiQEjyE9kqQnd4QxxudVS83z4kOpZN2uHM0RbXDbdIcJrXIgZ+HePR0cHBxmgXt6Ojg4OMyCaevFFisIxZWPy3nCC1PrtyNaVHg0lWiQ8eVy6jmytdPCb3N4G6+Q8l+8qjR2eQk86OIdMJfER/9Wa9XqBjORwyMQ8EYBDO7aBTSGX1ay9mAbq6vFGGu7px1Q0aNjJZ43r8LRco3yQjwBiRhblhmeB7rh5VUcOIt+F1z12hUc+euvgPqtLOpKcb8N/hhQNqmWWRPFslpoDaTeA3hHkezVywlntFkdF8fJ3BsV/PDautLkWwzRv3YFI1NmZQ6/qMumVdZeOIzROOFK9HBskS+GT+fyODafPpI5T7tFIwx+zLjl9FzCef/Jh9gIV0vLRRpPFNQCNeBOy3TumByTkv/wHem28xEmwwkZ5Q4bh10NchAvnMsXMURf/B0Q9pdevSW98kyFOKU9a8LoCLGJsU9wyHO/e/9B1vju936YNVonLel/0EI7R2nl8gYmpxZ6MVZRChaVKRasIIQzaNBZtVCQg1S2W6ngzq1J7MQpTur4WFWFwUDcPCm8VDFVRjzj3kjzDiSlZZRgsx06sAShHTWA9phD5HEpP7YMXvMk4I05HG0iBjcT7TbhxJNGJAbEsSWIcQaKICaHIbktU+HePR0cHBxmgXt6Ojg4OMwC9/R0cHBwmAVTdM+cJakkjOAvUEnpUeGaJKqvFfmTHFNrLl2+kTVqC6qv1dcQvVQpMPGGQRxzTSvbJ4D6ME+P0sBA3ahv3JBui6uQRw/f+jc4mTpzM9aRunPhgobIPGX8084jNGIqaIOhaiVPWG63HODIK2Xm81jVZatV1mii3NOd5nGxdgF67m9+/pNZY6GBkRx0VEWSxIcasyMmDJ4IrdAin2Mu2l/I0qxScjW2wqoCRpXNNTCAv/W5j2WNkwNV99ZXcV1eegVGKtUmRjKZ6Fltl2g+nbSyxrDazBrFksalVXgVEgpVbUZffRhrCEswZHRUCecysrJEzqJ1ip0GTIZph9DjDoxmB5Vz1D2px508xAXdfUeLMHcPcCQSqLRPdW9g+UfUG9jazVcwo175+O2sEacaY9frQzPd2kXaW7EICbI5p8KxVPE6YSjP/fs4pEePocP2rULHY0a5xUx7293D1J1jSS5jTIXxdhsbKAhWKEyLniOWF5pZQy5QraZxaSFz5wqcUbUq5lino5YZp236eeeYgUbJPk8/42bdqsbsYYaMW5jYkxHuzVJR9y7pSSVazDSZgdZoalxaifJ9OsHWdrYwMicnOhnE7VtC9yJxY0ns24S3FeVUsU2JvWkrQ4R793RwcHCYBe7p6eDg4DALpryXDidKSSRlQj0/SSTFKsJY4QXLF0FUP/mXfi9rlJpqgzgh42uSm/cZA+FbckF9Ga/oMTM3cgW8vdeNxj8dbcFRsxo2s8Y7DxhFUQGvWX/1s9J/aefrWaP39JjHBjbR6ild6nfBwsJgjQ1+51nGKH2WXBZry6oGXQk+/okXs8bcHIhqMmZVJcvOY8yICsOUIXEu8K1xlowv8bjUci60coisWrg+vV1q4m3xSRzP5sNn0m3/ANWJaxVQrTnmtMS0QDXGFDZwgsvzNF8g/SmEel3KdGYUvnnA2rMLFpXbuYAQrtU1zJCPFvLs/8ScQesQR5JnMonvs7iNr9el4kMlyNFm9OTuZtZo7+7r1jq44s/aUDDapMxh0SrpvIwzXbqMk9o/wVg93dmUbsUyhmtrG7sQR86FRaXYwtwPDlpZ49FjnKl4mpQayvTHHgbww8cg+Ns7UAYkY80Yc+MyVIVaHRS7VLEo8xlETA+SsCe7IrdEksm0E+8Y2wZUjGpkjgU8O1G6btzQctZ5UvLdYxED0a1m1fQOOI0PDjCSL9+GWLd+UbcWUlYYdKGBeAylHFo1muIBy4YzsygIsa/E9lnhNJardtqh4uefF0Xn3j0dHBwcZoF7ejo4ODjMginMfRzpArTPkp/DMQhOKm/s1gJ0mufbPmnK+k28bxerV6Tb8S4WFg8PPsoajTmU25U0CWNM0Ghmjdo8DR9Z8HZoWTRuffSdrFEmu5wLwQ5+9v1HWeNv/Mf/ofT/xJheFf/3V7LGzjFe8vet8ryiUcQeSSuTZ5KJnTdC7lxQDnUWJZbQMBOsVxbJTX7yI82DunwN5qNLy8wOYgJGnOgS8IQlZBNyjVhywMhEIksQ8ORSpthIlGAL8/M64KUa1mrFaTGmehNZhSJKdSgqlSYovEwUT6eMiblT8cJYWMJ1XGLDGNNnqkmRKUOVFKT7j785hbn/8Dvw9SgxzECIZ5xXslbhOnLzlGvBx5i6/VNdMt5tY1/7UnuZJzpfV+58/RauS8ookW9/95v44Y5OxbEs4DLIIZdj/RUrkUa0mt4A43x4CJmoyuIlF69qDW1fas9wsTtmPYyDY9376jIux8EhPEd6VJ+mYosFb8QQp1TSpKD6GWEkETtXS0QS+4wya/bMs7jO4hKeAhc39HJ7JPglqhx5mpUUizmrG4eLK/IxL9CkpyJSncOV58VaXkK+2cCq73LMOIpJxCQ9qhC+benL6yI3teTLBTRsnQr37ung4OAwC9zT08HBwWEWTGHug7GGmxZqeH+uL9Fhky/2Qd6O4sb7M0OVTbsDI85iVd/e93Zh9PCd78KI8+WXX88aN25ot6SPLU8Y45oLQVgSywpwdRk/aT0BE7lKpnDyHhZG2wONCb/zmU9kjeOdx1kj/SE8GqK+qhCHR2AxKRfvwir+xowsEi0FPKyaoVNQZiBug56YO/ewbJoeaCR/ZZ0sZsTCJ4EE7louJFwtFYoRs36hYUkDY1mZiNenVYKRti9WQHW1TM9WsackFY9se0ryU1m0lVIQP8fpWAXE57GJY6y9gFkIWA6EVRNKlfNY0vZDhFhI/MAyK6ykqV7lAfldfsB6imM0Dnva7WgALjygzlBpYLPX71yQbq+/ASuQUp0kuoirlivrQr8YXnQ6+HBI/nh6qCS6L+v7A3w7nGBEag2cyzxlK2OMz4jxY4pOcYzDiBK9+27dgvvn2hI28vWvf9P8YvTJhUWkssW6GqM4fM6BMV1c65bZa4v1OFPGeNRqvC6MmZmzvH1lxhaoZUndylJJ7z5Zc/dYQXNIterUEiu8EANYLrN+J+dztaKSVJ+/nUS4qQO68JYs4VGegqJc1WgOEoduzd3BwcHhVw339HRwcHCYBe7p6eDg4DALpuietUWl+vU5ZhaFUBDGIwZPDLVb4NGrgt60UR8Sz/HeI+n2bBv2tIuLUB8mfaQK/OTb35ZuPg2V51YQSbOyDimqlNedlhfgjpwvwflivPA4a1xiPMrTh3el/53f+A+yxqu/DgXt6AQaVvTkQLo1c02clM98G1a+jSz/iPEA6nBoaUZnUafrh6FmOqBlxsa6Sr3VKnaRxPRnDRgvYomt4yE1WX4oZiW+wSe+bwWc0ePAZyTNSCoRtVXd7rItMT05ZpQVy6pFSh2hHAXNkCbNYah/hvNa9BXSUplSlO3p650pYFulvchUjFhuukLhbHmestpQ7UUGbQZdjTAgXcp8x33NKOvxUuYpul25gcn2+sfuSLfFORxSFNHLmV/VLE+TRabYHeyiW4eiYXFRT2rCQKXdE9wdT/Za+IoFlE62NSEq9nGQHSq24wHVwJ6lurJ2VvEKbpN267yIJZ8Xd8LJkA/0ukwYrBbmJJuIWnmskWFD1h0y9OnI5+Cqs8I1kkpZHy8FXuWYyqOE4tkzXDIY5+iucnyEQWtZAWcPN+H8Mse0Nyn5tbffkm5tyRsMeTk4US3PcTXiEZFTpN7Ekj3PLnC4d08HBweHWeCeng4ODg6zYApzX1nXuIT2HsJrThnHEzAuP7QIa8Qoh8USIv5rebKqQJnLlcsoHRzG2Nrje7Ba2H20I92kQmlMi8bFCzAIqFvkLk8i+dIbb2QNKXJSehd5Ke2OJlEY08z+2Xjli1nj+hbO4cc/+0fSaWMNuxiQN3U7pEt9zXYIWOxofl0Z3Fk0mjikwMdvVy5CcMil+qerSS+JOBDKQOZuyQXCoeo15ucwTyMl5T+iJYcxZmsTisTeLqI9nm1CtTjtaPxHRC/RLXqb1kjYr169Jt3KczjICQOhOi0IL22rGo+Qr6VFTIY8o7WM5UZRZrpIswkFI6GMMxXiJZEj6ZIgmMFAKXmH9NxnVFC3iyvVGln+EXSx3NgA2/3UJxHQdpG+scaY7gEGs9vBscX0t/WMbu24i3E+PcK3Jy1cvoUNnbHXr13JGlfvUBD4zk+zxt5jEPatkVLyEk9Zsn3EkSax6k7v7WHvw1vIU/LD896KFuegM0geVNlSVEpMcPI4A33e8sWc7nSpzsCgCDfpPCOWFptoVIt6axRLmANjv8BzwVj1rEJSBXYLKIycUIo5aGmE30dPERAZIHvRJAzU6/WU4NdYzXttBTqAZPp5kV4+j8MrNi5Fqhax2iMZvfkJ9+7p4ODgMAvc09PBwcFhFkxh7pbXpbn/BP/xmLlRlVqdJX3ypvSdXORCfJv++/OW6f/Gpdezxr13YPDx+AmY+9HOlnS7ugHDxzbp0t3vwVWkWFAHhy5zgQoGB7lew/v2aY9rbZZb4vFTrP4vX0MCyZu/97msMRxsS7cHP/5e1oiZeOBxSS5Xtk55yHoSQhs1sUIh9YTll81F9Etth0GuI0daS4Bf+XqNpEpBjmQq4YalFvRHD9Vi48t/9LWscXSMASkWwKouXNCKKRs0iFxdhKoglaJLec3c+Mk96CEVLqoeUyV49IEWvZAq1v2rKCgt6/t7B4fS7fbrH8e55FtZo7uHI3/5Ra0wLFhcxYfC/rt9cDR7NbYr05fZPgPmGg2NEsmQDFFMKwa0dPzgZ+re0j+FviFlJGTVu1TSrR23YFPbaWFO9ni/dGM9tgpdU+eZXmV44wwofewdW5l+QmMrVC24hchK23r3nXezxkIjz+PB1rxpRZ4DRkpUuP1CTs8lT+vM8URKRsfspgT/4gXcpCHz05osFpJnoWO7qIwhKQ4YTCKFiJNYu8necylvOtLqZ9u70m2PGVw5Rm7IDZNYLjk1iUihBmiY+yd5jMaYwQDazmAkH+LYJhMr0+8M3Lung4ODwyxwT08HBweHWTCFuUdW8GqpCI5WY2RyQqe83kjXoAYDtK8UQP12D7AO+HhHyVqV5QSePUJZiH4Hy22RVbtR+ODaCthlMpQFaF31G8XgaO1dsP54D/SqdQImVStqHb4HH8Kj5IireFevYvuvffx16fbkPSyDThgPn3Knnmcxd5ojJKkukp5FLkfTf/pOSuC6OKUaowU55IrkyX/3d7X05r0fgyCvc/k7ypOSzDF6P9D49pdfeSVrzHNNf3mpmTUaNV0LrtLnkauspkLyNerr2e30cSSVNZDoAo0ePrKYu2E4tO/h2KRo4mikWQkvvIoch2IFW+vvQpdIh5phIZhj2LkoHjIVRtbCaEcMNci5YqkuE1puFCFOsE3njp+9/UHW8KyY8FCSF7jTId1YKjXVNGQBvNXGbz1uP451zpwcttifQ0S+GdJHddBS5j5k0odPG5d8jUkrevZm2IbAxdqX5qSDHS03VZ8RiDo0YvxAzlqj749wd0QMm5faGznLWSNiOoBYwQ55C4848lFs6Qak/2Jcm+NmR54VVcL3uQm3lmf/Y8vb97AFPaTgc2SYGGKfizxUBnyAxCEG3NbNJE7jlJNB7s3Uiiw6W3jHvXs6ODg4zAL39HRwcHCYBe7p6eDg4DALpuieL710Wdp+DD1rYCDBjJiB4+vqvqn0oLW1GSr03a/9Eb6zlvylbm1XJEUmbIgdqTFmQunq6ABKR0D5YTzScKpVmrCWi0zyn0CtqFag0uZyBenfoxg64ZG/9xZqFB/vaZRPjfa0J8wzyXGUAktSUYdgS4E6iyQJeOT0M2YgSN8qpiQqT5fKy3CI7T+891i6ffQjXI5rq9CzPJrCHk1Q4HfjY5+Q/iuseBOznFEuh1NotzSxZ38biV6STFUrQ6kshDpDxiPMgUoCFbVII+SC5bXsUdTr9wb8lqVyApV6B7yUYREDePEiknye3p+iexZY+UpkuPFI8rJ0s2MKiLF4W0jDkq1jCo7HRxiH1j62VrZsv8sSGMSNxAVM1EHfmtjM1RlS1C4zWatc1kC2MKUYyhkYiMwtOqlnBQzygFOGo41Y4MguqDtiNNUeU8WCut5NZ3HK3J4gwFX2rX2Kjl/gbSV222MrD2o8kqUIjNsRiwg922a2m5UQVSwyJYnhbj7rGk0sy3Mx+Y5YrCzHS9Bta+zX02eIXirTj73JW75e0XMfjnAAbbpWGylebaW9deiofUrrk4R5gJ4V5eZ0TwcHB4dfDdzT08HBwWEWTGHuC8ua6VFeROWf3TZIwaSCl+Gcr+EL5QHexk/3kLWfH4Fn1fOa7RMnzHiR3aZT4gykKX6aMRu+lXgjJhEH+2AKV9bBZ2+/8lrWaA2U4Jwe4m1/bJBZ9PQuYpiGYzUgWKG3RYMuJBrCYllnylEmyZR0DsFRB4zsxz9AutQJj7bdm1JquM9Ume4QR35qGXDEJxjAzil+WyIp3jz9YdZYfell6f/tb3wra9y9D5/TS5euZI1KSSOWkoQ+j0wmWV+HLcvSkk6GQResZ5PeFmNGqongYIxJqRKMRyIXsCjQQE1bvvnHf5w1VhZB2H/7118xvxgxx1mysLrdETerzFOsVzxGUwn5sqNPfKnkHOPTAYv9RmOl5GENszekx+tItmYJL+3BgPuifSSPsnWgEUhSamm43coaJ12mMI0ZFWeV5k600DN+WKaQUiuqJCUFiBJKJc21pvnF2KMkFdIyo9uz6hox7imkHDGiU+rItlnh5PUZ5nYoCU6P0K3b0VkR8bdLSxDcSjzTrS1NIqJAYm7fht1qGJV4tPqIGDP+adjH8CYk6cbSNKpyLkxwGsYMkbRmw4QRbwNeeq3DfW7VMvfu6eDg4DAL3NPTwcHBYRZMq0hsLSZWanjN7rHAxpjZ9cuLytyLZEnbW3iRPniIpcxCqsy92QTruUJmUSUBCXx7ORu76JLXFLgiP7Fey0Ou4N969XX8kERS1kAHR0+l/7iDNXc/AQNtFEA/KgU9l3QCGrjMyh+9DphOb6QEX+w2fE851FkctDAyf/x1eE9sb8IYRcwXjDEBVyeLRVa/qCHxplTVArl+HtJEl+kiHWZ6NOiFsb2tfivf+CaY+9Yuijbv7YM7r61uSLcBjTWvXb2SNe7cwd5ri6vS7epNNLa/+VbWOGlhAAvWynLM+IGUxHDEZdmSRUtP9uBo2czh4hZDK/nqDLqkpdEEZ9pukYXZIykbkXQR0jDPYu6S6BKRgAt9s98oRrKaT50qYbdiSdd2O10sB9frmO1z882scfhAZ2B3jBkYcWTG1H8C6hL5kkXJK5jGHoUdMdVdKGlGWSjlozkMoX/eW1Gba+7FNvZVtHSzHO/llFEfE6YkSRyIscbB8yRzia46jEIZdlWb2n6Gy12j0YnPiIKWFf4hmUgF2oUsMBVNPjGWbW7rCOcy4m0rdq7GmE63wJPCxOuz5MYg0vgBFvUwEU8q5UnlLWOUs3Dvng4ODg6zwD09HRwcHGbBFOb+8H2t6nflKmjjzSLY6/Y+2Ic3UYrhzWM7i/Ngu90qKHx7T9lu+4TF/7gse/M6nCU9K3i138dPelxULVWxQHznxRekW2UJa7X9BKzqAsOt9zcfZ43OiXqUrCzj2No0X6iQ6kwm+rYvBQP6ExxGEuBME8sTMB6BMpTPrYLwwT2sJ27u8qRGONpooIUPm2UQhGIJlCRHLnzx6ovSTdb3h2N6tXAVu1AHlfvR2+9J/93jVtbwyFwiVk8cWJHMT7cQLd+oIgw+YNCyvbK8tAjd4OXXXsdhkE37VsrAJGJtDE/CEkgtE7sbGgv0Kwm9KbNRkESMoybREh/J1Cr8KaukY4obQjy9aQUSJew8pHoTWGaXMde4PbqxBFyWzectxSYCcx9wxp6krazRt2xWUhYLiZmzYFhYRbi2nVBQ5AmOGVRwQKcMz1r+vshQk+biHE+Z8oIqUoo8NQdJH/At+48eq3uOGI1gdNbp/JdrGjAQJRdgp31261tr7jv7GKJ+nS4hJMXimWKsSqiPn+HOTRj04Vl3X63G8h7sHzItJbWWyUVRKVWokPA6JlYWz5jyl0gfUYwHVLVmT63n4d49HRwcHGaBe3o6ODg4zAL39HRwcHCYBVOUpuOdlrQ//caNrOHfgvo2OUJ+TmdXlZcx9Y0ywx1euQ7BNLyiOs6wAzF0OECMws4OImlCy7mWvsOmxLCMgMpjt61a4T49RBLmdXQ2IAA9uvs4azTpy2uMGQ8g6Ax6OAyPPr5RpIqYFOGROBhPJJVY80YqRZzCyhL+Ah1NS0v46BHih8ICJMVFBoHtPf1QuhWK0CWLFZzykKqTb2whjIoVK8YElPwkcOTdD96V/gmVxEoNe/dpm+JZ+lqBQpi4hOxRCfVGas1QpsXy5VUolZM+ok88K9gooRAssp2I2hJ1ZIzpdLCveYqttgXMWfgMHvIM54eEGFlWJkGBc4Z+EHmJOrIuEKvsaDXaSonKpnUuPrW5MedYxEZsiXqDicTBsEEvmChWqTfikaf8MIxoPMwzSKy8Kam9M+K4ifGwZwVplSlfym0SetP0TiLMyxzw+YkqjxNm1k1GkuBH/dqqkpRy/nii+9POY8SQMtv+w+MNXm5C0K9WMXkmVs3tAWOz2tSL2yEXGCwnmmaNV4E5WvFQ5FfrScIB7DMZbK7SzBqBdWxS9FvS2DwGtNXP9Vtx754ODg4Os8A9PR0cHBxmwTTmfqD0amcbhWguboBv3riI5JO7H2hgU9LFa3OBj+MKKbxJ9Q15fgk8ehyDXkktWUkgMcYszrOCDd0S9/ZB0o9ocGCMqdHIb2W5mTWefvgz9N8CrT7N1fW8jsiSmGIUS1LEUHnQhBEmXWZK5ANEBa0s6chcvwj609jAQb6vSSWKCksGrW3A9aDA9KrDHXWxFD4YkpvXS4jUGA91pxEJSI+ZFR7NKIVr9HoaIlMugSW9/PKrWaN9Kg6emgM2f+d21vBp5djuQCG5rF4i5vIyLv2Tg1bW+OBd2IV0+lYWCiN+6g1xWcUQHTKCyhjT7uAUPv+ZN7NGyaoUdBYx2a40pLSUbdzpCxtNxVSUCSRWrdqUA+6RDgYcQN9K1PHJRqWkVZBnLJeV9lbgVJQ4HhEr8lVLVRDXG9uwxBhjjEz/dKzMPWY+jNi/TniQfYtEH5POh8y1q3FWNPUiW/viuPUo1JStBKeAMsiEIV+BVOS21B6tMMxKQQHtQlKGiEWJDniOwkixXuApc87kdCQlH2xMwarHBKdmU/PZFliEXGxG2xMS/IqeS0GKHjO9cMxcozjSnU4Y/hXw7ivTXsQq/DwF7t3TwcHBYRa4p6eDg4PDLJjC3H92T8vGSj7F3/jrn8wal28gO+jZrq7GjiUBI6SdB0lKar29j8jvJJM/x5W4el1Zm0cTz06b2T4kNqnl9CAZAvuHTMCgT2WY4/v8YE/6+zmw0FwkKRb4KrJW/YZjLKwz68csrqAxV1F+mi9gF/U5bmUac29W8W2niRMskTKL66UxpkofSbEzWLsEF4/WicXcuSo9YI0Ej4MQMokoTPUv4kIdesuvffJTWePRQ3iUHO5Ypop5sfPAZltixBmo9FEgJwr7uGqH9HfY2VYZJ0olGweHlCMZ7I11AD3OrUnEurvmPEeGIBBmTTYtDg6W72pMQu35LFxMNh3b3hmJ8E1sTXxqbOZu7ZRFMrgrqe1hjMkzT0nYvDB336LYCaWDmPlsUhxYPklja52aOws5VQI2Qsvfc8SfSMmNmOEFa/MacyIolRm5wR+OrRX8PMUB8dOUhmfX0OYg5fLUNNSshLXKPb3cPou4MC7AjJjUZK+S9/uS9Ydjq9Ww/ZwVC8GiHmauht+KVUou0AGXujLyEIoozPnWnKlRMirIvZnHDrodfcqdhXv3dHBwcJgF7unp4ODgMAumMPf7my1pt4/QfuPjCJuvMpz72aE6a1TrIAhl8seEb8g2KT5tiY0+XunXVrgsaz3HI0bMpqyrJ7HNdkD6WJYayZsK3Hsuh53mPKUkPg0CigVQ+CCQMGAl+M0l/LY+h0ZzAd1qJR2uIqlcWJwyhoJ9um0e7EIPEfIVsXCFMWZuCXqIrPrV50C6n26pkBKRsAttDLiyPGF4dhhYjJLWmaM+OF2tCpLy2KpQ+OQpqoYsLsHW89atO1mja9VfufsY9FzqOLzw+utZ49od5WgRxQSPoeMeuXZkOaoEeRyJRKdLnPZU5PPCjoUhcvHdusq5NMdu+ESZu7XmLsw99J4n7IG1spyj+CBMXDdrrZsLV5Sledv1RiAFZiZUMOTY5KQS635J5Vs25CDzdhFTDmq/LzPqvDqvecbDhGJaasUPyCiFjJ2Qyh+eNc5SQyPk5QspuMXMU/BD3azPe3fE6ZFSYhr29ZTjCcat02WtGimQ09IZKxkT9QL619itb5uQ0rCmytCXmOJDPtVuRYYcjKkQDujnoq6v0+DePR0cHBxmgXt6Ojg4OMwC9/R0cHBwmAVTNLvFhpWgQF+MTq+VNR7chX/E17+uqTK3X4EnSPEaHIgLDFvpnGjRkgkliTkGUuRL1LxGtkCJRrlOqYtpP6mWCtWQlJDxCBGFtkqeLr+eBkINehBq1fq0ir3PLakEWa0zXYoJJMUK1JOCbYQ8hM7S7Z6ne+YoBuWo1BzTGCWw3F7v33uSNRbWMZJNFiJ+/PCedIuZdxQWno+xqFYlIkoPMs9vj04Q1HXArK2wqOreyhpishoUW9cu4pOFjRXpJtE6ASNjLl1GAlViBZ1I4WJDl+tRyvyZvO60zOH1xjR4LpynMYmQ6DOkJqXwnVjBQ/KtJTxKgSNLK0yfb6h+Z3mOSFsziyyjjLMHKQlLsvvEjtij1XSURDwyiYiS+smWwQflTnG7EMPpvOXrIccmuqoEQk1FgcJukWFPieVa7UsSEc9UTiG1ZqxH1XJMpVJqjonwHYbWNWB4WSxW2bRwHvdUgizR5KVUY5AWI9sk0c4+trz4TIuPtVW4OGFCUZUuPIYuOadDVVFT5kNGokdL9arieflv7t3TwcHBYRa4p6eDg4PDLJjCOi8uaDZ+nTYTKw1YZ27eRXRL60gjlp7cQ1xOPaGPHn3xBtb7do6FheM+68CQC8SBJkVMhMUIT2GUTxzpS74UMJH3beFSHmMmglA9F0p50PO5BcMGy/k2pZepVsBr4rEwEXCNbkfpUvcAFKCgjqNTcOXq1ayxdgEU+OQIBVsOD7Xm0g6rRY3JDB89xDgfHexIt2hIT0ky5RyZRbWGASxZjg81/nXc2YZcsPkUuUbzDS3YcuUq6kEJq+q24caS95ekW0C+1GHFp14Lc2BztyXd9nfw23Eboo34Qbz42nXpVq/WuFOc1LB7Ht8c0AJDYuDEOyO1ObTEDJ3JHbJ9PYRS5iSEhRFRdq6RFVFEPqupLJaIJJlFvkQs4avIKqYkZYeDM0V4rEwn7R+eUQlkR7m8rUIwIIzVruw0trMokURLtFZiqRDy2xET2+Tc7c1KkXDJsxLGzJpYJo20vyQ4iZVurowf5q3a5uLeMk4ZnMcIqlxBJ3ZOko3EB0RyxixJyqe8ENIMJpIBzNnSB7qVeUilMpOUpkWe6fbP+c7BwcHB4RfBPT0dHBwcZsGUN/zbt5Vebaxj0fbKjZeyRodrzb/d0p/0uJzdXAR/LJaYdFRWu8kRKUaJuUAN1t3tJPpaHrOuQIkue7Uifhhbi4lj8hRJFfBlTZzZEUFoLedVuNNFcIHmPMhgxVqnNgnGpHXAbAfW3BgnC9JrQqIX5vTDs6hUMIAVronPzzWzhpB6Y8yYaTMnbdDY7T1y51e1CHPraO+5b087YMft01bWsFlspY591Q6gDEQxBm1384F0E3ZSYkrGCeWCQ6YhGWMqXKIVcaDeQGzD2997W7r94Psw/UxpQrq6giH6tU/oKYf8NmVCSHJuhsxEc1Sep7E/l9gjaWnJmZSkxM4O4k/E/ZObncrchbD76fNmJfZelenLJ5ZcIN6j+aIkRD1/LqHN3IPndQCBTTxFcwhSlrM+t0p2Ts6ULiF5K71K5LKJZIhFz2c6ZR3RTa4L+4vOFtqpZSJNhOLNw3VtknpjTMQsxCHDSyZk7oG1NamYIu4qvuzU12famMVFIklj497DwHr0ec/HaQwG2KkYrxhjrpjn4d49HRwcHGaBe3o6ODg4zALvS1/60p/1MTg4ODj8+YN793RwcHCYBe7p6eDg4DAL3NPTwcHBYRa4p6eDg4PDLHBPTwcHB4dZ4J6eDg4ODrNgSq7R//PVH0q71WpljQILeM7lkSFwaUGd75bmkYC00KRxJ0P5xYnSGGOY0nBM088xK5rMNbXyrc/EAPEpGA7RKFoWGDFTU/oDeI40mvQ3oRPleKSZTgEL3ko5o1oVR1upqGVGjg6AA6Y9pFI4xteUjDG/jZhAdWwV7xWsfuw3s8azD9/OGoeP4NcZxzr4y5duZo1L129ljbnVDZ6ydrv33vezxtMH72WNSQe5EAG3VpvTwwjpIfKpz34ma1y/hR0NT0+k2/vv/ixrJCzpPGaVpA/e+0C6tVtIcBoxy2syxsgcH6kXTK+PixXFaCwuwTZ0bl6tY+MUV02MX4YDTK2PvfqKOYP/8r/5Eg/yvJSkP3GcqTlsjBn0cPrHxxiiuflm1kjGah1bLGOaBVKjSX06aHLxKz3Yf/D3/v7ZD//H//5/yBqlEqaHfS7hGXeVSCxKrW6tUzj/FmnHW5FC4iOMhl/Wu7XEQkySfSeJai1a2Rpjxj1moPGTiTwjrIQryTsSf8867WLXlhrSbXsPNcF6LJleY/ZdPNEcsF4PTj8XLrDMmqZL6d338se/YH4e7t3TwcHBYRa4p6eDg4PDLJjC3N97/660Tw/JRJjI7y3gbXwxVkdOrwQXyF7SyhpdWtunntoZDFiZoz/A+/mEDg5HaptoiixkGkXifghCU7A8/vpDVtpIQCS94XzWEIY9GVn+niHOoUs6f0KLgVJZGaUn7qKsbyzFV+1ipzHNEfwQh1R84U1zBu0TEOSFJthrugSn1DRUH9W1S1ewWdZs8OmUmlj1WocssJFyANcXUUP40sVrWWPjxkXpv34BlTOWl3GBQuoScVMVlYsbOCQpbSIeDa0TdUk4PMS5hCyWIPYKcwt6XYoV/PaU/p4FFm1OUj2XXAjG1z4FbxqPbIeTX4if96r4dwLjPk7h+NnjrLH5AYjtaVsH8LNf+HzWqKkARR8Q8tI/hXPLUUATt9wktvxTqCqM6KMqJaNt5t6sQYWoU/Uad3A/JgNstpzTOdagAY1YzFRpUXo00Js0oa1nsYjDWFzCTW0T/GKJPH0N8z8g119entMzZbfHm7tZI5/DKZSbKtZRSzALFBPkcvT66mJ8Fv/OTUQHBweHPxdwT08HBweHWTCFuZfsz8gwLpOaXV7By+3ykr4hy2KiLN4NZbl8oqveKb/Nc7FPPP7SRN/e61zBj7guls+xpIcW9TM+KcZ4zEVeuv6XxHyzotaBRSnO54FM+Vyaj6z1PJEQqhUcZJcrqpFVF0RsGLttsDbdkw0SfFmjH/QxIJdvrUmvbg+HNKbr5fwieH2Y079wN2/eyBpvfvqNrHFhBdy80YCH5iTUMSrTkTOQlWLSsUFPGeWIB1nmdWk2wfSvX7st3T78gJagngRFYGQaXMo0xuRYZ+G0jRXP1OCUk0RlnBNqAgOu0ae/FHH/uQIbf/pQG1BPD2N3EyVR3/nOd7LGhAVpclVdAh5wqtRkRZ4TTxbf/xTOLc8Fa7HynFucl297A3DVXMw19wgzyrNGfnUVlHmVzPrxAxTZXQzxiFhZX5b+fsSlfD4E6qTw8w3VANMA+2o0MP/LVAYCX2WfJTrGFkn/OxzbKNWbtMFInnXRAPlwC3N6LoWAtUW5NF+vYe/p5LwAD/fu6eDg4DAL3NPTwcHBYRZMYe5FT6lfjTXpb15oZo2FEhbgchbX7h2zAF6Cx/Ggj434Wi/P1JtY3JJo4dNThExbQalmvgb+2GmDRIy5vD60Vr1Tsh6JdZ8wMtmPeZDWGn3MIPyQ5HxE4/68VefA56r3qNviLykgWKHMMYv/nTK+d9FMQTTEIXksb5DPg+JLPIMxZmEVBPzSS1g6X7q4ilPIWSNI6WDCahZ3d7AK3/8IFTonvkold99BUP0n74CA/8anQPlt/tsm63n6RJYmMW65vAYGLC5Bani6iXIdeUbjdwe6NNlmPc6Q65v1OrrZBVYZ76CFHwoF60x/Mbxzyxz+SSNljsZkpOeyvYlKpXUWmSgxbeTgpCvdjnZQ72T54gV85EtxTcDz/8TPrlHHsRXInSUkwxizf4QZVeS902Zuy/KiFqEpsKpliUrf+kVUjbXuR+XaeYOLW+C932eR1Ivruvc0x9uE0fVjVsxdXLDSQJi8MxpB/6nJHBtpekLn9JTd8DhaYOmgUkWfOCGfeOEYxzbssYDrSB84Z+HePR0cHBxmgXt6Ojg4OMwC9/R0cHBwmAVTdM+5gsp7JUpRDYbvLNbxkzhReVRW9TUtgQkho0S1D/k2ZKhQTJEiDfQ5vr8PtSKZYBedPjSmfqyiXrVESY6iRsADkWiSwNI9BxQoyzkGA1H7s+XU4STiSeHbVhc/bPW1W5cpQMMJjlzrOFsYMVehKlV85yGQvvHay9Lt4jX8usOIonsfbWWNdl/1tW4LI3Pcgji1s4tP6oxYMpbu+Yf/+J9njdy/h4P83Jufxic5PZfVVWisJsVmW1TrfvKT96VbQDG0wniOiIrwuHuq3XgllxjTJorz0bFKvb6BOiZGDM3mFJuVf0dwNlDp8PhYvn3y+FnWGB0jHavGmsP9bke63f0p3FhWr1zJGo1VRq1x+3Y41p+QwrvA+CTxWxkPVStcWYUKWS5Cwy3wiq4tqbY/mWBaHh1CcK9RTpUYu2Ss4T65kMlUPs5w0OfIWGfpF/GIGI0HbGA+27J4t435WaliFsUMZjw6bkm3Qg4LLTKQEt3YsUoN+zyCcTtmN9yGVcs/6Czcu6eDg4PDLHBPTwcHB4dZMIW5LzY1a6aWw4t0schUAaatiDmgMWbCbATNnWC2/zhSKhLzfThhPkBKJp6G1mv5GGw3ZuBRny4GkWVn0Olha1vHzI5gHEO9i8OY7Cq9Gpyi26VFRgUtI07Iq7Wl25i+Ht0u+p92wGsOT5XgPNnET2KmL3zOTEGhgG8nAXjNoAQ28bitW/vpt36cNU6OQCi2tvdxUpZ/ipzgSO08MIBrS9jR/u6m9K8z7KPbAku69whZMWtrml4iboZrF5Ecss7G5u6WdLv7DtpLa1AJnjzl8FopGcLXYmY9FRmkUghtyxjmddQppIS/VMTSnxGEWeOwt55ty3ePnqL97AGGd7GGq3xhUanfzlNcmnd++JOs8fHfRCZSuU7V4k8+HMunwDWmbhaPda8RmfWIYYIhmXu7pZ6wnsE4pKTMWzt7WaNB29yydbnbI9wvooHkSdInkYp7EzJrj9JfIs+WQOdYgSlGEurVH+BGyBd0wPOM9isXcYJC/09bqjW1eXdUipiKHk9Zr8s0uHdPBwcHh1ngnp4ODg4Os2AKc19f0lffeh4v1ZUy3ng9TcJXSu5xDX3EZBJZxpqv6atvpcLshVNxl8Crcmeob+9PtrAy2x3h3T7Pd/YLZY0HCHPYyOMj1vlIuabPhVFJqzDGvPni69j7DklHn90WlWKM+thIl/S/wEyki6u6NUnP2GtrztVZlMugwPstnODDTfBfu+iFT+4cM/1pICU3fCUsgxFdIztodGj28eQZCn6US3qQt66jDoeJwGve+ua3ssblq5el283bWPFfYDqHOHI26sqm/Qjkqz+SjDKc+6ClGTUxC3IUS9QNuEIq5gvGmAKJmyST9K3ognMhAzKV5f5i6pvaTf6HU1fWZb3prxT4NmGoySSyAzBwys/2wG332IhjXae+sIwt3/0BhJrlVeTn3Pzkx9hLb0mfRV/UkISHllpnKXffLwmP557PY1924lkkdXF4S86V8EDIWXlQoY+LOxzzJmV8i5TDGbc1Ay1fLXKnfJJQFYwjjRIpcaFf8pRqdYgbxaLGz3geTlmWzic0+PCs3LwityZmPSNeqXisVzkXsljI/By7Y+/t3nlz0r17Ojg4OMwC9/R0cHBwmAVTmPt8TdfcwzHIWoHUsswamaOBtVLGkPhms5k1hAuMY31AT2heWa7iVXnnAJ88fKKr3ocdbE1qUlyiNclf+w2ttnhhDRv5P3+Ehc7vPsCqX8TakKGvlKTDkpADRr/XaiTssVUXhHHOsiZYZnGRyLIXvUgXj9qxktazaM5jafvh5sOssfMYR1vOKWFp91pZo9tG7LHHSOZWR8N6WzRWCLmYvrgCYlikQnLhihWEz0iJRz/9QdYIaM05saIXDg6xdP7yKzATuXETvH5jTYln5dMY/Hc+ZHD4EGRqlLPW3A3oudTh2GVIf95KXmjMyZYZPj3QIIRzcZ4HZnqWuUt3i5/K/EwNDlIJuxWjLhUabKKc/XPpyiX5qExFokN3CUPrzHc3D6VbiUVcQkZKvP/WW1lj4QKEoObGFd0T41U8EnU5u8Sa2P554zEFUtokpdNNybLBHZIU5xkoHvc4US1L05UVHHB0xA9JwCs00h1ZU7exClLcP1PrYnFFzUdGXczPgDed1LgtWtV5hwNsuUDPHT+Pp4G49hhjJky3CehJMxSFMLFygkjwQ6oKQzL9w0O9fGfh3j0dHBwcZoF7ejo4ODjMAvf0dHBwcJgFU3TPpXnNQhkcQ0TwWXtW3TEs69OA3/YpNMhT2a5r1JiDPDSmu8RHz2DHe9JWSVHyjgJG/NeL+HYpVJGxSEvmm3VEBe3OQxXaa0GtGPV172/fQ1ken3a8kwoDaBpagMWwInG9AZ2lSk1vNLa8mceIGbrMAC8dDgsPHyI25cOHqPqys4NG3FEBqNbARm7dRD3hl+7cwUkdWAlOB/jJEq0cLl2HQFlbgIy4f6KbTQ+hsW4+QZjUAdNF7ryoB/nbt25ljV6XZZB5NdKxDuD734V4euM2BNCVC4gm+d7335Juu3sYGQn7GLJEbcuyCi6x4E/CgJteX2Wyc3Hen3zvjAio4TiJfpekzwce5Zm+4v3cJkRwlA+w9+acqnWf/dxns8Y7b9/PGk8eIa0ojnRiPwiQP1a4AtE8vgs1vPn172eNT/0VFZqlULbI8iLJ2uJudEYI9s7NWNo+eD7tpzxS2brKqThkDFCVZX8urDWlW6GMXQTMP5pjUGOzzJrDqzpEY6qz93ZhEd1s4u4b9TTtZ8jHS447nbQ5i6zq4gkfOEFO4gsxtSIrxEieM0usPzxfx2bvdx5Lt4U5nBe3auoUgpOJxv+dhXv3dHBwcJgF7unp4ODgMAum+XsuKneYY4aAz9SCVruVNSZWSVufoTziiZnyjbpiVWSdsMDxhx+BsPRYlqRgJRI0mAJRoqnoXIC39x8/OJBuEZMcRg0kbCzN4Wg9Bs1I/R9jTJ+OgT2mGI2l1KolLwjpkcyKlBEeuVCjHCImVKTxeQEj3/vG17JGuIJiwtfImUuW++ELLyLb5/YtGJfEQ4aV+Mrce0ZKBuFMg6DBM8UA9jpq5dAg+RIjzs39VtYoVtXkos56wteuQzdI+Wd10FIidPd77+LbAY78pd/9YtZ4+dUr0m3wQzD3hw+eZo0yGWijqVWsDW0m2pxR49Evl2uUTqOv+q2EIqXP9YpSFVcePEBppgErMt2+A/miYPnb+mccNhPmsyWa8mTe/OybWePpI9DSf/Tf/a/Y6UDVns0DUNQCSe4Nmr3e+yZ8Q5Y2NAfs9meRgDTgWIUJa3pbB3bSBxMXK0xbLjiLEZWrY7qRlvs6x+Z5L+T4cCjSQ3MojpyWgifjG9DsY9TBFhZrSnvv3YeIVC1ia1XaDI2s6z5HccCLGSZI8a1oPas6tJgpMAZudw8aoEk0+qraQBjfkDmQEUORSkW9yrUK9nXMECtRCWqMrZwK9+7p4ODgMAvc09PBwcFhFkxh7rLobIzxrFK9GQpMxSkbNRMJ+RSWNIYJeU2hpC4hR7tYF+sftrLGtXm8ZltlRE2RhP3WddQt8Pl1FFiOgWR8YQDmUs1zZW3uata4fvOC9H/0FOTo7j3Qq3yI9/M0VRUiinD6Pt0Jc1yNlUoGxnIy9bzz/gIdbIJrv/7a72SNQgELkXNWfeO1dUgNx7TbePYArGqcqKbhMw8kCOmhmVKaiOi5YA1lyoSiSgNxFEc0LZXcDGNMokk4sjyNf6t0PDTGXF7HYArp8Q3G7eWXNfGm2YSYMBx8NWvs7rSyxoXlVekWezjyXA7CS7utxPAcyNHK2rhtcpEyq0QvC0nus61n0u1f/Msvc6eYPJ85hCj0+c+rU2uhwNwe7kJmQGwpNhVS1N//a7+bNR5yMf3f/NHXpFubQQh3t7D4Pudhqhcp1HzvX31V+ocLuEb+Coa018LR5qy6ODttiDBtescMh+dlbS3N42ijIcv5VnWOpQxC8EMcUqkEncGuGtKnIjGOaKZDav3Cbfjn7u7uS//RCD9eXML8j3ilEqM3dYlMedLnVC9xcd/XU+4dYxxOqVqI31C3r0cZMwdSjH7ES/TCJX0yyL180sYBy53enNewgbNw754ODg4Os8A9PR0cHBxmwRTmPrCsNr2JUAB82OuBHYwn+uSNfC749kE822ysX9RdpBE+vLyIV+Vr6/i2P9Q1xAu3XsoaedLSk1OulDU1kt8cgUBepEliqwdaevUFcIf6nDoL1OawqHpygMNonZIH5VWF8FOG6ZIcCV+PJzoyYnVo08azKFexypxjrxYj+QvzGo3Q5zKoUK4iMwsKtnkjlxpTDuqIpQ0LJQoOni7yJj6GqLoAypxPW1kjKOne0zyrIHjYmsf4bD/Qy5fj0mSpyvXQEUbyaEtjIeYrIDt/9S99IWv88KdYfO8NNLZhOIKmMaI5SLOmh3QuyOBIzlsnWn/l9KSFL1nRZPcAx/bdH/5Yuv34vfeyRvuY5rBcjX3xlTvSbYk0Mwgwkp0OhqhllXa4vIFIibUNrKH/R//J38kam1uPpNv3f4oCpSMWlbn/DL425VWcy/G76vra/7/QuP7Z17LGCWPC+9by98iTU8D9kiTnzckqgwpeuL6RNUplvU3kiu9uYhU7isTcZ0m6tbq4aoFHv07y384pju1wX6/LRGclXV9pzZmoZbAZMGOiS9vcehk6w9jKR0k9tANKhXWKJyXbAphBMjU6H0l/W4WTwioe83Tycrn7GrRzFu7d08HBwWEWuKeng4ODwyxwT08HBweHWTBF94w9y7CDUQWi7omTaLVmlVo9gBj06Jkkw7B2iuQAGDPcg+R3Yxnaxxd+Ezk2H21phkz1AsSjxQUWBaJ01WxaAmVCG2Oqe/sHCEUKi1CCDlp70n97B5JKLgeVp1GH9jEYWCEvDNTwKG1KKRs780Qqpp6bamRWL11+rv9wCFVor62Dn6eeO4mk6gvObtDVaKpJio1I8d4oQGOOERtLC6rHpce4LmO6t3iJhKHYUhca4mccM3nMz+nf15SmLd1el1tjXJqv3Tq8WMUyNN/feBNC9r2HWi35vfcRHdJt4wTzOc0SmQYR9aQSEf49bau+9q23vps1nmxDzDpqY0BOeNjGGJ8abnGEGbV/dMQtfEe6Xb6CuBbJadl+hjk8GdtqHXbR7VBJ57V94ZNXpNtPH0DTHHcwabZYC7dMR+GNhgYPPf7hz7JGUGBE4DqG9DRSLxjV+VJMhhFTZaZmY1XzzAMss2BRXqdinflgjBQyJ0cY3vc/eCjdImY9FfIQHOcrkK13tnAbHh0eSf8hc+Hap7wKlK3tskyiJkv2n1RJKluC5jwLcEm84IjpVaml+Q64jJAabCSShCjLcyTmDV4q6+MlQ2hVSToL9+7p4ODgMAvc09PBwcFhFkxh7s2mZqFEIV50u6wFlJIDnpKkGGOePCULY0RFiRV1dh5paMUK85QuXEASUXMdOSphx3p9Z7eN1+CSUKQnYClSLhAbvJb3WMlkrYz4kjFzbLyKnsuFCisRNSEIdI5w2Pt7VmiFhzEZirUlrQkrBaWWY1ZWkUykqcYMKS0Dxeyy38EQFUq6tY44ZQxxLn1W8c1Z7KtWAY9YnAO9qs/jBJea2FocanLXoICdnlzGuY9iShkTtWaImV6SkI7FPGUpG2uMac430S3Gb2NOhkZDzyXPHKDTDlhYOsFYvXZHfVSbNZzLH/4LZNcc7PHivmTO4v0PEGMUhrhAwp1PrOChVhfTcnMHc6axDFVkvqFixfwipsrhQwzIB+8inOgrX/madGvU8ZOAsS+jMS1mrJyuL3+ZeVN8G5HQpbJV7PrV11Eg+u1vwWq2z9yl+5yKpVhn7FwEUvzwuz/F2S2B/x5bdapzYwx+JHOMtZ0/98oL5gw2VnFswlilHJmxvHpzi2DiK0uYbF/9qrq4JqwL1KgxMmwHA7JCs55mQ8+ltY9vD/cx4E3G5FUqyo4b/G21gp3W6PRRqepIRnT9ePQA+WMBMwP7I1VUxryFxyMWOKL65FmmqKUiDiBmMaUJA6wmo/Oytty7p4ODg8MscE9PBwcHh1kwhbl3Wkpjw7HwRz5nyeTCQDldn3RproZ39WYFFGNwosx9aR0cav1VVDJ49xlere8/0CyUN+nx12rh/XnlOqrsekb55mSE1c8mF+3a+1i4L5HTrc2roWQrxvt57lVsf8AV+bf+5R9Jt2f09QjywhTATayVeTPhh76VRTEFLNMaJmgIx73YUE5++xroSbVIqsgB77WVlg7piVCqYKe3buJcLl5Guoufuyj9u6S0F9eQjnXrEQatPq9ru/PkULKUL+uWqV5kdW+JmI0mtXBz1pr7kIvj84uYDF0SyV5LbSPWmcbz1/7Kb2WNP/jDr5tfjO98H7UrBm0sN1foFPn7f/n3pFvEdecfv4MiGXWmMA0TXWZdX0bazGQP1Kzdw0EO7j+Qbk0udldYr6LKQsrFinLnRhPDVGfkQ72Ocy9VdRn381/4eNY4PcR1fO/dx1kjnmAybLb0IHNcuQ93MeCdEzSimhUyUcJttb2J+dzmEJlpzD3l/SLFSITPGmMmTNgTm9OU4lGcaDefXkL6ES05Ll3GZBNDEGPMBUa8FFhMu84hDQK9Efb3obe8+WtQ7VbXITpFqY5MmwESJ4eY4cctbD8M9C5dWsRtJclXCTW9hmXcecIwgJRhNuMB9mWnF56Fe/d0cHBwmAXu6eng4OAwC6Ywd+s92sRcWU6FqDJXP/aU1J2QvLbbrIvAla+1hjKXT/zWr2eNjdufyBr/9//0v2eNVWtxPBiDTG19BIeFlWu3s0Zx4aoeXErHhGOw0VLSzBpjLskddpTpN5cQuD6/ioX+YRc8y9dlahPnsXeJlp8wctezCh54pD8RDT6mhtV+7k34O1x7EeLDNmOJL6yrqnDz5hWc6RIYZcD6E92OMvfRpP/csVU5bhXWUAnyuvydo1ww6IHpvPEyeP2VW+rIOUlwsaQgR0QWllqzISCRnAzJg8hrxAvSGOMV+RN+KAYcgWXPGo9xXouLWFn+7G+8kTVaJ1PiFx49fpw1Tllc5MZVpFqUSlbixjYmw5NHWI2tsj7iyAoz8Nq4yoMWqRmH9Pr1K9Lt2hJmRo3ixsE+SHdzXk957SIOoNPGLvKk9UWL7dbJZH/7L+JGOKGutfcMKQaHVnnLEu02likIhIxnuFBrSrfKCpjy9mO4sYz7mhdwFk83kUdQreCwu1Z51waZtbhyxAxyKFmVNiYDfLu8hCMpsITM9WuIqMkX9J7wmaKSL3BrJezI93WOpQMc+Yg5FJMGNruwpnepH+HDyxchWBWKGKu2VaEzz7yAkIE0UpkjsKrsxHxYBdSCUhqjVCtN84vh3j0dHBwcZoF7ejo4ODjMAvf0dHBwcJgFU3RPz4rLiSkTiMmFCFypVWqV5XbM/ALUjdUyZJE3PnFDut35DOTOk32GL0RQka7SX9YYk3Bzq8uIDpEQmX5LdzqhCjkZ4CxiAxHw4RaUnXfffVv6v/lpiIDzq4jwaHegkeUsc4DFK9hIIj4gUtfXSmNos7rsqENBx0zBx16FJfNLb0D3HLwM6bbS0JJBqnXRiERMaucqmp9DkxD9oyeWGZJnYvvQjpgpce0GfXDpAz3oaapYKjYhlIdSTgKr5JGJeWwS/yH6cpxY7i2hSOQ4zM4Ruj15pMFAn/n1V7JGf0KnDAqmLTMFPVpZ92n9UCgj6Orn0t42of01ObwxU9G8oUbF7eyiIvHONuLzPB/d/s7f+svSLekiBu6r3/o2tv8zyNYLDdVw9+7jyNfXMc6nE9pF5w6l2/wCRO2Xb8OAefzXMeD/8z/8J1lj0NHklh1G4RhGko1Yxbp7qK466zzTPJXExeXzfKb79KiWej7jSCfgHHXMhNr3aIgZdfGi1gL64F0EhOV4uVdXoeoucguBZyVEcbRE9yzz8tkRS2aA2T5kkavjAwxg6mvEUolTRTZSr2FOtvs6MlItvMRAQPE/nlhFyGt0zIl5LnWWjLZS7abAvXs6ODg4zAL39HRwcHCYBVOYe2K9xg9YRzTP4AbxaAh8ZYg3VptZo1jC4/jyZbzkv/rrvy7dVm/D/uGn3/k/ssali/jh6ksvSrf80hXsq4wYhQFLpw7aGoqxt72VNU72wNMThqQUa3ifX7Q8Gp5tv5M1VtYQURGxiEpq1dvxeq2sEac0BySNLRV0uPKraLcLFu84gxLHrUInArUptGImJLfHE+YuNNmyP0wmCT+kiwflhYiuB75Fl1ImLFVp2hgx18LOGzE0B0lpdaJBJLGenUSupOKwQHsRzyqQW6B/RC5mos4Qn6R7GjN0+BG47YXbEG2OfHUyPQtx5eiPEF7z8BEC2v7pP/3n0u3bX4eThceQrz3Gvhw+2ZJurOhsIh55fhWT7a1vqL/niM6h7zMBqb+H/q0DPeXmAubbAZOC2qc4yLmmJgWNY/hjpl+DcWepjusytwiR6nCimX59eltsk86nnGzlUx2rgNy2QdfLIJhyXwtEFBpRyiiEKjuJs0aeRj8+Z1081svXOaGQ0gXFvnoJklSJB1kta4ST2H9ITeA4HvNodSouMHztgK4iuwdg4j9+9z3pdoNK1P4B9r69g0GIjBL8Zh1by0mBdPq0RtbdN6I1D+8DU57HSHa6581J9+7p4ODgMAvc09PBwcFhFkx5w89Z9h8tJiHErBhcLEttTysbfwH89NkOVqKvf+yLWWPjlS9a225m/0y42XoNb8hLt16WTr0QdOa9n/wka4wH6N+2LDMOt2jtRwpQLOJ01q+Cm79yS3OTooB1CAKsSObyIJ7hUBc6+0+wqJpwTT/in5iuNTJlnvIKrU8OppXoqDVwLinTbAZcuE9HKheM2O51caaSnzOyFvolr0n9B9kQS8eBVX8i4op8lUxE3BKbNXVwKObpb8jcJMOKr75VBlbKuh7t89iYipYwy8tY1SCSeMQfYvuXL2n8wKCPM025ttuoSb6ZrqELGjwFqYTdJmf84O13pdv+oyc8cua00Pkx5ys/TclPfR7vBeo5c1YaT6vPoIUrsOZ8EreyxumxUuy4gBm1x/X9QZ8E/1iNUTzOn5HX4vYhPvh5EPwksA4yj42IDWjMCVDJqyBQbeCAhQIn6VSzWWB1EblJBdp/lK2koGIZH0a8rXLUlepFnQzXLiB+oMkHwhoX+qu0VqlX1Ilm6DM0hVJJ+xRbK1as7LgyLtbeAabWJqvL3HugVa/3yOvbVDAm9JC9c2dFulXpFBxLYWHKSnYh8SILk8jwehQ3ovi8kXTvng4ODg6zwD09HRwcHGbBFOY+GlhRqVxl9rgAl/PpHxHra3ypim//yt/9i1njzd/7XNaoLS5Jt/2P7mWNgBuROOeDxxpHvd3B2/LX/+m/zBpVhgEPR7oEtrpCL0UyvkfPsKg65vbnWfnDGHPrldfRotHncQuL9f2hriy3aH/gpSzRMWCIsvW2n7JUyQtNfjQtXP4P/tlXsM8cDrJ1AgLSPdUqI6KCCIXf20O32KoROE8PkblFCAIFUozeMTSNe/fvSX9ZMdy4CnMQcfqo1+al25WrWMHcuAhmfYVGD/NWRIHwoETi/ElFJ9ZkCJhQEfC3K1egEhTrSuUmZJdCVefmJX1gCnOvkLkHLOY6OQL3P7yni+kbVVZbJE/vDMDyRr5l8lLCkeRpdnOwx7Xd76kOsEJfjKMTDG+bCQJdq5TM4FBMbHHKIc+qlNPLN6Tt7AF9V2PmKZRDhnNbTql+UZQi7izFFno9Xf5u05pkboFB8sl5cSDiYlksSSCN7jTHqzbsYIZPuObeqGl+x2uvLzx3gjkGxItLrD11DT1ECqTJVVbayFtzLCWzDjkOH3yIJ0Ovbxnpxrj0oxEmXp7KmO/rHEslcIWXXi5fp69PuZC/HY9pn8oAj7FVevMs3Lung4ODwyxwT08HBweHWeCeng4ODg6zYFquUTq2/gO9wONafpRSFrTcRIoFCCivffz1rJGnCPLB2+9It5NtxGeIe0XnBGLT5oO70q2XQgPKMeSlysSAelHdKJbmoHDt7CEoRCuydqD3PaNFrjHGmA+zf6RscjHEKUQFFWePIog7JVYMLjNSp2SlZHT6kLrESHgqvvJVVONpbsAtJY1xbG+/9S3pdokmKYsLEDS3n/GkrDSe8jyErTEL0u5R6v3Cpz6ZNV57VUvZ9Kna+HQ7ePx0M2vcu/9Yur37LlJfGiwh+zf/NpwyPvPSdemWTyW4B4lkY+qenp24xHkxkcylEI1CUzWpEoWtJGBkjDkPSZ77YPpTjgE6uYkO0SVm70SUFDuUuvy6REQZP48jGe5BYx21KGgeacjXITOyTkf49vLHkC/nH2jEUouJN1VWMRoyHmuS03CcIXOHBlQSJaeryONJPVX3YuZ0iZWvH9G9xZIUD2hYI+bdYZ5KonrvKMYcrg5LGDVqelud0JpEkoJKJYi/ga+X6PQI839M3bPdxU09iZs4l5HeGtTbTY7Xpc+7267lLQWFylxx2d3dzRqjVEdyFGCU8rwlA2rE/b5uLqLQXGAFp1PGJu4dtaRbKsXaOMM9Dxux0wvPwr17Ojg4OMwC9/R0cHBwmAVT30stWwraQDBfw8SkB2MrC2WF2Q7/+p99OWvMr4AmL62pJ+CYBXVzOfCUaoUFW3wrjYdv+avLYGGDTitrlAKlfkekThP6b9aKeLcfM1Ln/k/elv47H8KjYRQx2oN8Nrb2Xtkgi2HVX7/AYAuLpDcN5IU7L6Fc0k8emrP42//+38Vvl0GB+x2UjX3wjkbGrK1ilJTKFTEyk0RjU26+fC1rzK1BaugvNrPG7//e57OG6AzGmN7oefuDiJ4jw0hDMfZZyfnpI7Ckchl7332m/PTJe/DE9Omq8NEuwqo+9TuvS7dLVxDtJGFMPkOdTM4yMpXBJEvKW/4mZ3HaYsWbPph+ZYyrtri6Kt2On8At4sFjaBSHE1y+Oas8tc+p0ktaWUNqAkd9Va6Ea0fUqQ52EWfW6+p1SSfiI4PJIwzUs9J4Io6beO6kNG0ZMlLNNm8Zs5x1IYeN5Ok1YxtwFNme8DB8/7y3okNGX63z/hIKb4yJWLd5jiJSl/WNo0hPecRkLZEQPnzALC9ex5xl/3HpCq6RX6Vm0sPYxmMd8Ig1zQr87SlVkXtbqsJdXUJC0RyTFUOGu/V7Kn20aB8c5kXGwdmdWHGZCX1zPVL4HOdkzwpsaprn4d49HRwcHGaBe3o6ODg4zIJpa+5WokKeSQhFMUQktUwDXadLuLZ1eAha2j3AknFp0tFufFjPzSHRpbmOjAWxJDDGbG/jt+Ij6fs4znFk5bR44IMVLsSLMWkgLSswIB7jNd7nCbb7oDDjglKS6jqOpF/Ctx16Zwx7lhFh/Qoay0zamcbcC1wpvvfhB9jaKYbI9imYkLx06RIiRp+Fgi50Rqw0e0pLkv2nWHP/V19GUtNJR5eMT7s45RpL2jbmQMcqdWWUz56BsC8vgnQX67Cb/NYffkW6ndyHwWLMlIwHu7hSW5Y1yY07kBca9TJ3CnpVKqvwUq/QvIOrpeXytGwtAeuviH9jxAnQs8on7DB3aJdzoDvmOB9pClOQwxXv00gl5awYWHMsZUJUntx5RL1InFKNMR5TjA5PWvyIt4nVLccKEHWxZaEOJpMhsNJ+Srw9fYku4GF4eZ0VKU/BY4kL3ztvpfjZ9i63hrGKLOPOixdBivskrW1qFFGkMzaQpXPKCx8+eJw1RIXb2dyT/gsMF2k00HhwHxE4ahdrzF/+fYSOFFLM2GaTMTBtvS5HTNZKWKpEzqXdVf+UHlMT+xQENNZiYl0+ho5InZsT3jiLlg52Fu7d08HBwWEWuKeng4ODwyyY8obve0qvigW8BqdcYa+QfVQsg8g+lzUX6OQYsv/4VP0NExo3DBhhu7ICFw/h/saYW68ixvc7X/0GNpKCO+Q8VRUGJBR1mhfkGVQcsFt3qEtmj3daWaPVAl0aeXixX7qlf0XWWUphkuJoTw6xo7xlJlK5wHgAKzr3LDpHOP2v/cG/zhqbu7Am8Tloxph3fkZ9g0ceSeiztRL9lX/xTRwJGdxrb7yWNcZ5LLy2R8rCHj3FAvTREaxDxkNsbWd3U7s9RqrCJ954PWv85//Zf5o1fvDdH0i36BRL8x0u5Q/IuR79UH06vvUj8LVKiGua44pnYC1AV8ncNy5jDvzVv/U3zS9GSJ4+Icntsp7KcVvVoZMxji1i5EbKuJGhtczq0TV1kkrgOuuIWLVOA3K6QKqSSE1ZS3jRbqTYEjthr34n8qFuVgqlkML7Osekm2xN9Bw7PyHlb0VyiKLzMjgiHvkRy5TWLUWlTe1ITjnhSnRvoFPLZ2GelDEhtRL9VujI+fY7OscqJUxFqdBpOHmkBIgx5sP7+MlKGY+XWoUROCyFa4w5eoLbymMVzAOWJ9nY0MiKmGrMiJpDvydJLjrOCU+hykoeY0YS9MbnxYG4d08HBweHWeCeng4ODg6zwD09HRwcHGbBFN0zb8VMiM1EUITDQsJsn/5ERRC6i5pCHgv8uRyiVfKsKmyMqTOEZe8AOSr9C5A4ly9qAaKtfaRzvPjJX8savQPUGvro3ofSrddF1EJIh+AGFSuPksru1q70f/qEEUtMCKmvoLGovrzGo4+Ad4xv504g6EhuhjHmQhPZQQ/fF2FXdRnB2griP25cQUpSylSuwFdJRYRaiU1JE1GFNDLM0HJifR2hRZ//3S9kjVoZcm2j2JTuH7yLdKb7jCZZvQCb5GGquk9Al9x3793PGu/fgyVt+cot6bazjS03m2gsMW6mXNUwkWMqqsdbyE06OMTlHsZWkBaFp90WLt+bXzzP07dL55c2U18k26fXU0FTyirVGelSKE0JhBITYnF+yTGKKLCqV4UUT8WnI2ZQixVmo/8RLVRLOlvnlMQiUNJinD/QOr3WdmWnUgZc+heKqlQWROGlACp1d6eiuYCJWqdtSjGnp3zMot8lrnBMqP2NI5X4wxwGME8texxD0Nw/hrY4ivRJMl9DoNLGNQiaE3r6tDtarOzJM9z7+SW6HdOWqGIFtHnLzaxRK+Eq91rY6eMnT6Xb9Vuw/R5zto/FmsTSMzuMt7vI50CJ2XGjgWXJfAbu3dPBwcFhFrinp4ODg8MsmMLcV5b0kTo5QpDKgCkTPRYWSq0qMaH4b9ZBCsTfc9DTaJKimPyN0fjRW7C/vHpbA5u2aG0p9EfM/gLLJaREvtmjseCAFWwi5j9ULNb25hvw6SgywikK6FMwUZeEAaMs/A72tUQXhtdvqXXmchM+HT/egTmCWTRncXyAAfy1X/sUDuPzn8kahYLFEDXSRYrKkuAb7SZ+KAMmhxw9w96PhyA4x4cn0v8RCfv2PqKIqss01ChoEoXH8rZiS/Fvvv69rHHp+ovSbWOemUhM/SpLBs5Qc40etaGuVDjOMTN2dk+02+IiApX6zPr46td/lDVeuKFGnILDo+dNYYaMfRlbNhO5oqQwkZIPns/YMcZ44gsjDZK7yKrR5DMgRrKktO6QFbGkdF62TynGM1PkiD7dPxPeViIR2BFLsi/ZmhUmZW2WnxWLuI6FwnlZW10Wr5Za0GsrOnfzJOx9BnVVaBnjhXrLewHLGeWZ6USe3mcdsFxJ79bqAq7phDXHopAViZsq+yS0I+owcOrGNWhN0a7WNNvtsSJxF7P9xg3c3VubmvM3kQrDfND12i3syBrAClWvKush92ibElhuLGfh3j0dHBwcZoF7ejo4ODjMginM/eJFNSCoM+/o4Sbe9lko14xjK2+kymwEOnjGCV6zA+sBfUyHhU6X5GuC/kGqDg61Kpbn9nZbWeMZX9QTa6V4ZQkqgZeAYpzQO6BQwbE1G/rinSdxG5H6GS5l9kZ6kOMuqR+tFm+wTu/6qq65bz474EkxUWEac6+Q8R21cQpv/4zOp1w3NMasLMtCJM+FJozGSpcKeaYXroKAb8zhBLfvIbqg31Uau7SCI/84a9UGtA3tDzTTaW0NS5O728iDOqShxuq60iVPknyk4gIXrCdW+ZA8FZUC+eb4iLWXrdIOyxfA3Ce0tkx/bhX7eUwmPC+m+8hKtL3CXGA9FWFm4pgRWMxdXCljzqg4llgIlUoCZkn5ssR8Zvnb/m165hysgVFZptls8KQwkmKXGVu+NmcJeyxL81Y20UilBnaLrb2eQblS4kYwnUYTXVkOuf5O7c0OQrCqJeek//OqxYiCgBfqSJYbLBDdkTV9XKmDA9WawhDzuVnCvsqMnagWdcYuLyGS5yhtoVsZ12VJXHuM6TAJTe54ydUSsxJjTK2OAWmf4qY7OsSTKvWniEgC9+7p4ODgMAvc09PBwcFhFkxh7vU5pVcD0tK5Zb6E87X/cM8qYEDeEebxmi2roIlV73DCUNX2AG/IZa6JD/v6Wj4YsvjHRNbEhRnp477bpktIvcgG9j6gncHhkQbiSr1DXcok48mHugAta9F5srbLN8AxB7oyb775DRDwn90Dhf/MHXMWBfKa0RBH8p23YH2STpSS18vYq1C5IZl1aP2Fu3QFIfovf/p21rh+CRT+dBOke/fkSPrnueh5fQEU/uAATPzl23q4L76Cep//5H/7x9wpLTmsQPQxTRJTsU9lfYvAIs9XriIvYH8TIfeyrl2qqNpz5w4WSYd9HNIGy40YozKOYIEx3j6PLY4l1FzJozDfoWQ90PXSs5w1xMlxLIQ9mSIcCGlNGDZgLeNOWUwXYig1LyMrwjzhAUsYvITNS7S8XUJVXEJksxqNb4X0+2cIe3ImBsCGpA/4Hl17rKCFQsKLRWNZj44/eSuo3nBU6w3IWcM2rto4ZD2Mgh7GgJMnYGqNTP/xQEd+ZwjKPH8BM2qyg/urZBfxreFSLjWglx0eoW7HvGXyIvpCl3Vobq/hfkmsJ0m/z1q8PTTmyOsnKpBMgXv3dHBwcJgF7unp4ODgMAvc09PBwcFhFkzRPcOiflisQziYqzJMhJa0uZKKGu0T/iSm+UIRGlZsBTTEI2h/uTKjIkKxZlAvjBHTbMaT52NZLOnDpFRSYkqmOUaTGFYvaUmdGWMGNGBuMAYipADqh6rH9Q3Eo/1DhFacdCF+dHuqx/3x12CosUcx9DNmCvpiKMt9/c7v/W7WSMYaDBRQ4ZXkk1QNdFWGLlB03m3hnDstZFYcD7AFr6ga7r23H2eNo+/AO/ba1ZtZ4xPMzTDGTGgbXOK4pdR77MAmn24sCQXjAfW10MrPubwB3XPIPJA7dIf5wY/ekW7bT6CKDpjXkfZbWeP67RVzBiJqJ7GogBKCpgE3baqoEnkTSN1pO46HzRyvS8RzSaxuIneKG7EnMXPTdNKEM1Wvo7HDpDix6T0RMVQokYQhK9dIdiA6ppQAKlkuIRJE5VMflViuqZDQvTJzbOyRCTg0QSCxXDzaSOXRlBvpdNAYMDxItlC0niSyhjEZcGqdsj5zqLlGNZY/MjRtmXBFJMjrgOepyabM0ZKoo4IVJtWcRyBgyhQjjxmSw47efYM+hrfIAfHOKs3T4N49HRwcHGaBe3o6ODg4zIIpb/jdrvVhgND/agXvz7kSXmXLls1Eo8Hqr8yo6bbh9NG1yv5MWFSnlkf0SZEJDdHIyqihwSir+ZpcQUI39HFfrpKw8HgjEpA8S6zUmyoInByDiXdIoOrzOIx+pNTvwWPETNx9B7V6lun6t7KhFMPQnXPRjpA4g0oVJ9ggA6gtgTKPrFMu8s9YnkEkKTMxCpatYTIE3eh0yJLo4LB8HZRnUNaIpfuPUPTVsE5vjlvb2nkm3RYWm1ljfpGBGgzOGo1UrOgzemlEdjxhDaWwqCOzwirTT3cwB/aePs4aw65u7aP3wOLnFxB0ks5pKtdZeBwij/LNmDEvw5HKC5KsJZ4gos/YxYGltPWIEUWe1A6yShEJF1b3FlbIsRmdkG3ZQcof2gYiKUtUyYwNA5Vl2Mf+j4QiURBIn//KGOOLnsWJHVlhgmdRISkOeeD2O1SRyk+3y3RBikh5y3ykRBEpz2A1JgeZAcslrSxvSP8h6Xyzgv65JU51K7xqwnrTci+XGGgYlq2x4ihFHOfFJSQF5RN9fPlUvcTwNE0hPpQt+49SWdOqcAoUrAZWKaezcO+eDg4ODrPAPT0dHBwcZsEU5r71RNujFt54q0s04ytx5doyvpufxxtvt4cX3dMWGidH+r4tWTABDTiS9Pk0CWPUWUEe7cKqAmsxccD1fVr3mxxNNKI+SHpsvXjH/O0pKzqMY5D0k7ZSvycP8GHrCOx13MPxrDSWpdsLl1FTpH3eq73pd+g2yFPOeaAYe3tKYx+8j3ICRa4/5hvwQViwzETWF/GhsNF5ZnoIQRwOWtJ/eRm8/sI6uu3sgk3fv3dPuo3HSKYa07Cj08Gx9fsH0q19yooLZO4xbUaDgpopvP/uPLc24WEgAGP9VU1wWl7ChwtLGNUiN5IatQEVyLrzSIoJMyRjPLYSongKY4YNyEq3nR0kbLRITudL7Q3LgEPsOWTvYgxqb014fT54/nVkaJm8SGZRwP5yGLIjW88ZcLlZloALktxlLc1HzBTyqWsVi+f5e+a4L5+REvlAbys5LxUreO5i2musHKokwW1S5CE1ahUej+60SA/ZZCyUHJ9EI9XNhrxhRVEp53FsubyGGfRpUVqgh+xwzDiQkWX2mtIXmFfNp0FwbF0oMSRttVo8O45M/ryRdO+eDg4ODrPAPT0dHBwcZsEU5h7nFqQ9yb+eNcYJ2UEE+l1s6Ht5cwmEYo62+/MMQJ071qX500MuafWw3zgiF7CS9hMaMQwZxS3vz4FV77MzBAEZsDJHjgtqNR+yQuJrXZD9Ce0gK/hhkYUlmnnlDlcN1p1ffg2LfbdffTlrXLmhhT8/+Wlwh2fbGnZ7FglD+n3+oQonjAew8gh+/N23ssbuHobXy4FifPJTr0m3z775etZoc1nznR+jmkWPDPHe0y3p/+gxBIGBFGOQKg51tSNts4xih9qKFDCwV4DDQKgZONc6DUGaCxrfvryO9vobqOoxx9qNNrFV10jGA8gc6IynMHfxTxHCLvTKXoDWQHGl2Gf2aNFSmXcTUkU71FwEJanSKiYXvrU0f9aIMxW2a1E/+Yk4mMgp5EiKpx6kHIaoCnmLm5cLjPE+czxTUcqLzUrMo7Wi5XmCmp4gqoW1WSG5Ut6jwSiRKrl2mui5DEYcScYNJBNIZLWKyj5yJeWAepRlwoml2vHJEDF84fAUc6Z7pLd8s4lH2VEPR1tkZECa6tZOjiE+SM2SIs+lZOUEnYV793RwcHCYBe7p6eDg4DAL3NPTwcHBYRZ4X/rSl/6sj8HBwcHhzx/cu6eDg4PDLHBPTwcHB4dZ4J6eDg4ODrPAPT0dHBwcZoF7ejo4ODjMgv8XIPfg9AplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjI5MTcxCmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iago0MCAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQzODA3KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDQxCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDM3NDQ1IDAwMDAwIG4gCjAwMDAwMDc4MTggMDAwMDAgbiAKMDAwMDAwNzg1MCAwMDAwMCBuIAowMDAwMDA3OTQ5IDAwMDAwIG4gCjAwMDAwMDc5NzAgMDAwMDAgbiAKMDAwMDAwNzk5MSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDA0MDEgMDAwMDAgbiAKMDAwMDAwMDc0MSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA3MjEgMDAwMDAgbiAKMDAwMDAwODAyMyAwMDAwMCBuIAowMDAwMDA2NTI0IDAwMDAwIG4gCjAwMDAwMDYzMjQgMDAwMDAgbiAKMDAwMDAwNTkxMSAwMDAwMCBuIAowMDAwMDA3NTc3IDAwMDAwIG4gCjAwMDAwMDA3NjEgMDAwMDAgbiAKMDAwMDAwMDkyNCAwMDAwMCBuIAowMDAwMDAxMjMyIDAwMDAwIG4gCjAwMDAwMDEzODAgMDAwMDAgbiAKMDAwMDAwMTUwMyAwMDAwMCBuIAowMDAwMDAxODA4IDAwMDAwIG4gCjAwMDAwMDIxODggMDAwMDAgbiAKMDAwMDAwMjUxMCAwMDAwMCBuIAowMDAwMDAyOTI0IDAwMDAwIG4gCjAwMDAwMDMwNjggMDAwMDAgbiAKMDAwMDAwMzE4NyAwMDAwMCBuIAowMDAwMDAzNTE4IDAwMDAwIG4gCjAwMDAwMDM3NTQgMDAwMDAgbiAKMDAwMDAwNDA0NSAwMDAwMCBuIAowMDAwMDA0MjAwIDAwMDAwIG4gCjAwMDAwMDQ1MTIgMDAwMDAgbiAKMDAwMDAwNDkxOSAwMDAwMCBuIAowMDAwMDA1MDA5IDAwMDAwIG4gCjAwMDAwMDUyMTUgMDAwMDAgbiAKMDAwMDAwNTQ2MiAwMDAwMCBuIAowMDAwMDA1NjIzIDAwMDAwIG4gCjAwMDAwMzc0MjMgMDAwMDAgbiAKMDAwMDAzNzUwNSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDQwIDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSA0MSA+PgpzdGFydHhyZWYKMzc2NjIKJSVFT0YK\n", "image/svg+xml": ["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n", "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n", "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n", "<svg height=\"255.952908pt\" version=\"1.1\" viewBox=\"0 0 460.8 255.952908\" width=\"460.8pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n", " <metadata>\n", "  <rdf:RDF xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n", "   <cc:Work>\n", "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n", "    <dc:date>2021-09-16T14:38:07.181826</dc:date>\n", "    <dc:format>image/svg+xml</dc:format>\n", "    <dc:creator>\n", "     <cc:Agent>\n", "      <dc:title>Matplotlib v3.4.3, https://matplotlib.org/</dc:title>\n", "     </cc:Agent>\n", "    </dc:creator>\n", "   </cc:Work>\n", "  </rdf:RDF>\n", " </metadata>\n", " <defs>\n", "  <style type=\"text/css\">*{stroke-linecap:butt;stroke-linejoin:round;}</style>\n", " </defs>\n", " <g id=\"figure_1\">\n", "  <g id=\"patch_1\">\n", "   <path d=\"M 0 255.952908 \n", "L 460.8 255.952908 \n", "L 460.8 0 \n", "L 0 0 \n", "z\n", "\" style=\"fill:none;\"/>\n", "  </g>\n", "  <g id=\"axes_1\">\n", "   <g clip-path=\"url(#pa6ef972376)\">\n", "    <image height=\"227\" id=\"image73e0c99637\" transform=\"scale(1 -1)translate(0 -227)\" width=\"447\" x=\"7.2\" xlink:href=\"data:image/png;base64,\n", "iVBORw0KGgoAAAANSUhEUgAAAb8AAADjCAYAAAAVKBpKAAB11ElEQVR4nO29V5ct23kdtirs2jl0jqdPn3xuBC5yBgFRpCxTYWho6NF+8Rj+GfgdHvaDrRcPSvYYpigG0SJBggAR7gVwE3Dvyblz986poh9I1Zzfur2PmyJFgdrffFq7914VVq2q6jW/+c3P+c53vpMZhUKhUCjmCO5/7QNQKBQKheLvGvryUygUCsXcQV9+CoVCoZg76MtPoVAoFHMHffkpFAqFYu7gz/riUwup+DzuHuftaj3M26Xiad5uLDmiT2ulhA9unDeHI2y7c1biLqZ74mGfQxxeEhfwo0y+s9MY25uMp3k7CIK87fmyT38Cket4MMnbpQznVnexrfaoL/o/iXBsRQfbqhSwz/E4En2enwzydrVaydvf/sbrefuN16+IPn/6J++g/94Q+7z2mpmFKyu4DhH9f1MoVfP22fFE9Pnf/pd/k7cPDnFNnUIxb3/+C58Sfb765U/n7V63l7fHdA2GE7TvPn0h+j96/BR9RuO8nZH+uNRYFn18D+Pbb+M4h70OjtlI+B7+0qyX8/bm2lLebi2tiT6rm7vn/m6hgTEMPDmnPA9z1zjUpvnaDw/MLBz2MD+mU4xbHOPeCUM5p6YxBitO0eYxEMdljHFdHA/fSlGcoL8jRzFJ8F1G+/HoevB27W1kdFGzFPdrkkqxOZ/rZCLn6H9CoYBnAR+XMcaMhqNzv+NnQbkUiD6VAu5lPms+/rde2T33WIwx5vHP/+Lcfdoyer+CZ11GO0ppPOwxbHc76JNibJplbKsW4PhdV17rMV1gh8a66GMM69Wq6NMd4TljEuzTNTi3ZkP26dHcLdD2Trr4++BUPkNbLdxXp8NO3i6VaX5m8hXVPsP1HdAzo0TjUS6XRZ8vf+MfmPOgKz+FQqFQzB305adQKBSKuYO+/BQKhUIxd5gZ8/OiU/G5EL6btwMXnHDgIwYRjyTX2t4HsT0Ygi8mGtu0TymWZ4yhMI7xUsTFPIPYUxzFRkDw5xz3wLvd8+V+Yp95Zfw9Il69m4CvHo/BLxtjTCHG0E0pNuES39zpyZjFo/tneftgD9z1Vgu/qZdGhvH2j97P2z06hDdfEvNzA/DfaYrzjAv4ey8KRZ/PfOkrebvk4xyCZiNvL622uItZXsZ3q5vYz8Ym4mcUzjA/ef8j0X/143t5u+BjPPcPjvL26ZmME1ze3cnbnWPEoftdjO1oNBV9el1sY0jxjI/vPMnbXvFE9HnDYAw+/uhjHCfFhDfXZTxydWUlby+trObtUpHiIzLcJFCg2FOanm+8FFpzP46jc79LMwy8Y0VBOQboUvzM9fF3jr0ZI2N2fGxpyjFIuR+OX3ne+f9nh6Gch7xfn+YEHzMfix3zmxU/8/zZYzui2JHroE+p9JKLRRhTHNZ1MQaeJx+vfKyDEe5zPp9Goy76NFst2jb+XuL90HZ9S9vgUawzDSlmV8d246mMI3O4l7dWCfAM5rlqjDHlMr4rVnHveDS/lqo10acQ4PleXULb9SgmbD3qazUcd6eP5zPPG47vvgy68lMoFArF3EFffgqFQqGYO8ykPbcuy8/TBqikpIQl5jTG+3M4EF3MkKjOQa98fnsgaYtogiVvmeg7ljd7nqQtHFrqTyagUViSnVh0UbWIU2eqIE2wn0IRUtyJ0xH9vTOcbKWG82ksLubt+qoc3p6L302mkP0HVVA6h90j0efjJ3s4Bw+UyJtmNir1a3l73AelEWXYf4vk+8YY87U19Akc0AYZSYiLFUknpBNQiP0+qMWzLuheh2mkckv0PzpCn5Mj0I7LK6ATb9y8KfosLWMbo143bzfqoGBbC5J+b7Uwd6dEe0ZT0F1+SfZ59fVX8/b77/wkbx9SusaThw+sY8NxLy6hvbuzm7evv75hZoFpumIRY82S+ziRcz9K8V2pQnQTbctOj5hQGgW3nQjb4utmjDEuHQMfJ6cZMX1njDEZ3dpRyKkOTFtafYi2jGkDYSbPIf+9lddSLGK+Ms3IY+AUZAhE0GR0DmEsn02zENHY+ER1hhZnV6L9Mj3LlG4YyfMsE4UYFEEtphPM3W4fc3qhJWnTCc3xoov+4yGlFsmsNkFRB4ZSJ+h8olCGFgoFmm9TonQp5adclM8Pr4jxGNDxxAme4UlsU/ZEoy618vZ4PKG2DFHNgq78FAqFQjF30JefQqFQKOYOM2nPWk0u2afkTjLok2PFGJvonEp1Y69Nm09AR1RLUMI1K3LNPfRAZfVHUPAlRNnVKg3RZ0rr9pA2F03JscKiZByiHWo1HJtPirdwjDHodaQKs0DqpjpRDV4B/WsWvXJ9F5To+iIogE9/BpTjYCj3Y0iteTIYmotgOAB10iUKcv8EFGoaym15EcYqTYjKIkrGVsyyqmvjynreTkKiQJ9DkfnweUf0Pzk8zNunx6A9m1Vc362NbdGnWsO4v32CuZJFuL5rW9KtpVwhpR9RNwEp1vxAnlurDmXazgbm66VVXOu3f/qB6HP1tTfy9piuo1O42P+YmWEKEXRPwcc5l4pyWyHRhjHTUkT5pZbXCCsfiy7aMd0TqaWiTGgbfCt5LA28YGXQlDbgWfQqs6AJ0Y4x0YHifFxJizlMyRK9mnEfy70mpbFmtSerTV+GISlWK/T8sEnTlFThrCTl44wt2rPXx1bKdF+Ou5j7Hikdnzx9Lvo3VlrY1hj35agL2pLnlzHG1OnZtFqDUnlEz8A0lBc7IAozo/uqTRL1qrWf1iIp5ol+9ws4z9FIvlPYHaxIND+HBoLCxa6brvwUCoVCMXfQl59CoVAo5g4z14fxRNKekx6W4+0elvbtIZITTw5FF8N51LUaLdkNqLDUEnEx5RaNsOSPyqQASiQ16LlYcgdEO8bEgdqMjENKUq+EdhRjP2NSNFXrOE9jjKnV8JmpFmEInMiTqxh8bi2j/0INl2F5cUH0+fav3cjb73xwbC6CChu7khHsH/3Bf8jbWSTVWg0y3Y1IGTshFZVv/a+0s7uVt99661/l7dd2QIH+7BTU9YeWUfHN67t5e+GLMM0+PgYlu3cg1a+vvnE9b/P1CTyMZ6UszdLDkNSnRJGXRRK2vBUOnkPV2SPnhedPKTE+kDTO5mUc24RUpdsbK/SrnpmFXg+0lGvYvBmzN4otaR4rQYm65qRfW7nJSsOI7rdZifXGWInxhmlCUoh6lvSS+/P+XTqfUJKDTDuWyxiDmKisiM4tTq3+dJwcwmCq1j7PcEaSemQl+s9CSGMYU2jCDrUU6Xg8Qa+S0YBlTG1oTOtExRdoHg/OMKcm1nN7mZ6HBQdjWCWT+6NDafBwfIztrVI4oBBgWyNHntuIn7U0bn2iPf2avMcOj8iYPsa93Gji2EoVqV5NDbY9IrV5Zvi6mQtBV34KhUKhmDvoy0+hUCgUcwd9+SkUCoVi7jAz5tdry3hVOEUcqX0EvveQwlBhIjP4ayRLZ9lwQNyxF8j3bzQlWTrFbiYReOhSRcZ06rUmjueok7enBZJ7W1YQtQbOJ6PAI8dd/AL6LC9hH8bIQqZTzq+g8xxOpeXNlFwZoiH6P7mPoq6vfU4Ws/36N27nbctUYSamETmFlHDcX/7KN/L2imVSvbYKx5eI5NbtNiTVZiIPwKdxax/idw8ojjMcY2zWF6SrzMExgsQPHj/L214JPP+9O9IM+8OP30X/U/TfXEVKRKEqY3EBOdNkUwQEaj7+Hlmxo8ePENsrOxREoN+NLQPtjz6C40s0Rex4SnGYV27IIqCMU4qPcooLh57CUAY0ihzfpCnOjhe2qTSHvBK6LxKO/yUythiQO0eZ9smG5LbDC2+Pv+O4uG3E7AYch0U7irhQMMcc5SNsVgHdhAbRTuPgMH2BilFfNNVhOsa1TslofIUcSIwxxqf4G7vHcOpJGMkxdEmt0Ou28QWlV2TkghJP5XiyS1afnkflEo4lKMtn40ITKVkNOmZDaROndL8bY0yPYnunR5jH13av5u2zttQsFAJKUaNLcmcfBZ9rzZZh1Om5Xali3J7vYZ+ZO/seY+jKT6FQKBRzB335KRQKhWLuMHNd/+yZpD2ffwyKZ5yASiuVQVEVXUknMGtQJVeWShlt26R6oQWpf/sMUtbOBPvcubUr+rx4Djl8cx2S9w0yrz49lvUJO2f4zCatrDSeDkEtDAayrlypTGNQxxiwxDwpSuoooVSHCW3u/jMs8xe2W6LPxiaMkBdXQMXJKmgSQ6LjvAJogs984XN5u1iUkmqfKCY2Ll7dRDqDZ2QflqmPQ9Aej9toO3WkPVSmRNsYY44Pcd1++fhh3r75OtIe2OT2L/vAwSKl75aXMKeCsqQ90gzHXSKqpEIU13QiKerwIVJEXIM2p9KYVBroHr0AfR1FuPa7lzfNRcCUG48t1+ybTuV4SGNo0FcTMvd1LdrT4UnObUGBWg4vDoUQfJzblM2bbdoztdIy/tP+OT3CqgHI+2WzdKZhuVZh9hKHl1kUqGfRmZyCwIbipZKkz2dhRKbM1RJSA4qWw1NMYQIOLQQUnokii5Ll+qRco5HmhEdjkFlm3KMu5kvRx7OR3avqVRnSaVVw3BVKMTkhF5bBRO7niNIjJiMc22iE/RwTHWqMMbdeJWcroo6zDq57tyPp1UmIe/HGTTxblui2yKiAwMugKz+FQqFQzB305adQKBSKucNM2vPw2DKcnoCO9MgdoFrFUppNnf/yM+2I6M2M6L9yVTqaBGVSjAZY8n72M1/I26uXdkWf+iZRmCMsmYfH+3n76GBP9BmPsXxukqNAswkKMyFK594DUFrGGPPzj6Hsc4sYg8Ya1FW7r0t3giL8kc30DBTCcRuUwbt3fyn6vBngEi1utPL2wfnlzf7ydysY03/3e9/L2/ceP8rbniuvLxsUM03GtdeCknS5MQWc6ybRs7/xm9/O23Uyn13Yx/Uwxpgr93bzdoccUdgQvDeVrjAZ0asB0T3/4JtfzNu/fE/W2dvfA73aaLVw+ORewTUZjTGm2cCcOHsBVWmbFJmZVYvu5MX9vL2xBfXpt775WRzLi4/NLCwvQWXX65GLUYZztthIMx2B75mMqW7hdLY0mKlBlxSvBaprx44uxhiTxjhXVrkKavMTBjHnm2Gzi4rFegolJocQmLZk55XkE6bd7PByvhK1XLboTDqGCSmyHcsAexZqNMdbNG+MRfuGREWzC1NIz6LQcrzxyRTdpe0VqL5hhe7LaCyv++AU99XKCqmtia3u9SW16FZwDPWFVt7+0S/u5e3jUCr7p0SDNmp4/jy6j1BNsynvsQIpfTM6oCopOm1Doz4VjR0QvcqOOdORVVh2BnTlp1AoFIq5g778FAqFQjF3mEl7hrGd5AqqIHNAtbgJaIZKRSYxjxJQVmFEBslELbDSyhhjjsj8uL64jG0T/Xf87JHow0bGP3z7x9h/BxRVIZAUBteFqxBtwfQIm6Wub0FZZIwxExf77HRI9UjGtidnkrLb3KD9bIDqaBdAOwwn0vh4qYMxvfYqEkYP3jMzsU918u4/Bu3w/ACqUjeSx1ZhBRybJbN6zJFzot4C3dI7hFqzaEA7fOrTb+K49mSSa5Hopxtkch1OsJ+SZaCbEEX01lufztuv3oSp9Du/8weiT7UL497Oc1ItU8W1kaVkzdgE2KfE5WWq3Vi0TB2qoFG3SeG50sI57L8wM1GrgzLzKGZQKfPfJUV1PMQc75FKbkwUW7ks6SamPacx5gGrfO0Eb0/UdSQ6kf59tpPcRR+hJiaK3fr3m0232YCa/56Q0YCt9uRagawcdURivLVPShhnY2zXvZirRIeocDclPrEi6dXFFoypeQwTCg/1LQUxH2tKSl9OTF9dxJx86st7dEqU5oNjNrDmcIa8CJe/8HreXiCVfkr3xHAk4y6nZC6SUrTHzVj1vCP6DAY4tskU9zWrm+NUXqwihdbaZ1SfsI/nZtOmtWdAV34KhUKhmDvoy0+hUCgUcwd9+SkUCoVi7jAz5pdmku+eTMFFlygGMSJp7TCUMZ36CmJ2pyeIhZUCxCDWm6uiT0rv43IF+5n24A6SWq4fd+9Byu7GJIWnGJVdWLJG0nY2uQgpTsCFQ+0YyI1tnFu6CV66F+I8Q1/y95MO4itcnLdW44KTkuMe9nHe5WtWocsZqC9hTH/tn/3LvN3vIhb4vd/9d6LPrau7eXswAP/e7yN+5/iS5//St97K2wFJvI8eIbAVhOjfKEoZ95UdjOGbr8PQu7kAqXS1IeNqz58jbrm6vEG/Q2z081/6vOjTvvejvJ2QMfR9KpS7+cqW6HP9FcRXmw3ENpsLiIGUrZhOg2J+hRLHwnHd3zezEZN8PSVHlRoZcxdimT4Tk5T9jKp4OuSIUrJiIB6lkqRdXOuMHF6GXeloxM4hAadE0LHFlhk2u7fUalX+IodvOQ1xqgPH7BLav4gtWsbYMTmncMpOQO45JrOO02Uz6/NTJV4Gn+7ZpSbmRxzK+79BBbB5P50O7pFqWbrCePRw8jhAGuH6rCziWfbpNy6J/kuLcG9pNtG+fw+6icxKF7l9A9soZphv/SHm18GBdGsqUPA3o7SYlXXc4wXrWoWUKlT0MYZlKmA7mUojd4di4ZxJwmlP1eBiazpd+SkUCoVi7qAvP4VCoVDMHWau611XLoVF6kOMdybTEREZABtjTJMcNJa3LuftxTW0axuSbgpHkKy2jyDTP7wPQ+NLV7ZFnxWqRdchs+SUZMdlX1I/Di2zhxOifkjyHhIN5BhJR9TI0DsmqrVWhktH4kpKZ2/vMT5UW3mzTtRPMZWXZHSCYzgVxrDSjJbB9QVv3n4F2+rj2P783/+u6MPuHkvLGCuubxhOJe3pVyDdbq6sYFsJzuEf/eav5+1KXdZhHJILCSuaY6KlJrGk37cvg255+ujg3N997R9+TfR5soBzcKkm4cIB5s0XfuPTos/OLijViJx+iiWMk13zzqH5ljqUSuK8zIacUGaTaPTxp7jHyokcjzSj2olEK53Q3F0g029jjHFLuA5pH8eZEHVkU35Bcn76y9IK5lR/IGk+ni/L5BTS7eAaFi3T7e6AHDz6oHQzGutqFfMusYycpzEZOZNblEOhkqJbE32cIsaD05vSTFJus7C9iTSoFaLFC568R2NKL2o0QO2nMbvayOcu1woskxvPzuXdvH3zGtJqXrspn407uzi2eg3nfXZ2C8dZkNea6dr+Ka5Ph1Np+vJZv76ylrcX6hiDRp3cr2ryGVwip7C1AH36FEprp3I/nMriUFpKo4X+bnaxFBVd+SkUCoVi7qAvP4VCoVDMHV4iZ5LvRdcnlQ370nItLEf26Xc7efuf/g//Y95ev/5G3n706Ii7mPYeVEjdF3fz9mCIJXecSqpjNCEV0hEolSVyIEksFeUamT+HRJWe9shhZglKpd0br4r+21dwDgOiako+luUf/PRHos+9j7CEL5MrTUq0y9SX6sYTMupOPdDApnTFzML/9X/+dt5ubcP5JEtAIw36HdFn/wAKzeUljM1kMnvc730IV5elU/yu/QKq398r/hk6WFT6iGhPl5wbHj99lrfv3nss+pyewsScDcn/xb/8rbx9+7Vrok9pDOVmkajJm97tvO1Jf3Uzpvp+LlNh1E4iSXuyQ0pKlEx8wf8xm+QAEhaJYj4DDXVCLj3GGFNycQvfvEpKP3L5iSbSzceQc0qV3V6I2a/ULPeaCn7XJeX3yjqo4/RY1muLyLB9TA4eTNVmlgG3T6bV4ZBqKlIIobSIYxtHkooP6P7h6RZOQB/2E0vJ6hFFTOpI15fzdRaWF9CHDeLrVWkEH5O6cUyOJDUKH3iuDK906ViLBfS/fR2hoxtXQG36vnzOFah+ZSFA/1L1/BqGxhgzHeN3J0eYO6x0frMlVfp1D9vga1AkZ61KVZ5by2/Q78jRaEDz1apvyGEprj/pkDq6XNR6fgqFQqFQnAt9+SkUCoVi7jBb7ekE1h+w3MxIVVYg2iVLpDoqolps7/303by9eBl00yuffoO7mKMGFEGn9+De7FHi86Xrt0SfvT4pTj30H5AqdTIcij4uJSEvLoG66Y5BF1XIaHjbUpjefAPnYBKM1dkzULUfTWXS/5IPCsMhRdWkC/pskEnaIitjfH2XqFozG7/+LdQ+TAo4h04bx7P6m1IRyRTRlAx0N7dBbySppIFKTHWkGOudbSh43/vwo7x99x7Gxhhj+mQCvn0FlJ1H6rMNSw385a+gbt/2JRzb7lWoMzPJrpiQzLFP2FmazNLXmsvcxXhc546olgIZuY86Ut0YEZ1XJmruJaUXBVxS6TI1GZHSMbJqZh6dgubnZG3OJE97cu5nGejEEtGzpRboouUlqVRcq4Oa22tjn3tEw/aHcjx8Sjae0Hz3Czi2Qigp2RIZU5Spll3i8rZAlU6mkhZj0+yUxjMh5aZN862s4FwX6LxTi+afhYCuCVOdvuXaXScVZL+PZ0FEJuTGl/usV3H/lon2bNQQKil4OM9iUSoqI5q7EcWrHHpOisKrxpiggvtiNAX9v74OenU4lLO6QVTjlFS+XDe0UpFzt17CfmJStTZLOB53qSX6+HSsIRlWxHR9+wNJa8+CrvwUCoVCMXfQl59CoVAo5g768lMoFArF3GFmzK9omeGOyWA0IUfREhnbepajSTwAN/+7v/2H6FOGI8FX/tE/FH0WdyDhT8jtpFkFX76ye130qfiIhU0Njjukwqe9niwCevICjjGdIfjmjQ1IiDcvI47UqMl41/On7+bt9TWkQTRaGI9KSfZpUfyOXR0o68E0LMeLyhLOp0qXZPQSFfY/+6dwVbl8FTJ/lptnlnsFx/mGZGwdkpR8ajm8xOT6E9HvuD0aIZ4xHn5V9qd5VFvE9a2TOXCrLgskl8iJJkkp8ukQ/9+X13owwfG0ybC5schuHNatkFGMO8Q1OTxAqsX+k0PZhQqMXqvtYp8XjEEMzyB/P+mg7VDsurgoY3HP7z/N2y7dzmVKTbLqm5qMXG5Cgy+3XsF9+dZnZWpP5xCxvaUKjmEwoeLPYxnzW6bCv2Pa55hjkJHsU6LUqVWKxXVG6M8ZVVkqTy4NcV95dG7GZYcYKYVvNOgzF6PNLhbzcyg+O6FnjmvFFiNKQ/ApvcJQbLPbk3PlyX3MsSvbcNOJIhSGjcnVpuJKF6XIUBoVjc1ggHuiar0GEjpvvkdfuY3n7t2PHog+i2T+3uvi+kYR2mkq01ocCs67tJ9GGdsqF+X5jEaIERfouJ2AzjO6mKOSrvwUCoVCMXfQl59CoVAo5g4zac8okUv+FhmUBmQsWyID3H5P0mIPSYbdbKL/g5/9cd7e2ZLv3xvf+Ad5u0D7HD+H68fx3Q9Fn2BlN2+/dhM0n0fm0eNQ8oSHe5C8tw/38nZKNExQB8/4wV04zxhjDK3MzdIy5PiTkBxRSnLJXiEaNcxo+U5S/KBoURBEjx7ukevIhpmJfpfk7wmuSZnrXPmS+ilV8Lm12MJ+HDb6lWMojJ3ZcJYk3iGlvwSeTOM4bcMRZEguJFwXzknlnArJ8YGPRrirWNO6T+bNDjloFKkupWs5ayQp6DeXZOH9PmjPJ0+lO9Gtt0DZO5QC1CU3j5ehS7Rnv4trvU3pHpd3Los+T4+JHqU0mQJR0idPXog+7FmdEmX3fH8/b+/2N7mLmcag8x4+vpe3R/u4HkVXPjO8KY5hjZ4ZvRhju7ApnUKImTNZkegvMoL+8CM4C42tNA6PNlAunJ+elVl19gbkRMUhHY+pcJlxI3BwAmoyo/SIRk0aaG8vYr4FJVwrTmXpTSSN+5DckhYWMR77R6D2l5aoVuJQUovsbBWRew4Xw5sMZbrJqAfa0CX6/9IiOdFcXxF9rl9HKtjhfidv7+2f5O0O1QY1xhjfofcIp17QvWPXYZ1QegM7jbn0nPE9GX6bBV35KRQKhWLuoC8/hUKhUMwdZju8WCXny0VyB6AaSx6tpBNLeXX/oJO3l8lFxX8AGub9739f9KmvwUVg5xqcXL7/wx9jW41fij6lddBnrcuv5e3VJXAVVTJrNsaY5gqOJxqTwpRMe+/fB71z8lDSnl/+0mfy9uIa+h+RoskpS7VXFrbytkc0XUKOFeOppCB6x6A3pn1Qc82X0J5jMgQekqHveAyKqNqsiz585ZjqZBWkZ80Jl1Rq/E1K88CnOnBuJvs7VLdvQDXeygHoHa9oOd6Qg4dxyF2IqGPjSarEI7onJucTQ4rIzFIqT4nCS0guOSzh786apI6Xr4IKapO6cZTKc5iFgJRtlSL2ee0K5tdv/ZP/XvQZkBL1Jz/5Wd5eI+p6eV0qZhNS4O7v4d5pUw3Cr3zjy6JPOsD1Scnw+UkEqnSpKanj+gLOe3MTtFgYsdJRjs3iEsbw8nW4KP3sXdyLf/7nP0EHy7S7QtTvRp1U4HQNK2SIbowxaytw9/nol/fRh1TC5jWpfmWkRLmVSCXvWDR/QErllOpPcjjBC+ScqpOZdKWGe/bRU1Ct18jdKAwltTgY4TOHVCoVVvPL++X0BH3CGOewvopxeuX1HdGnTDRuRqEalxxZnh8+E32GlEHglXDeMf09spSb05jvS9yzAwoZFPyZrzUBXfkpFAqFYu6gLz+FQqFQzB1mrg8tf2XjFYgiIuooNliiOmVJezDLdUYmwP4Yy9Wf/+g+dzGLG+/k7ZtvwMR4SrWfHj0/EX1eW4My7eAI322+idMrtSy3Y1Ixjc5wDpMpqMFrW6BNlptfEt1bK1fpE1RLjTqogWhkBE4eY9uOi2V+SgpGx6IG/YTqiwXSeGAWfvY+DKQ7MaiWvRegqLY2JQ1848Zu3l5bAfWUxqB+eicyeXxKJ+gQHVkjM94qUTU8h4wxpkg038P7MB1YWwVFtntTGk5HpP50HFzDhNTJqVW70SNaiY2PgzKugefKWyFlVRlRZvUl9PnU6iuizygmxWuA8x5NpCJxFqqU3B+TUnJKCd5NMkc2xpjLl0A//dH/i9qJW5fWsK1AhiM2roAma22h/w8++Hne/rf/978Xfa6uYL+XthBOKBVwnq1FOXc3LoG+7vdw/7eKmNNVSxEZBPj84R2Yov/O//MHeTscg5arufJab9Sx7VWi9qZFMoVelnO/WqI6imPMr5NjOd9noULhjRol0Fcqsp4f14LMaL4VyWD92UePRZ8iUaURza+DPaiOT447eTuwEutPuwl9RyYIZVJ0WmP4jOqbXrqMxPZFooeX1uQ87JxClTqi+dpjhacd9iBV5pieMzGZZHi+DEf0yUjBK+E50xvhfGpVeS/LAA+gKz+FQqFQzB305adQKBSKuYO+/BQKhUIxd5gZ80ss42Sf3DBS5m4p1uJFMrawQCGecgW8cquKL9ptaeT6zneR+lAJwPe++QXIve+8947ok5Cp6dZV/C6egrMf7h2IPtEUsUGPJPfjE5J0h+CelxdlnKAzfJK3p4+wn3EHEuTUMvbwQhynQzx/gcysrfqXJpogruZ7F/tf5Xs/RBHgX+4h3tTu4DgvX5IODQcUk62VHuOYyUV4aJmDT0Y4wTLFOm6+ciNvX7pM5rUF6dYyoDhQuQo5/s8/hCT6wZ50UXnzU9fy9mILbD4XVY6syZtEFJemuItLRXPTWM7djAybDaUdBBS35BiMMcYEZcSbjk4wr3/w54ilvfb6m2YWruzu5u3xIq57tUROR2MZSN7YRBzmMhVcXttBXG9iGQpfuYZ75MkHJD8nt5cH9x+LPqfPMA+q5NZUW8D+e1aqUzfG/Gg0cK0cUvNzsVNjjDk7RizrP/4hngV7L/B3Li68bDkibVAKj0/XdxhgDrzod0SfgIzDM3IaCSoyHjkLO5egOeB91usy2hRRsV82Icmoz9hyQWHnoSOK7dXJNP/BQ8Tyl1dkWsujfdz/xSK21aBr6FkpGcdUaPd6A8/9QhXPr7OufG6zmfWTZ3DMOuvg72Ei7/8iFa1lpyF2jipYOoEepRBlIY6bY35nXXmPLF8z50JXfgqFQqGYO+jLT6FQKBRzh5m0Z721KD7HA8qmpzp9ToKlfGyZYVdqkMPyMnkSwZWB3TeMMeaYJLx77/8gb3/zN+D2kg6lnHcQd/L2dUppOHwAA+xyfMpdTGJwDEMyg2Vv3pCW3/tHkOIbY8xwiv309nFu/VPQdO3DM9GHHTwmIUuNyYzXqivnkfuCSxSA9LWw4GN8YhftLnV61pWU3eAhKKpwQrLlHmgYy4zD1KvY9vIGrvXJvU7eXumhUzKQFOb4BOPz4ftw0PnoIajjdUtSvb0DI+RGDRRVkVJHHKsM24ToppSoFzZeiSzKjmvjnZ2AusnIPaK+KCmmJ887efv3/v138/bxycWMrb/8hS/kbZ/2ExH9fvehrKP2wftwOyJVvOlRTcPYlxdu7wjXtzvA7xpEXW9tSQePJtW8Y/n5lAzjQ8tt5egF7qv2PsZgYxsHWmxKB4+f/vnHefvRx5gvJTJBrtAtcqkl3VpqVBzTW8bciVYwVyKLauVwxOZ1pHG0RtIAexYcCg2EdK2SRO6nUGb6GtvOiPKzaeCU6guKb4jmf0p1JT++84R/ZR4f4ZpwqkNrAZRstSqfp80FjMdf/BiuQVxns1qTdGRM59PpECVK6UhxIudhj+4Ln0zIPQrvTHuSBk6J8k7I1cnxcX09y6R+FnTlp1AoFIq5g778FAqFQjF3mEl7djrSlSIjV4WMHDN8MhFuLlruEwmWn+Ej0IZ+AX1WtqULwuQQKswXL9Cns0fGp5bq0NRBNTx/D8v0o2PQO62W3I+b4tgCogb3j6GcKpFR79AynN57Suqmu+gThqSustRiTJkVSGHlkQJxau2HiQKmv14GJwPvVyAlWaWO43E8SXXUGy3sJwAV1azh7+OBnBNTcmU4bENh1XMwbtMirtvoUNJ/2Rnm1MMnUOMeUY229e010ccjM2rXJbqI3TMiyXt2zjp5u0SqY4+olsFYjvuwT/XBKlD6OuRO8t5H0qj3F78EBRlSXbeVNUmPzsKrr8CUPWb1KU2CZ0/lPh89wD1yaQMKz1O6R9pDSR0dPQdNNjnDddvawFj/+q9/Q/S5vAs6sFgExbRHbkv2/Lx5HbUHB1TTkKak+b0/+APuYt4nA+uYbKZq9H/6jSVyGmlK1yOHanBe+9Kn8rZLjkbdWKoBPSYU6dk2nZJK9lCGZxjPDzAGVXJ1GYcy1NJo4RhOyS2pfQr6f3mxKfrExM0XA1C86+v4XToFzdjpWm5CpF5dXqV7ycE5j6ZSDRye4fPeAY6Tld+VinReCYhqZBp4yqbyqWWMT+GZgJSoMfXJjKRKx2TUn3rnv744vPQy6MpPoVAoFHMHffkpFAqFYu4wk/b0MrmsdcQSE8vS4QB00aQvlVvFApbcV7ZB/fSoxtQy1e8zxpixB9pg7xAGzX/yp1C5jSZyKbxFCevHVCuw3e3k7XJLGiT3T7Gcv7QOOqBDtfCu3IbCdGVBHqcTYT/3f/Ewb3dJnXR5SSa5JpRMHxEtxsayiZWgnZEK0ZGXZCYOnkLxtXQF5sslSpZea0iV3BIlck9iXMfpENe6WJD/K/VJzZZRHz9Be9QDJXN8KumV+gTbDoiCzEjxxqo4Y4yhYTMu1fPzqB5fapktOHSctSrooowcBUJLmVcnc+92D9+988Nf5O3hWM53h1SQNRpfz73ghTO4X1yXEvOJRmo2pAr7a1/5St7+OlHpB0T5/+idn4k+P3nn3bw9ofl+/fIl2q6s57dCydMeZWjvbrPKT4YjLm8j+dsl1fJHH8Cw+uO3H4s+0yFR2aT626Ik+RuLGIOsbBlbfw7zffdzb+Tt9gD3pTeyajc6OO6Q6scV6WfySksMQkzKcQzasVyRtfl6U9CgB89A8y81EY64cXVL9NkjSjU1oAmndL/tbFE9P8usYdLB7xq0nx4l+jvWEqjVwj0y6OGebYgwjhyRowNSttN91VjAthwjx71MRv31Oql0qX9qqbAfPUUC/XBEdWVpTlo+3TOhKz+FQqFQzB305adQKBSKuYO+/BQKhUIxd5gZ8yuX5FeOR6bM/HeSI6cTyQP79MsKFYxsLSL2VCxZqQ5UOPTJCfj/3hQxECsbwNw7Qhzm5jXw3+4UXHzcl24rvV4nb98hI9SYYo7HZ4jlXbshufhHTxGPPCJzbnbZiEIpqY4pFuayZJ8kv0IHbowJSfbrOxcjs0cDmHNH1KVFcc+KtamKj/+DCqQU7j7HuYWplJX3M0jbgyo494UCuYFQnCLNpPOCSwG8wSliICHFoZKxjJs6IeaHm2E/GcWk00TK0qMhjnM8QNsvYk7WF1ZFn+fPEQf63d+HW8vBfidv71yWLiiJQ3E6ir917NScC4Gdj9FsWc5LXNyWYzeXd3Bs66vy3GIq2NrrIQXhK1+Fw8zKojQ+L/p8HXFAbN5c9OX8MCRt399DesW//l//bd6++750JKmQNL9axjW9QSkvC2Syv3Ab6RTGGPPVf/FbedtdQ7xp2KFUi1Smwuz3EEfqkanzhBxrnjx6bGZhQM+Z/T1sq16vnvdzY4wxJRorn9qjgUxL8SiVLKI4X0aP7jqZeS+vyvlxNGRDf1z3GrkjRVYWR7mC4/YNxS0ptulbxZ+dDJ/Z6YuL2Q5GUs9Qr+JZG03IKYyeeY2mvP9XlxF77lL6XEjPhRUr5W4WdOWnUCgUirmDvvwUCoVCMXeYSXuaVK6FswhLZoey+acT/H00lDSfS8tfh7gbh2XtkXSFqFSwHK8st/L2w31IfjOLfiuFWDKffgS3lXUP5/DpBdmnEZI5b4Blcofo1ZMXMFv+xV0Y7hpjzIj2ORzhPDc3QBk4jqQ9uDRVSu7LETmlWGUUTUp/ybLUXAQrl0ANHByAuk2jp/h7KLeVJSh6desmJOrN13EOT/cl39w9BA2TUE00z8FYF3yiQC3nBSekWoFd0NIujUcaSscKV1C/1KZ/4wYTWWvsyR7SUm4ukzOGwbl9+KGk395+54O8/eIF5lRAlO7RkawRWSIq6fgY81XcF2+YC0Gcp2hafDW56WR8j9FPtre2DeO3/vFv5m2uD3jrlZt5O7DqqDlUw5MF65mhep5WXbghUV6/9zv/IW//2Xf/gjYg+zRov9e3QL0uLeBa1ZutvP3Ff/Qt0f8apSeN6dh8orUDK3ywNsJ8n4Y8p9H/yU/+dzMLx+RUxHXlYkc+XhdboNlTmrBjSplxmvIeqVSo5h1FldIMz6mP79zJ21ubOBdjjCkWqeZdF6kWrQUcSxLLZzA7OTnk0hXzcVpvjipRpUVKW3hIaVdJKs/Nq2MjSYTnaYGcsF48fSH6rK3jOi5QOtGEXGomI0kdz4Ku/BQKhUIxd9CXn0KhUCjmDjNpz/bJifg8PgP145fJZJocBZLQkg1RvSU2cnbpd8OhpLWaVGfq9lXQHp0zojbbcj9DUgeFIzaWBm3xmc9L6qdBxs4/OcVy/OPHcMY4pLpUqVUkjlgH41KdqoBcPrKCNI/m8YhoDFJSn00tc+ACuVwE3sXUnl/8xq/l7R+9B+r28V0oVJO+pKgfPQKte/8ulHWvvQLHjMNjSXueHGMbK+u4VkkCGrlCqtZqXVLPzhRj5dO5XVps5e0rO5LGYZXu3buYE2+8eTtvl1vSWePWF1/H9m6Bdzw6xVh/+P5j0afbBnXSbOB8Tk+x/8mZVBBfvn41bzfIKDyObTL7AnAu2IcoPKZE2Xzesziq69dBcUcx6KIgIAWyZfshts01Jum+cI28R374gx/m7d//XdCe05D2WZKuH5dWMNbXSblYq2Lu3Pz6W3n7ylswAzfGGI8eaTVDZsvu+TSyMcYsVMl4vMo/u9j9ViSl9PoKjrlSlUr2WhOfHXJyYuVkqSLVjcUKniGDF6hvmNB9VaxQnVCrluUumYvfvQvTcH/MdfZkp3YHz/4CqW89MtkOi3JOpWQ/xUbZ5Sro1YFVHnFAlPsKFR4IyPnlxZ58D/VJuR2UyBWG6PJOp2MuAl35KRQKhWLuoC8/hUKhUMwdZtKexxal4xNVUSlhKV4jNVLBYglCykavkMk1UzKOIxVA8QT7IcbPXN1GLaw0kcd20sd6OolxSj0yFD6OZW29YAX8xr1HUCQ9PkMybpxiib3YlP1vryOJeHSMY15tYDxaRTkgPTq3XhfHPCaFaWwZW69vwlD7yTGouK1dMxPXrn0mb3cGUDdO+xiPyo40JOgNO3n77kPUjLt3H2PT6UuKukM18Hyqx7W8hmT6tR0c6KVd0I/GGHOphAt86TJMBHavgOpctIzPD49wfV5/A8o+9o52Akk3v/qlz+ftDz5GYuw9qhc57Uvqp0L0EyvrhkSrN1rWnCLqhWvbsRLu4mA17mz6LZtpfHC+EtYYY1wyrWeq0xFU5+x9im9Igdxpy/p1P/jeD/L20T7CJuUyrk+jIa/V9R2o+ZolHNv2LVC1b3wTyfjFhrwGDkUNXFKS8jilFqXsX5DenIVNompZIVquyGfbhGroBUQbhkM8F17st0WfrQ2MB3uvd0lVGsY4/mcv5LOxuYHnZqmBZ8GEtlWtyjE0ZF6R0k4LJTzDS5Zp92SM+yIiGrVI254YaWwf0HtkQEnuozOEmxLrFXXaxn6ay7hHB0M8i846qvZUKBQKheJc6MtPoVAoFHMHffkpFAqFYu4wM+Z31pey9kZADg/shEpOJ4Ev+XuufSrlp+CoC2VpQlolXrlHrh81Mrm9tiv7lI7BUQ9IPk81Uc3v/PQD7mJ8OriPT4k/p3gIG0nHqYwL1FuQR2djcpIoUDzBk9reyQRcdEia5BEZDTuWOfD9J5D6HlJhya3Pm5noULyWjYc3diF7TmIZG2gUkN6QTBFfGVOcb8eVrjDjKbj5LhkC9yl9ZdKHofCL+++L/lNy0N6lY1vbRNrE5SvSuPgGxX6WljAPfB/X5/m+lEe/8yNc+0FCZsuUPjM+PhJ9koRSHcgNo16HJNu3XFCGQ2wvpFSWouVsczHMcLJ5ya9mf2G7wlCTDIlnpU3YyGZ8evr4qfhmRHOiXsUY8KjdviSLTC8uYr6Wm0gbeJWK9i5uouiu7RDDhuJ8oBzOdK04qfMJX6W/Hrjgakox0MFAxsiTgNIBBpD5L5QpJcKqxHpITkEDci4ZURx6yuliU5kGFh3gORNQLNwnA/2TQxmrLRXJ9JrmMcfVJzJ8JwptT0kL0Odnm5X6VShwmgule5QwBgMrFW5MO66GlIZB83VtWZqyz4Ku/BQKhUIxd9CXn0KhUCjmDjNpz71j6QDSL2D5urlMjibl2fRMRlxDsQLqaDwFzXjWl7LUiGgDh5wPuuS24hbl8vnyVitvJwlRC2MskaeJlPbHDqS6i0SfFYjaa5C7QmS5ZLx358O8vbMMZw+H+nTH0q1lQnW/2CVjSubexyeWKfMz0IYJ1az7ppmN0QgUXq0KefPNW6A2N9dviT7dAcaqfQqq4cUetlWwHGZ4rKZkjjuiuo5XroG2PDqA24wxxvzs+2/n7d3tr+ftr3z9a3l7Y0PWJysQXeMR1ZmSi0q3J6916uP6lqroX+6QjNtKWxiRk4RDNFm1jt8dHVkuSBPM90YD9F3FkoVfDH+L/5d+gvVk3lM6rFwIlKrkEhdW8CUNXCOT+u01SO6Xqc7d1rJ0QQkq+Hzr80jZuXQbTkMumXl/4twukLXwiZ84f7OxzmiLTHc71sEUyqB+S+RiwgYrUSop2JgKDCTkoBOSi0qzhjlZqcq0Gk6dyugeyTIyJLfm2pjqGDouvmtTXcriVIZn2OmHwwEFonoTq44i10vkEFOR5o3Xl6GWiFKNemc4Nh7qVat+5Szoyk+hUCgUcwd9+SkUCoVi7jCT9jzpSLXn0EDJtlgDNZEG2MRgKNWNvGDNiDaMSHGXWQpCj7bnO6AQHQ/LfK4tZowxtQB0QhrR0tojKs56z7N6dHkB5+ORC0GdjJiDOuhDY4wZD0EvVDxSi4agwiYTSXs6pNCqValOVobjzzxJQ5VI/TqJLva/ypQUXzWXaOUprs9uQzqnXPo0DIL7ZBR+7/le3u6N5PUddECDnHWgHt0/wN+HRANVl0G7GmNMb4rxqbVAE94khWcUS1qc68ftP8M+O2RE/fj+oezjYdvH+3B1CVzQQHFBjq3L6uYimQCn51O9xhjj+5hHfVI6BsF/BrX4Kw1ShRJturUtTcjZlHyzBuq3Ts4to4Gk+VcuQcn5xudgYF2mZ05C18Bz/zN4z79lcG0+n5S9QUE+XotESVbIlPmM6PPGolS/RlTnbpTgmby1gXuJVZzh0FLplzDuPtXJ43qT9pixkwszwlxb1PXl/cIKT59cgyrk5nN61hF9HFLQjyJc05rBM6NUkWNYqSAMEvaIBqbnzKjXMxeBrvwUCoVCMXfQl59CoVAo5g6z6/lNpTJnEIHa6xK96aegPcpFqfZySY3HSp/xiGqAWSXv2BjWEAXqEZ3gW1Tp6iooyX4PKtUC0QEVi4JkhVaZaK0oxP8DKSXQJlOZ1VnwQQGMQlA30xDjFFjjwconj1y7K1VQEytWPa6E1F8vDiVFNAtFUszFEeiI3hmojrfffk/0efgMdQx3roN6unkVhtMFuz4h0X5cF+5wH3TkaIRrFblShdly/0ne/vQrN/J2lmBbUST7nFACf6eDa801wN5661XR5/vf/7O8PSQ60iP1rOfKW4FNhI+PYTa8vQ26aWlxibuYQR/HOqX50ulcjIb5+wJWMfI9srAolbmXd1FDc1jEfC/TfX1smZDf+tSb2N4qKMBUKFSZdv27pzltnJ5gThbLmIe1uky2PjzAPcaJ5L02wgSOtR4p0rNpaRnj0aU55VJIKbH6RzErc+mZR8btrkUdZwU24Ka6gWTWPrWehzUyzU4SUJAFCuOsrsj7pd/FNkhALO7l5oJUA09HbKaPucNGA6wifRl05adQKBSKuYO+/BQKhUIxd9CXn0KhUCjmDjNjfpNMyrPbfZK/v+jk7XgN7hlbZemSsbiIz8zNF6l44ySSRqzjkMhfivckZIcbx5JvPqNitqTSN0EJfHFJht9MRPGqIXHEBYpHpMRXh4kMxrFDw5jcY1yKIyUyNGkMfVepYdwiYYgs+WqXYh3NqnRVmAWfZNRZxmkY2PbGssW/U9rBz9/9KG/fJAlzyUoHmPYRq1heQgzh1jok7s0m9hP5cgz/8VchZacQqnEpzjmJZDyi0cC4ra8h/laka/2nf/J90efkGIVUK5Ti0u9h3JuNlujjOTjuZ2Tu3aM+RSum6/kYa5+k4HEs5/hFwAWff6VBMb9CUTrZbFLawk8fPMzbp6cYw0JNphAtbWzgg3d+igiH+X4VxqnbQ5qN00ccOo7kA2A4xndtejjEpHPodWVqz/oW4oYFSiHYe4Z0Hi6GvbYp3U3cGGPNht4LCxj3Z3t2ahCOp9kkp6Iq7rGTUxnHXlnD/VIqIqWjT2kHqysLok91vZW3j0/wLAkjXNN0KLUa0xE+F6kocYMM/OPMEk7MgK78FAqFQjF30JefQqFQKOYOM2nPscXUkCrVPDnFB9dgWVtw5DK/UiFja5I6c5qAb8nnx5SSMBpjyR4RTeBbBsu9M8jXY5L2ei5ohmJRUoajCdETKeiqJXIQSFhyb0l7zRTUYkJG3SnJfMsVaTJLq3QzJqrTI/NY3zIHTsj0ukaU30tBxrJBinaZTLcPD/ZFl83dm3nbJUcUdi0xRk6Ke/fu5+3vf/dx3l5ttfL2ziWYfm9TCoUxxmxugR5dXQW94xRwrcqWUW9A4xsTdd3pQEb+4OEd0SeKQFt6NPeYtuz1JN00nrCbDTkNUZtNnY0xZmEB45YSdRtO//rU3K+ChP8ieNlxrl+CU88bX/5y3uZ0k25P1msr8xyn+8Kl/9M57eFXYZTCmM34ca33D6XxeZHSOpgK93yaR9Z47h+g1t6QwjvhEP1PKO1hYjF+LTK9LpPBukem/2cW1RpRGKdH+6zTtWFHJWOMGY5wPBsbSMnwKFzlO/LZ1qbj3tsH9crHyWlgxhgzGeG415fxnBiSe9Uo0lQHhUKhUCjOhb78FAqFQjF3mEl7vvaqrPfW6XTydpEcVloBlvmVJblEzcqgzKpULy0g1aNvKcRY4XVGzgfkW2oWWpL+c5PznQcmVPK+VJa0Z4Nst0djLOEbZLBsqLZgOJVOI7ycL5FbS51ohmpVuhMUiM4bk7qS6x4ai0oL6XcxKevOzGw0FqCqev7xk7x9eozafEkit+CQ1c7ONVCgqYvr45bldCktgCJ2TkGj7p2A7jk87OTtjx+AJjXGGJ9Md7/w1a/k7Ws34fYy6bZFn19++D6OLcXYhER17O89FX0Cn8yoyYGH3Xzap2PRZ0j0Skw00LSKiVipyDnFJtyVCmhx1/nr057sWPErDTo1mwINKrhPF7d38/a1xVbeTi11c4nciVKuRUchlfRXguwEIlKClwMyki7I4+TwBrtPxVznzhrDThcU8YSozio5ErkunkUTS2HaJTP6mJ4zDqnVPevZmBATzdGewQt6ZliXgO+/yRhm+I0q7gPPmtN7hwhVjEKqL1hAn/GxpMWHQwqzFTJq0zvFn/laE9CVn0KhUCjmDvryUygUCsXcwfnOd77zXz9LVKFQKBSKv0Poyk+hUCgUcwd9+SkUCoVi7qAvP4VCoVDMHfTlp1AoFIq5g778FAqFQjF30JefQqFQKOYOM1Phpw9+IT5vb8Jg+Ne+/bm8/cGHj/L2n3zvQ9FnOIR59PYyGayW4UgwiqRzypRcTC4X4GiwUIHzSj+VjgQTB24J5RrcAeolbCuJpbtASE4f7P7iUh02Q64Sni/rSpXJ6aO5DHeC1iKcDqoV63+LFMPdOcbveqdUv24q6+wN+3RuTZg/v9ee/X9LPVg/9xwM1blKrPqEIRmHt8lseO8QxrpHe49Fn87p4bm/63J9RXbpsI6zSjX06gs4b3ZUmfSkOTAbYLAB7pXdnbx9c2dL7qdERtl0rRtNOJD84Ifvij5v/+Q9HHdEZrpUt+x/+p//leizuIK5F00whgVyE3p0PDuz6Hs/u4t9Up06Nsm23U1EOTvn/L9nlrOGQz/0yVGJ264r5xc7znhkLO8XuM/FnFcymgmpVY+PPye0Tx4PdpLxrX365LY0y3O7UJAGywG5rXhcn5CeBbvrshYd46Mf/1ne5jHwrHqEDhlYj8nZhuv5lUry2VYgQ342guLz5tIAfiavtUfnkLJpPhUAqNekY1ZMrlL9CRxmHD5OX86PEpnmJ2Q+79BYp6583YRUh9AkdLHoOCMj3w9cHYDLZI7HOLbuQLrCvPX5r5vzoCs/hUKhUMwd9OWnUCgUirnDTNrzzp0H4vM+eQXv7oBuuvP+w7z9H//gbdFnfRMmtTVDJsgN0EPTsTQULpTI1NTHutZNsZRNvJrow8vpEfkgc52tJJa0ZUZU1ICpE2JhfDJLTYys55dQrUBi7MzCEvbTbIkuplYF7ZGE2FHgYwzSSFIyYQi62Ys+Q9+smFkYDjFW0xDH2T4FhXhyIunE/SMMXGhwnEEVlMgv3/9Y9ImJ2jMeaI9KrZm3a3WmuyWlU2+CSmqtrObtZ0+f5e3FnU3RZ2UVtcKYFnvl1vW8/bnXb4s+TNMdkZku02+f/qI0WN/cgbl32IPBepQRJRTJ/x0bZFyeEifjijqXs2uNMR3H9d6YRsosmpBN0ZmqZBNluw+zgQWHrnXAtQpn057cZgrS2o34jrfHdKQwdTbGxBSOYFN3vlZ8bqlFe6bi4/lUaZbKA42ohijvn82SX0Z7RkzDkml/ZIUWCjQPmcrmY/ukoTmbNxM9S5cnGmFOhbGkCctksB4TfV+gUNFkJOckjw/XrBRlB63rllLdUcfn86H6m9YEYYqa65jGNB7jqawhmnHoJqZxJzPrhYWL1T3VlZ9CoVAo5g768lMoFArF3GEm7fnstC8+ewk+H3ZRFy4gKq+1JOvXXb4J1d3OVdB0xQxL2d6ZrCs3JQrCaxBl1mrl7WgqqYGyxwogfDccz6aLWDkVBFhyZ9TfJWogjCS9Mg7xuzjGxsZDjEevIyndWgOfq1X0KVG7tiCX+c0yqIoiMS8/k2JcgcePoMB9vvc8b5+dMO15LPp4Aa7d0iau25VrV/P24Ys90SeZ4Hy4LmOxAqq2VsPfs1RSxwGpNTc217AtokB6VNPRGGMeP0LdwCbVLaw1wD2HqfU/HSlZTQH7rFZxnDeXpEL00mWij6gm2jSjcwikmi/OQDE5RIuXqkxl3zGzUC6jfxgyTY+5F8aSbpql6pRKSbkfLi+YzlAA2zXRZtVISxLe+GwlK9N0xjmf/jPGmCQitWckzzWHy+pXiZjpWTo3l274xKp5x8+GhOjmi9ZUHBPlx7U9U2OfG37niwFhZa/c52RCqlBWvFI7HmGuGUs93x+ixmSpjrkfT/D3cChDQj6FMMp1XPfFCquZ5b085Xp8RbovaN5MLMV9SorTWgn3oqHwQRTJYzMG+xmPqI4rhZE4dPYy6MpPoVAoFHMHffkpFAqFYu4wk/Y86cpEwTVK2K5XW3l74zNQ4x2dSaq0RrRljRLbU6I26gtN0SftYBu9Dpbz1SKS3AuePOw4BFUxGmDZP5pSgrUjl9yFIimNSNFULOI4B6MethVLCrNehXq1WsQ5eESvtI8PRZ/2Gca0sYDjbC3R2JYl1dN0QU/UapISnYWIFFYRKTdbG6D2Dk8l3Xz75uW8HZShllpeaOXt3Ws3uYuJpziHhOgrz+e0W/z97HjfMKYRxmNpAeMZ+KBnzvblcR7uY0wTogD3n+HvzUxSP40VnENCjMzTJ6BxWVlojDExq9dIKeyQujC2En29AMe93MB8rfgzsq0tMD2ZkpqO92knkscRU/vcnyhQS3Vo6Fr5NF9jUq/aCdoFCg3MUnimVoI1H6njsNpz9njwNc3i8+nAmPbDqlpjJPXLbVabJkXZxyMZIyeF+/7F1gZTGt9kTMpJi29mtadD5gAFogYdR44788UBqYE5yX1Cc3U8lnRkRHO32gS16NIt6hu5z24Xz5wxXY+mj3vU9jPguyciWjkjpfJoIu8xpj2zhK6PQ4rfTPZhQxKfwg4R7Wcykc/qWdCVn0KhUCjmDvryUygUCsXcQV9+CoVCoZg7zIz53bjUEp8/9wZSFXYvQWKekMR1e1mmOrBc26Vgi0ucv2dJqFcWET8bDBDzG4fEZZflO9snaSvRxSYLwf3aYYaAOHfPsNwbnH1EsuFobDka+GS+nMDU2WXv2FDGQAYDnOuwS04QRFHHVem2UGxgv/HkYjG/VUpVmLpk6kwpCA/u3RN92sc4h+U1BAR6bTi/NCrSWSf2cE1GFP9zOG7ikXNDYsWraKw5PWKP4nwcNzbGmLc+83renpLcetxDrLiWVkWfW7twj3l4jPP54X94N2/v7x1xFxOTTJ5TYTg+MwxlPMIJMNa/9Zv/EH086dozC2y2PqXx5NBRksg5xVJwYYAtYn6WZJ9+l/C9SHEx2+GFHWf4Ow7zJVbaQpadn24hnF8so+6U42cUx+EUJuF4k8h7wuF4IMf/6DdlSwpfa2KOVUjOX6vJ59kssEHzlI6nXpX9OY6akhVNmlE81Iq/cS4HG2CzhoFjm2lsjSd9LlLMcJrg3mFTamOMGYwwWjweFbqvCxV5bkeUdtAb4oHWp2d4wZMOT0W6l3p9mu8UUJxaqT1nZ+c/MwoBtlXw7PSI86ErP4VCoVDMHfTlp1AoFIq5w0za882b0jj569+AWXCZVry/+CUcr4ftEXcxqYflOK1KTbVIVJZlClEsYWleorSDDslvez2ZUrHQIiPlBuTmwymWyOzwYIyUDa8ut/J2fwBHkbM2qIFGGQ4kf3kOJFUm6XRimHYRXUwpgPx9MMC2D8fE1a5IimyxyBTkxZbznQF+d0YuM8UJDiiKJF006COto0luOs+fwiFmobHMXUyxCEqEqQqHLrZDaSSxlW5y2gUF+eO3f5K3e11ct2ZNUpgpUdRFcoJp8aS0lP3TIcY6HmKsl2neFANJ46REizealGZD8+bkrCP69PpkHOxje+ns20yAUxKYwmTXkciiMPk7pga5Ll5mpTqwcTF/49G1smlP16UabXQvpVyv0ZL2c9hjlhm2E8s+fD6JlTqB/aDtWywhpwBwGkhEx1KwqNbiYitv83yrNyXlPgtjMoYOODXBOjiPzjuia+W41LYnr3P++HoO/Y6pX7s+IjvmUOTIoyqAaSL7VIjSXKjjdzWi/0NPzo8pHU6bKMxeG8+felXSq4nPbjroU2/hfhPhLmNMv4/0qBG5ulRqOLa1tbq5CHTlp1AoFIq5g778FAqFQjF3mMnHLK7I+mYbm6BB9w7g1HH/2UHeHrmSlisQazAtQunDyjE/ldTA6TEozckYFCRTJb4vl89HxzBsLpDibmUVZsdLK0uiz4gUWiOigXZuw+kkcR7n7VZJUiALi6BH2h3Qd0nC6rme6JPEUDHWG0QNpjjmjvRxNr8kGnY7uxjtOSQj5v3ncDEpkCNCEkk6wS+T6S3RxZMpjnlt9ZLoExSI9qQ6Yiw4c4jqrFq0x7MDXOsPP3w/b9cbmGsW+2YOnsG0e2cZVHTjVbjSTAuS9nhyBDp+kuIYrr/+Fo7ToqhSUraVaO7yPBwMpZNEv49x88ntZTyZYdBswSMqidsp3SOO5WiSEg0a0zUQptAvMbbOElZ+kqm8NR4+0b3MeLHam2lTY4yZDkm5THOiUMAGooGchzHRk0yl+3QNCnRtgkAqCIMS7iuX6loaMoiuWAOySBT3IpmL+8WLrQ0KdN7VCjmQfEIxS+p3GpBMqFotxxoeN6YnXTIhZ8ce65AHFFKZ9DDWNaJ3I3ufFIUJiIatUkjqqC1DT6c9dOrTNU3JuWU0tMywaduNJsaN1dXeVLo1FSgUxorsPoV6VlaV9lQoFAqF4lzoy0+hUCgUc4fZtOdGS3w+6KD+2/t3f5m3jxNQfsG6RUEs4vOojaX14zsv8nbvUCZ1p1NQVDcvg8q6cW0nb3uWsXW326c2qI5qQsbaDVnafm1lI28XKlgmbxFVuriE5OgHRMsZY0xQpppXEfr3exgP35f0CqsGE2IwmaJipsYYY4YT/H9yeEzqNym8FLh6BUnuP3/7x3m7ewoaNoslhUpMp5kMKTG1AXoktZRoI7pWU0rQHo7x90YDyrHXX7km+t+9/wH69Dt5u0rG2rZScUrGwSVKwF3bwvVsrK7KPkSPPHn4OG9PiLZ0rcpwUQyKhpXCwmA5tfugPapDnfz6rR1zEXD9OTFziPqyjZwTugYJKQhDMnt3Uqlu9Mk8OaWDnpKyzi9ICrNWp9qapAZko+1qXSpzI7pWbIxdq+K50B+fiD4ZJXknzK+ycQLRieWyfOaUmlRXkq5pPMa1XrEo3SVSlVZJEimFqLMT3mMat4yowdgyQSgUiMYt0DOMrs/kJbQ2mwYYD79jM35WyxtjTJuo8BEVDYiIkk6teyyhequNSxRaoGfB6EAazneI8u/1qM4nUfaJRTfXa3iGVijsElFIJrFCPS49U4n5NTGdQ68n3ymzoCs/hUKhUMwd9OWnUCgUirmDvvwUCoVCMXeYGfO79qqMmzzdf5i3702e5O1wFfzw6rIsTJtRbOHkDDHD8QDcda0i4wStTcQW1inumLE0OLMdCUimT7LhlOI2H737ruzTgjPNl7/1jbx9TFL6CsUp6gsyyHZ0BM7bJXeCYR8G0bHlJOGR63WF0gSGEzLGdqS01yOJdxif73hh45Wb63n70jrGZu8Z4iuJZbbsUvxqMka8KvFxnM8e/VL0ydi4nI4tI7Pl4grSFr72z39D9P+LH/xp3n5xgGK0PgVEy5ZcfIdie0srKK6ZeJgTTlGO+/FzmFZ/+N67ebvd6eTtaCTTFhKKhbkUe2IT4dB2W6Hre2Ubcb7bNzfNRcDxDJ807iwRH42ki9J0QvOF7jffpVQaaz8szWezZC5O7FgpSF6ZzKQpvstm1qHlxsFPl3IN83iBHJWSrkwH8ikeF1Px15Bifh7FyHwrXpVOyWWHcjoWlmDGv9WSzymXij93TnAvLqxsmYsgpNimUydnn0jey9U6Yv7FEsYj4vis7dBCqQKcNsQOLxVKYarVZIpaPMbzlI22OY7dsfKrXCq+vLuN594SFfDO7u2JPn1yN2L3qhLFcOvWs75eozgyjccoRGzSdeT1DShNxnUxVoUAz6lwosbWCoVCoVCcC335KRQKhWLuMJP2LFck7THsQ8JfLWNZHFTxuyiUy/zJGMv5BR9L/rVrWJo3ArkU5qUsuxtMKB2gYKUQsHKZl/0J0W+BVdDPjUFz3X3/3by9uwm6dzqhdAarll0YcU0y0BFdkgBPQkmlrS1jG0fHoIEdovkcV16SyRS5D5XMopVmYKWFcf/2N2/k7fYRaIveUEqqI3LAYJptMMFYd9svRJ+kDcpqiSiNMplxVwKM5+ampJG+8fWv5e079+7k7Z2d3bxdLUuKKk1B6bI5ebcLSjcIJB356B5SKgKi7xZo+j87kY4VGdF+7PAi9jnoiD5rV3F+xSZotkl8MYeXWhVzl82jif0z05HMhRlxjTeReuHR3+V+MjJSNpReUaAd2eYmxYwdRcjNh+jzaSQl5h5J+0eU/tI+w5xqrcsUpFWiwiZEabYHuJfCLtqp9cxJ6RgcktbHlFpwOpbUsUt0Yo32b6eyzEKD3FKWGnjOuZaRe4mOoUIpCVNR31CCPauZAiwR/bdMblOby5h3xhizQ6GjlRV8V6bUghcvDriLcFXa3cX95sfoM53Kc+t2ECqhqSLoXbs+Yp0+V2jCuWRYX8rkeygil5sxuXRxiM1kF1vT6cpPoVAoFHMHffkpFAqFYu4wk/Y8PZIZ/KMTLM2bBvTOlBwJRlNJ7wyGWL5u1smhpUR1yyLZp+BjaT0Yg4rKDKhBVpgZI2tgVWmZ7WU4PbsulEMaODbADmmfDx6hll2wtC36V1ZhgB1UQN3skKny2eET0cfNQMl0iXop0GXwLGdaVnjWXHnes7BUx7l+86tX83YSoT2aym1xfb9BD9TahJRTD+4+Fn0e/vRB3r66DnrTIceJtVc/l7d9qv9njDFf/QZozy99Db9jE/V4IumVCakdJ1RHrU60dNGX0/pjcpJ489VP5e3xGVRu7SOpeGOz8eVl0MVFonT7lunu17/9bRxPE84YQelidLVH15eppxopJW1Hk/4Z5mvKNBBR8ZlVyzIlyoxDBtUKqZEDOYYBq+7I+qTIZtxFSVEVydy738N145p7rRVpQtxq4DqWiE6snIBif3HnWd4e90G3GWOMQwPn0HmOHVZky+tRIMqtUcXc65MLSk1OXYE1Ms1nCrRclGOYkmM0X+symXHbczekZ2pKrkzLVPPu5hXMte11SXuWSpiviysIIZRoHl+5Jk3/uWZlHGIMOidEI1tKZ65dWKHamq0qjrNRlYNYrZKanuoG1tkg3QpX8TVxMzynUqI6HSPn4Szoyk+hUCgUcwd9+SkUCoVi7jCT9vzFLyRl9+IhlaMnI+iUjFPTqVR7jdng+HOgBt/47Jfy9iSWy1pWWz1/BJXe43tIsO6fSjPcpQaW05z4fLhPJtNFmfx5cITvtiagi6pE7wyGoJTqDUmVVBdALyxsXsnbX/z2V7APMm42xph/96//j7xdLGCZn5Gs1fIgFsqni/6v4pLja7lESjKqfVavSWrAJbp1OgV14xGNtL4iFa8rRFFvLuD6xgHObevTb+Tt50NJUR0eIaF4cRn9owiUStNKSN7chPqMmbgqKTKnI0lHBkVSmZGadhKBxplOLRNiOu9KlVVuXKtMUvZlOgaf6MBnz/bNRTAl82XbyOE/wc2spF9K5E5InZhwnT9XTiqP6OfFJYxvs4LrxgpkY4zxE0qAp2PjUoVVSyGepthGRO2MqOvRSNJncZNquRUxJxKDezQk84osk6plh9SBXKMxINPt0DIxL1INvrUdhHTaZLb+MjSJ6mTaskLUojFGFEKMKQHeI6WiZ/UpkGI0jdB/iUzytzdBW26uyPvFpwR4t0T7oUtVKMnXABtlc8J7lIJmrDXks2BnG/dl0cU5BA62XbBCOiWmm4n2DOiZl1khroAofK5jyGpPW908C7ryUygUCsXcQV9+CoVCoZg76MtPoVAoFHOHmTE/33IaGU8gBZ+0EaupUcyhWpbv0ooPPt6fwtFkncyOFy/dlDv2OF6DuNzRAWKQ415HdFlcAud9TK4jR13EFkpFq9CmB56/sQ6ef7MO7vh4H8dsy/Sv376dt1ev4hx6J0iPeO+n78p9UkzUJdLdoeK8ke0qERNn71gxhBmIyH3Gp/9v0pALY1qxI+L5DaWvcHyktSDH4M3P4LwLFP9KKca2QFL2xz2MjTHGfPgBYqKnZ7huJYrPbm1Jg/X1VcwdLkZbLeHYyoEV332O61il9ISzA8SObfNnQ6k1LFEfTxC3LFoG2h+//7O8XQgQrxpQysvrry6aWWif4B5jhfcCxVOLntxnnWI6hmKYY7IGGVqFiw3F2RsNyOTffP163g48OQ9HXaQ+RVNyWxlTfLksY35nHdy/boz5NaQ+nrUfNr1epBSkp/u4VjE9c8otmSpRpFQQj6T1hSrufzsGepnifJ/93Gfy9nvvU8z+JSY9PsWyipQi4loyfXYKCinVgY2Yo1i6QqV8L1O8i91iilR42PdkjIyNwhN63NOjwKSZlZJBv3MpXSSMMfcXl+Q8Hob4XYHdVigulyYyPsv1fMsUjyzS2MRW+pxH8dmiKGpOscnoJReLoCs/hUKhUMwd9OWnUCgUirnDTNqzKo0kzI3LVG/plGgLklT7qVxuuiz7LWEp22iBvotjKX8/eIGUhkYNfXYvX8rbQSoPu0MmzUkBtNitL76FbTWkBDggF4Lrb+F31RhUTfNn7+ftnrXPxR2kN/TJpPqHf/i9vP27v/2Hos/2Bo7BJyeahPIb2LXEGGM8cl4pWu4eszAlToOdQnpk3lywzF8DSjVgmX7GUmuLtui0qT4g1ZVrXoL7TUZS6avX8XdjjPmn//wf5+1DMgR//gxGu92+dBp6/AR1GF88hdF2vYLreeXKVdHn6lXsl2nlwhLRZ69Z9etoDFbI4SUo7eJHFq1VqeP6tlqog5aOMCePjh+aWTihMajQDXhpC/UAvaa8BgFRvy7J/Ad0LyYTeV9yCtKYjLLLddDFN6/LGoRcp3LQBz2bcC06I+nV5oBMwE/xu3YHx7O0Le/L9TWMVYHSnpgjKy+08vZaC+2/PAdQnSldw5CGwK6L+fobr+ftN99Eas6zp3CSaZ9KByAGG+gPKfUiCwridy7XWOTafPRoGQ/lGB68wPPIkIn5YosoTEq/iRP5jPAzjEeSUK1EupfHYytdhNIT+HcRPeu3KeXIGGNCSk/iOyklN6ChlepkKH0lTTjchS0ULHeiUplN5jEeY6qp6FkOL/KJCujKT6FQKBRzB335KRQKhWLuMJP2PNyTS9TGApabi4ugmEIy03UySQMFDpbcManx+lSmvmRRA4+f/CJvc9mtpQaMpdevSvNWtwC6ZmENdM0a1Y8rB/LYkmEHx3YMeuP4+DH234cCsbC9YSTQ//kHf5y3H/wM7WZT1q86PsUJXd2A+qzewBhMypax7RC0QZRcTMXEtbWCOrZ3+Az0bNKXVEd8BdtuET3rBmS2bCnJUqIQe30yLib1muOB4l6y3FqWFqE0dN+i+mYh+veGkvYc9DB3Rl2cZ4G4o1JFjjsb7RaIivfJwNe33CcCUlEG5MbDbi+BNXcdUrmVAuzz8YP7efu3/81s2rPfB0HD5sJswl6xqG+H1G9+QqpD+k1sKXtDqjO59xzU8U/efidvl2ufF312dkHjlhYwHiOisqZTWRNxYwv3jH+AcwvamDelslQwP3+2l7cP2vfy9ovDTt5eWoMCeOuKNJxPXMzr/pCMj0lhGg+lA9AaUa2syEzji1mFnJD6nYvZVctyHm4kZHpNbjgpO+ZYSsXjHqk/yRXmrI+/n3TIfatpqdpTUKIJKbeTlO5/S2E+HaPPsIt9ZsTIrtSlotrdwfxYoPdDTHTowUFH9OkNQCX3aZ9OCdsuenJ+uLRey8jKhcfNdsmaBV35KRQKhWLuoC8/hUKhUMwdZtKe/RNJcTG92VjB0pqTv71AUjIZLacdojf8Cvovrl0RfbY3obb64Y+QNLy+DsrslTdviD4eJUUXykh6LfhYfqfDtuhz9BSUSoeo1skBlIZPn3Xy9te+eUv0P7iDPu9///t5+8kD0EgHIzm8J6fY3uU1HJsXs1mr6GI8SqR2/Yv9r9IbgIJcaUKpWCY68OM7kn5rUk2yRaIaEoN54Fm7D0rYnuNin4MBmT/TFEtTq64cKRIdonRdUsI1G5I6WlrA3KkSPbK0gPMME8sMlyhRnpOs1nStk+MEZaal2HDaypU2WcTnA45o0JutFGQUq1A3ugXcV0dUsy+jGmbGGBMQFd2kenotCjlEiaT5hmQyfUZKv8f3QTk2mx+JPpXmm3m73CClIf2mH0rKbo+S9vt9oqhGVGfv4FD0GVEtyR4d24SSvQtrmJ8Lm9IEwa3QvXSCez4Z4ZzjsVRU1uuYY5MJ6NlGCwrobldSugxWQRepfp59vwoqmyg7Dh84nqTSS1V6TlCNSVZXHh5jnFdXWqL/YITjnkwwhgHVGixb1DPP8XYb14O931ukuDXGmN0bCDdVKtj20SEoYdtw2qG52x+SiQrRzZag2oQ01iHVaA1JCZv4Uu05yxpEV34KhUKhmDvoy0+hUCgUcwd9+SkUCoVi7jAz5lcOpGGsOwVzOtxnBxCK1RSkfD5LSYJ7o5W3G/Vl+pUsiri2DsPoX/sq9tmsUVzPyLhHqQKevLJIfK9DKQSJPNWDI7iTVMjt5NFz/H3qg/NvlKWE+KO/+HHe/vmPIGU/6IKvPptKk1o2VXHKOLaYxtDx5HFyekF2QQnviLS+3Ql48cVdxPWc57LA6rCEY5gUqRAqOfqWLecE458vx/fIGNtweoQvT8DjASFnDIeLslpFZgcRxrRGsnSXtuVbxUodCs55dMz8K8eKR7BEO6I4A8f/7P8cRz3I/ksUDx0PR+Yi2LwG14xyGTE7TqlIAjk/ysu4Tytdcl46wz2ybDkvjRKMx3iMqF1It9WDj15wF1Np4XhuvAa3pQlJ4Y/2ZXpUyIVIucAp6QSqVfmcyVZwrEOSv5+cIO0oI5n+Wbcj+rsxxmdM919G8yiN5Jy6exfxTd+DnuDTnyG3lydyPBgViplVK3hOlcsyHcAhM3uW4/t0TXtHUpswIgefBm27T6kOR9SnvSENpx0X++xS2lRA8eFSyUrZoftl/xgxu5VVpI41F+V+FpehyYjJnHtC4z4YyufhRNzbGIOEniWjSMaruYB0RnqEPsVqvaLUCWjMT6FQKBSKv4K+/BQKhUIxd5hJewaWTDclU9QCuVewctyWsmchmcyCtTB790ATlltSQhylnby9XMcSd3QGF5ZkKJfpBReOL50ejI8LRdCrmZEUxNZVpFic3MUxtGNwP29+4ZW8/fwO3C+MMead7yK9od0DrbWwCBrHGUoujR1rPDKcNVQzi+v8GSNl/7FU8M/EeEo/LIBWnhCv9fpnZbpIRiRgym3aqWdRbgVy7XGZWhTpAFSPzD43omSY0/VdzK9n+7IG4NEx5Pg726DfPJKRm1Ba2Y56oIWGI6LCiJYr+nJ+VGoN+oTxOD0FDfTw0VPD2N+HXHt9A+41D38529WF8bkvoz5iUCTJPLmTJ1Z+RZXoxEIH592+84x+dST6rLskmacQwhGZXHd6MrTw4C5ov5vXYRz+1S99MW9PLAPtEqU0vdjDMTh0PkvLDdHHoVp9x8edvP3Be6AmX+xjW8+saxCQIbhPY9UgKmzJouxqFNJYWYbby/oGmzdLk3rG1iauNVPUlaoMlWTkXCSuIj1r0+SUvzExyflH1OuMTKEpgmKeUdjGGGMCOreDM2yrSPU762QGbowxHtWMPCZHpfXLNNeqss+Izm08oDl1DIem/QM5DwdjpkHpOeOf737zV3/ImzyP+mSsbyRTaiSxDujKT6FQKBRzB335KRQKhWLuMJP2LBUktZgmWDKzR/SQlHBJKmmtEqm62pTp//bv/37eri9Jh4bGBhapp0UspStEYQSWG0f/BNRHlWgPz2D/Y39J9Fleh+nu47iTt29cx9L+9W1wtX/2R3dE/+M9qm8W4zi7HVBf44mkPReboHhiUi3FZEicuVKbVKPzKdegqOpKYa3AT99BTcRVqne21CJK1qrNV2BKkhwWMqJQU0+ej0c11ljtmcTsdIL+vmupRYnS7ZJ67e4duO+0jyX9trmOMeyTQXFWIBPjSJ7b3nOiKu8/yttcO7FUluRIleo/pmRN0RuBu/747iPRhymnxUXQhEePcD7Xr0h1M6O1TG5JdD18MuP2jGXaXcDnyirmTsnDvHFiS2VrMEdTumcnpKI8iaUyr3OE8z5+gvti9Vtw9tiguoPGyHqei0tQF5fImae1IKlBpj3bZzjOQRf7PzyAQfuoK+dHjRSRN3dBz16/vJO3F5qSaq1WcE2aDXJEKs0izCR8UhDHVPMusyg7dsOKSHGa0D3CVPxf/oEMm7lGY4Z9Dsm95v59GSYYTXBN9zu4X4qk8Gw05ZysEI9aJsecD+8gXPXsUNKz5Sr6ZKTQ3N/HtRpPZDgipvNmM/uYsgQ821aK6NGYnlMxyWcT52LxIV35KRQKhWLuoC8/hUKhUMwdZtKeUSgTQZmwmk4podAletST9EpM9FMSYyl87yMsn4NAJltfvgk6cnMHlEhGlMFpVx7b6gr6nBEfmLpQ+XU6Mkl1ckj7pfpzMZkQd/ZAp754IemEkCi8jStQe3HNPs9K6r68w0mioGeSCH2SVEqV+n1SiFECr6lJGpex/wKUxJ/+2dt5+5u/9oW8vbYsayKOeqCyum1c31r5fErHGGNSoqKjiBPB0Z+Txz0rSz8kg+U21eb77vdgaH58JBOnb14FTd4ntViFzsctyXM73Mc8ODnCcbaPOnl7Esq6gSld3yKZC7ukRDs6mYo+J1189srYz8gyfJ6FVrOVtwtEKVcoYb5ZlEpFj5TXFTrO9SXMyWomQxj7Ce4/n9S4PSrYNuzJeywa4XzufYD74oOfIhywvX1Z9KlUQBturWM/RapJWCxZpg50PJUA9/+NG9fy9iGpBjvtjuh/7Rp+9+UvfDZvX95G3T+7DiMrjQNKTHdsU4cZOCLD+iIZL9gG61UyLh9TwnePTLOTWI57s4ExcMlwvl7D3ys1nM9oLA0VTttUbzHF72pUDCC06haGpNYkRtWcdujZ5snndrlE2QBEVXJNRDbjNsYYLl3ItKWhupiOFSqJyUAjovAK58un7ktiQgRd+SkUCoVi7qAvP4VCoVDMHfTlp1AoFIq5w8yYn7QgMCbmgosO3pmsRE1TybUGFB/xCtjg0QF46FZTvn+3Nlp5e7mFGFmvDY782SOZdtA/ou+oOGZ9CXL1cUc6H5xSfCV0wNO/uIv40ON7iAPtnUoufmkTDjFlcqkIQziQNCpSQkymCubkrJO3SwWMbbVmy6upEGt2sRhEhUzAj2g/734Ayf3tW1e5i1lbRiwpnYLz75O0P7H4dy7WydLt0ZCKrxLP79qTip1kKO7SHWJbD/ekK0QWI55YobHy+ph7pWWZLnJM6RIjMmIeT9lY23KvYVk6fVegFILI6iMKO/ucBnIx6fWNyzB1L7BTSJncSaoynulQHMihCeZvYGzrlqm716MUEUp52aBYST+V8cxuD58PnuGa/PEf/Xne3rmyK/p86rMwhi5TekOBnII8R97/HPPjVKlbN67n7Zj0CJ1TaQR9efcy9YGLUY3cVj5xNWjusWvIJ6brDHTJHLxAgaxSJGO9w5BSe87IqJtiV4uLLXlo9NwdT/E7LoxbLGFO1iqyoDjHm8MQ92+9QfHHiYwTppRuEVBscEpFgO3rxk5QXTJ4j2gM/KK8L12f3Hh879x2uWzHZ9GcTjHuB1S4eGKlVMyCrvwUCoVCMXfQl59CoVAo5g4zaU/HkWt+Vu0GJMMWLIEl0w1DLKd9jyhR2utRWxpbv//e3bx9hVIdHtxDqkI4lX0OE1CdMZuljrDkHieSguiOQYXdfQDp94QoN2JdTLsvyZJXtkAbVEmKvrMNKX7B+t8iJQojpipTtRLa5ZJlhsttGveumY1aHccTJ0QRDUBH3HssHRo6JG2/ug0at0zmxJNYumn4Lo4nCHAO4z6ue0ZSZ8+1KC6mcYl6CokqmVpONGGIz4M+qLhJhuveCmR6RDTF7yKqb+gRr1UtSrrIpzQVr8gyfRzzqCLNsJPQctT9K3ROiLq91Tr3N8YY89p10IRsFM4OIpVAGgobop9ichRhgjq4bQRGx6CIBoMO/u7i+CepvJefhOjTH+MafPwRqPQ//eM/F31WN2Asv7ENk2iXjs6mwh2aBx7Nr/VVGE4XiVIeWjXimk2EOqo0dx1B01n75I/uBblOwpDoyB6lz5TKck6xaT2HA1oV3K8Fq56noedmxHUZKYfII2P8VlOGTUY0J8dt3JcFCkNl1muA3aeqlD4TkMOTm8px8iiEMU5AO4ZEuyaW43QashsOUZ30PDRWLUp2kuI6iI0ansf9Q5m2NAu68lMoFArF3EFffgqFQqGYO8ykPbPMcvNgQ1FaijrcdiU1WCjQsj9De0qKvb1DSaV1+qA3P/gAy9cCKYOWF5qiz5IPmq5CtKdPtbEetKV67YAUcOy2MAmpXtQIfbaIwjHGGGIqzKSH/YRUS25iqb0yWrKnRIEU6e9lKYgyGdNP3sWcC8KYDWyxQa6N1SEVqDHGPHv0GN+1Qd1+/rO38na9KZWGyZgMeelUE3LMSUnJZux6fhHNjwjbiiagLePQojDHNIZEqY4n+HtTMnamQCrVhFxqElINJqlUlbl0HRdIGcclDZuBpLWiegvbyzCve21WGt80s7Dc4PqT5ytEHUtlxx6+BerDpsr1JekKs/MGVJDds4O8PaGxzjI510Z0Tccpwg6srPvxX/xE9Ll+G2bSv7HyrbxdIjcQx3oEMa3FTTb6XiBFZGtBzknPIZUtjRXfe3ZIR+KCRTMJnT7G4JTCOMWxpPkqFVDWRY+dk9isWT4zmI0vEL3J9/KE5upoLOdHlZTfHtVonNI9VrQVlfQM5PvXI+eVxHq2eQ4pUcn1q1JBn3Ekn8GDIYXFfIwN36+2ST0jpXqpJXogV4qF837+CejKT6FQKBRzB335KRQKhWLuMFvtaZlU8yKX66AVKRnXEvNJA1k2j6UkRieR+xmQeGsY4YObUWKukXTT7V0owb6wA1VZL0Gf7/7xXdHn4QjHEJBSyadV9hLVGrt9FfswxpiCAb3RJaqjRIpMSRwbExG9kUQ47wGZA0zGUsnK6tXyIgy8zdaWmYXTNujiGqmgWNY2nMpE0E4bar7hCP3LVdCmr74ijYvrAbY9obp/p1SHbUrX0J4gTHkxfZdOiX6b9LiLGXVAnbjUyeGEWVfOD6aLY6ptR3n5JihJSnahgbnbJOPg0zbO7ezgQPQZkFIwJdqzWJntJcGwR+c82HOKf+fSl0zeeYHk0pcvY+5ceeOVvN0/Az1rK1e316mmId1XvT7aZyfHos+7P303b3/281CyCirOs3j+mcDouMJswVJu8igyvUnt7CWs50UNCRhs6s6q58CixcusnKRzcMgphGsgGmNMuULjQzT9kBXMpCKfWDRhkQwSmgu4XwcD3FdeQSrMWe3Z6eB3LinUnUiOU+CdP74eKZUdywQlpWsXU2J9QiG3YkGOR0K/65ziWcmm30wvvwy68lMoFArF3EFffgqFQqGYO+jLT6FQKBRzh5nBiCfH0jlhbRHOAWd7iHX4HmJH5aLc3OoS+N4Gma+aBgptFlakDLuygt99fICimWdU7HR8JmMt64vggd+8gbjU8T7JzSfyPZ9MwT87FOhbaCAecXUbMmovkTEQNrMpk+sGuyDYkuphDJ6+QvYxxTLGbTSVOv1JiD6DM8TiGrNDfmY6wfkUi+Dv2WT29Fg6vHCsIwxxPHc+foQfpXIMX7/9et6urSI9wj9G7Cf1KTbgyDjB2RnijCHFIBtVjE2zKMcwJlm3IWPtgHJEfKuIcEyy/YSuwWiEmEFak32KZMT8/BnMyu/dw9x78exQ9EnoGJrLuF+uvzY7vYHxUgX+X8F17BgXQcTpKRZojXuJDNc3qPjr6SFifp2+NDteplv7GgVLH0W4hrHljjIlKfvBHoqfbmwhLl94SchPpj249Hf5K/mRUhpmpE28HH99h5dmDc+sOqUNlMrSAcin+5zdjnwSVERWwWhDhYg9Gt84wTzm2JkTyv7jmFxdyHEmGWH/IzttgdIwQipibEh34X/C5J4v5PnxO8dyrymUcZ+zAVdK85Vdi4wxZkqOMROKO/aHGI9i1Yr5z4Cu/BQKhUIxd9CXn0KhUCjmDjNpz/svpHVyn1wMirTkDUly61ka4gm5QjSqcGHxC6ADyhbdVG9h2/Hes7yd0bbHlgvCD+7gd6cnZHLNteQqoovZrGE/xFqYrRUcZ40Us6OBdKJpNEHXDomaHA9Ymi+X7BWiQTwDOmFEFNPUke4Ekwz/n/ToGjTMbBRLOPAS1eNiiTgbXhsj6zLW66B7Y6IWnj+ThrHFEujN9Rb63PwcZO0Fkmrb9FunC8qs08G2L2+DFs9CmfpxSsbBDtGeCdE7e89lDUBOSdh7CgehHqUtjM4kRXV8hAnzgvq3Oxi3zHKSSYmyurRzKW+/9ilQi8MDWYtSwPn//1/0E6Tc+UznS6lBduqotZby9qVboGc7p/JaJ3dBeW0TFeWQm0dvIh08KuQuxHPPtSgzeag8R2ZRmLNNqv/m+Otvr0419BLi7wq+vJ4x169kaT/9ZjSR4ZWCi/BTyubRFB6JiVocZjKFies1lhqY0w47zFgm1eIKEFWZ8HPXOjeHrY/I7olTZpJMPrfZRNwX6R5oT63UDT7vhOZhf4D731WHF4VCoVAozoe+/BQKhUIxd5hJez56KBWVT2n5fPkSSLeVJii2mmWQOqE6V90xlqs+1UFLBpIqee/Ow7y9tIj9rC9gyd4bSWogIhXipEjL4gBU5XpD0loVUj55Ho4zJuVTtwfKYaEl62R1yIqGl9xVWnKXi1LKJs2KyRGBBFoDywx37wTn0I+w7V0zG2VSKgqxFFHHi4vyfEKiE7h+HFPHnbZUAH9490neHm9in9dIJTum8bTZrsVFGJS3yRXm0g6cbAJfUqXPipiXwwH6HJ5CvXp08Ej0OX4BqnNCJuYe/e+XTuT86HQwr12idFprcPopB7JPlMIp49LGVfRpgsZ9Ke35XwqWYk66v+Bar2xi3Hdv3zCMPplzF0hV2iBj6d5EUlT1VYQGlpfR9oQy0KYZL0I7/m1TnX8zsIm4ME5PJtYPMY/ENSAK0q4nOGInFzKsn0R45kREoQYWdc633GSI42GjcNdSRw8n2LZHblwZ7Se1Ch9kRMmyf31M7w2uxWmMMeRLL5xx+KhdT55PUKSNkxJ1Sur96ehirkG68lMoFArF3EFffgqFQqGYO8ykPa9eWhKfSwWsUWtk1NusYMleq0ra06Hk4nEfyr5SCQnrq+tSt1hbIVqJkhXDPiiufl/SOAkZXU9D0IQDqm1VsZbPdTKQdanuYJeSqB0ySw1KlvkrLfszoiqihJJxQ6lu4vEgNsN4xAdOJrLPwTEl9xtJs81CiWjPkNRWQ0o6LhbkeNTqoJWndHD9LiXJBlaydBFj0pngHH5xH0nhwSqu2y2L0llbwxxjqrbZRBJ2pSRNakdjjPsv7t/DF3T81bK8VtMaqLmN1c283ahiPwVfji0r4BoLmKNFUq96lpo3irGNKkmIe12pnPy7hm22wCyoQ8nWZTJB3yS1qjHGnOyBOt4ng+JGFWM9tJKlK8uoT7i8jGvNlNvLa+v9/UDK9DmPrUVBFojuLbDpP6koM0fGBkjsbRJWwtKwBQX0YfMMYyS1Lx29iaq1EuNd+h3XF2QD7iiU4SpR49VhChR9vII8N594zzjCc2pKBQA867ntFkghSuxmRPTqdCiV+bOgKz+FQqFQzB305adQKBSKuYO+/BQKhUIxd5gZ89taljGQrTVI49lglWncgl0Al2S/KRdSpThU7EgHD+bCE4o9pcRD21x6OWBtLU6pVG2hXZLxyKCIbdTJ+WBEctyYUihia58JV0KNwVGPqeCkqUiO23NJKsx96Dy7Yxk3OR5grKKXuQATApLmy5gfZP5RIGXHayThHw9xDgVK3ahUZTyBv3MK+C6jQMXZGfY56kvpt0dODg26BrdvIU1gZXlV9Pn4PszOn1IKwxe//dW8vbgkXb+f1Z/n7TrF4iqUqpC69q2A+ba9CSPmhRb6FFwZK3nyDM4yJRdjOLYKFP+qgmXttYWm+G51E2N6tI90k5RiR0XLWWOdxq1Sw/PDvn//viNi5xOK3xetVCc2f2Zp/zhEXD2zYvGciuJTLK0SUHFwKprrWflEKT3Pih5+51Kst01OR8YY4zi4FwLaDzuvRHZhai7knJyf0uVbrjCcIjKNKSWCH01WTLhcwTZqlL5WpWdTMrxYQeL/tmahQqFQKBQXgL78FAqFQjF3mEl7Nq2aSGwszQr+Atelq0qqdNCHyTMbvrpklhonlnye3A64ll1G8ujMl4c9nIBOC8g8ukBOJZ7lLjKO0Cfqg5pjkwo27R0MpfMK68V5n6UG0672GKLP2QlSGA7J2PrRYU/0OerjgMoLLzEEJkQRU52Q/RYKOJ/USFfmhDTaGdGzLtMwviVVdqg+4BFSWdZWIZNv03k+foAUCGOMqTcwd1bX4ABSIEplh6gzY4xZWGph/1QPbHENv3NSOT+WVkHhra+j/0IT7UpDOt5UaV5vLyFVIqBx2nssnWT2RnCZ8SjdYppcjIbJsov97m8TItWA6EjfksxXF1r4js4tSTHXSlWZYrKygbSSAtW8dEjO/8lT/rsfg78p2medvB0Q1Vksyfu/02F6kZ6nZAheLMi5W68hHFAjp32frtWInFv4mWfvJxOm2/h7sSRd/9OUwlXkfsX0qu9Lyn9K9TgL9DvHnZ3KUqZQVFCgMNAIz6bUmg5FClUUqB2SM9Ynx+B86MpPoVAoFHMHffkpFAqFYu4wk/asNaTzShpjae4TfeYRRTUeSuNjVxgS8LIWVJxrlbafUM2ncUxL6SIvueWxdolOCCegHSql811LjDEmIUcCn41dE3JrIRWWsZwG2Bi7XMY+HSrylsZyPKZTUAVHx6DInhyCGnxyKo+zyx+lQHMmej1sj+nmMh1zrd4SfbjG2tkZjq1ShuuHb12rmE13yb0mDnHet2+jlh2PkzHGjEYY39NTjM0SmSUHnqSb33jtet5Og/8ub18lI+ZCKvdDU8csrbBqmegRX/aJSM17+Gw/b+89Qrt9LM3fh+Tk0iTqh2nxcGBm4r+G24lklUhRbcUJaq1W3t68ciVvV2s4t8CiShfXYJTNdeGc2YUH/16iR8b2DlGQoeXwFFENTa6Nx6p4m+cbjnAvLWcIDTjkaHJ8hLqaxnKIaS628raf4jufHqKNeo27mL0jmJinDvZTq4HWrtRkiKvXxXEWqY5jnZyX4liGWvidUBduSxgPLhpgjCyWEJFiPqI6iKmnak+FQqFQKM6FvvwUCoVCMXeYSXtu3nhN/pBMVQNOZmVDU6vGU5l+F0WgyNhguT+QPBBTAC4pEllFZawk2azUyttM07Gp62Qs6cQSKbGKxIulpDZtd6C8DKyRWuQkYKIW4ymW/2EiecpJB+daWQVF1PLQZ3dF8kAblGjfIurpZYgpYZQTS5m1dF2piOpTAnoc4Vo310BBsvm0McZ0e528XaeE07NTJHvHEc75xq1Pi/5Tqhs2GmMMmP7zLTPcr37prbz9xS+jPSLFXCGWfbpd0LgvDqE4PTlF8nmxLpO6J1TH8O4dqDoDMkFvLC6IPu0TJN1zQrBD88vSDAuwyu6/JLIZbYdqstmJ6M0lGFO/+YXP4nd0v3S7lmEFm8HTjkQtuP8GaM+IDJodivWckArUGKm2zlLuQ3SzlTzeYxo1wn1VoOsz7GNWjUI5w8Y0j9nIvURKaWcin42dHsJSIdUQ7FMN02pNUqWTMRlY0DVlmrJekerXkqgbit9x0v1pR86pMoUQ2FQhInOR8qKk32dBV34KhUKhmDvoy0+hUCgUcwd9+SkUCoVi7uB85zvf+ftnqaBQKBQKxd8AuvJTKBQKxdxBX34KhUKhmDvoy0+hUCgUcwd9+SkUCoVi7qAvP4VCoVDMHf4/qr3B0AS5ChcAAAAASUVORK5CYII=\" y=\"-21.752908\"/>\n", "   </g>\n", "   <g id=\"text_1\">\n", "    <!-- Augmentation examples on CIFAR10 -->\n", "    <g transform=\"translate(120.904687 16.318125)scale(0.12 -0.12)\">\n", "     <defs>\n", "      <path d=\"M 2188 4044 \n", "L 1331 1722 \n", "L 3047 1722 \n", "L 2188 4044 \n", "z\n", "M 1831 4666 \n", "L 2547 4666 \n", "L 4325 0 \n", "L 3669 0 \n", "L 3244 1197 \n", "L 1141 1197 \n", "L 716 0 \n", "L 50 0 \n", "L 1831 4666 \n", "z\n", "\" id=\"DejaVuSans-41\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 544 1381 \n", "L 544 3500 \n", "L 1119 3500 \n", "L 1119 1403 \n", "Q 1119 906 1312 657 \n", "Q 1506 409 1894 409 \n", "Q 2359 409 2629 706 \n", "Q 2900 1003 2900 1516 \n", "L 2900 3500 \n", "L 3475 3500 \n", "L 3475 0 \n", "L 2900 0 \n", "L 2900 538 \n", "Q 2691 219 2414 64 \n", "Q 2138 -91 1772 -91 \n", "Q 1169 -91 856 284 \n", "Q 544 659 544 1381 \n", "z\n", "M 1991 3584 \n", "L 1991 3584 \n", "z\n", "\" id=\"DejaVuSans-75\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 2906 1791 \n", "Q 2906 2416 2648 2759 \n", "Q 2391 3103 1925 3103 \n", "Q 1463 3103 1205 2759 \n", "Q 947 2416 947 1791 \n", "Q 947 1169 1205 825 \n", "Q 1463 481 1925 481 \n", "Q 2391 481 2648 825 \n", "Q 2906 1169 2906 1791 \n", "z\n", "M 3481 434 \n", "Q 3481 -459 3084 -895 \n", "Q 2688 -1331 1869 -1331 \n", "Q 1566 -1331 1297 -1286 \n", "Q 1028 -1241 775 -1147 \n", "L 775 -588 \n", "Q 1028 -725 1275 -790 \n", "Q 1522 -856 1778 -856 \n", "Q 2344 -856 2625 -561 \n", "Q 2906 -266 2906 331 \n", "L 2906 616 \n", "Q 2728 306 2450 153 \n", "Q 2172 0 1784 0 \n", "Q 1141 0 747 490 \n", "Q 353 981 353 1791 \n", "Q 353 2603 747 3093 \n", "Q 1141 3584 1784 3584 \n", "Q 2172 3584 2450 3431 \n", "Q 2728 3278 2906 2969 \n", "L 2906 3500 \n", "L 3481 3500 \n", "L 3481 434 \n", "z\n", "\" id=\"DejaVuSans-67\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 3328 2828 \n", "Q 3544 3216 3844 3400 \n", "Q 4144 3584 4550 3584 \n", "Q 5097 3584 5394 3201 \n", "Q 5691 2819 5691 2113 \n", "L 5691 0 \n", "L 5113 0 \n", "L 5113 2094 \n", "Q 5113 2597 4934 2840 \n", "Q 4756 3084 4391 3084 \n", "Q 3944 3084 3684 2787 \n", "Q 3425 2491 3425 1978 \n", "L 3425 0 \n", "L 2847 0 \n", "L 2847 2094 \n", "Q 2847 2600 2669 2842 \n", "Q 2491 3084 2119 3084 \n", "Q 1678 3084 1418 2786 \n", "Q 1159 2488 1159 1978 \n", "L 1159 0 \n", "L 581 0 \n", "L 581 3500 \n", "L 1159 3500 \n", "L 1159 2956 \n", "Q 1356 3278 1631 3431 \n", "Q 1906 3584 2284 3584 \n", "Q 2666 3584 2933 3390 \n", "Q 3200 3197 3328 2828 \n", "z\n", "\" id=\"DejaVuSans-6d\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 3597 1894 \n", "L 3597 1613 \n", "L 953 1613 \n", "Q 991 1019 1311 708 \n", "Q 1631 397 2203 397 \n", "Q 2534 397 2845 478 \n", "Q 3156 559 3463 722 \n", "L 3463 178 \n", "Q 3153 47 2828 -22 \n", "Q 2503 -91 2169 -91 \n", "Q 1331 -91 842 396 \n", "Q 353 884 353 1716 \n", "Q 353 2575 817 3079 \n", "Q 1281 3584 2069 3584 \n", "Q 2775 3584 3186 3129 \n", "Q 3597 2675 3597 1894 \n", "z\n", "M 3022 2063 \n", "Q 3016 2534 2758 2815 \n", "Q 2500 3097 2075 3097 \n", "Q 1594 3097 1305 2825 \n", "Q 1016 2553 972 2059 \n", "L 3022 2063 \n", "z\n", "\" id=\"DejaVuSans-65\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 3513 2113 \n", "L 3513 0 \n", "L 2938 0 \n", "L 2938 2094 \n", "Q 2938 2591 2744 2837 \n", "Q 2550 3084 2163 3084 \n", "Q 1697 3084 1428 2787 \n", "Q 1159 2491 1159 1978 \n", "L 1159 0 \n", "L 581 0 \n", "L 581 3500 \n", "L 1159 3500 \n", "L 1159 2956 \n", "Q 1366 3272 1645 3428 \n", "Q 1925 3584 2291 3584 \n", "Q 2894 3584 3203 3211 \n", "Q 3513 2838 3513 2113 \n", "z\n", "\" id=\"DejaVuSans-6e\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 1172 4494 \n", "L 1172 3500 \n", "L 2356 3500 \n", "L 2356 3053 \n", "L 1172 3053 \n", "L 1172 1153 \n", "Q 1172 725 1289 603 \n", "Q 1406 481 1766 481 \n", "L 2356 481 \n", "L 2356 0 \n", "L 1766 0 \n", "Q 1100 0 847 248 \n", "Q 594 497 594 1153 \n", "L 594 3053 \n", "L 172 3053 \n", "L 172 3500 \n", "L 594 3500 \n", "L 594 4494 \n", "L 1172 4494 \n", "z\n", "\" id=\"DejaVuSans-74\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 2194 1759 \n", "Q 1497 1759 1228 1600 \n", "Q 959 1441 959 1056 \n", "Q 959 750 1161 570 \n", "Q 1363 391 1709 391 \n", "Q 2188 391 2477 730 \n", "Q 2766 1069 2766 1631 \n", "L 2766 1759 \n", "L 2194 1759 \n", "z\n", "M 3341 1997 \n", "L 3341 0 \n", "L 2766 0 \n", "L 2766 531 \n", "Q 2569 213 2275 61 \n", "Q 1981 -91 1556 -91 \n", "Q 1019 -91 701 211 \n", "Q 384 513 384 1019 \n", "Q 384 1609 779 1909 \n", "Q 1175 2209 1959 2209 \n", "L 2766 2209 \n", "L 2766 2266 \n", "Q 2766 2663 2505 2880 \n", "Q 2244 3097 1772 3097 \n", "Q 1472 3097 1187 3025 \n", "Q 903 2953 641 2809 \n", "L 641 3341 \n", "Q 956 3463 1253 3523 \n", "Q 1550 3584 1831 3584 \n", "Q 2591 3584 2966 3190 \n", "Q 3341 2797 3341 1997 \n", "z\n", "\" id=\"DejaVuSans-61\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 603 3500 \n", "L 1178 3500 \n", "L 1178 0 \n", "L 603 0 \n", "L 603 3500 \n", "z\n", "M 603 4863 \n", "L 1178 4863 \n", "L 1178 4134 \n", "L 603 4134 \n", "L 603 4863 \n", "z\n", "\" id=\"DejaVuSans-69\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 1959 3097 \n", "Q 1497 3097 1228 2736 \n", "Q 959 2375 959 1747 \n", "Q 959 1119 1226 758 \n", "Q 1494 397 1959 397 \n", "Q 2419 397 2687 759 \n", "Q 2956 1122 2956 1747 \n", "Q 2956 2369 2687 2733 \n", "Q 2419 3097 1959 3097 \n", "z\n", "M 1959 3584 \n", "Q 2709 3584 3137 3096 \n", "Q 3566 2609 3566 1747 \n", "Q 3566 888 3137 398 \n", "Q 2709 -91 1959 -91 \n", "Q 1206 -91 779 398 \n", "Q 353 888 353 1747 \n", "Q 353 2609 779 3096 \n", "Q 1206 3584 1959 3584 \n", "z\n", "\" id=\"DejaVuSans-6f\" transform=\"scale(0.015625)\"/>\n", "      <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 3513 3500 \n", "L 2247 1797 \n", "L 3578 0 \n", "L 2900 0 \n", "L 1881 1375 \n", "L 863 0 \n", "L 184 0 \n", "L 1544 1831 \n", "L 300 3500 \n", "L 978 3500 \n", "L 1906 2253 \n", "L 2834 3500 \n", "L 3513 3500 \n", "z\n", "\" id=\"DejaVuSans-78\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 1159 525 \n", "L 1159 -1331 \n", "L 581 -1331 \n", "L 581 3500 \n", "L 1159 3500 \n", "L 1159 2969 \n", "Q 1341 3281 1617 3432 \n", "Q 1894 3584 2278 3584 \n", "Q 2916 3584 3314 3078 \n", "Q 3713 2572 3713 1747 \n", "Q 3713 922 3314 415 \n", "Q 2916 -91 2278 -91 \n", "Q 1894 -91 1617 61 \n", "Q 1341 213 1159 525 \n", "z\n", "M 3116 1747 \n", "Q 3116 2381 2855 2742 \n", "Q 2594 3103 2138 3103 \n", "Q 1681 3103 1420 2742 \n", "Q 1159 2381 1159 1747 \n", "Q 1159 1113 1420 752 \n", "Q 1681 391 2138 391 \n", "Q 2594 391 2855 752 \n", "Q 3116 1113 3116 1747 \n", "z\n", "\" id=\"DejaVuSans-70\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 603 4863 \n", "L 1178 4863 \n", "L 1178 0 \n", "L 603 0 \n", "L 603 4863 \n", "z\n", "\" id=\"DejaVuSans-6c\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 2834 3397 \n", "L 2834 2853 \n", "Q 2591 2978 2328 3040 \n", "Q 2066 3103 1784 3103 \n", "Q 1356 3103 1142 2972 \n", "Q 928 2841 928 2578 \n", "Q 928 2378 1081 2264 \n", "Q 1234 2150 1697 2047 \n", "L 1894 2003 \n", "Q 2506 1872 2764 1633 \n", "Q 3022 1394 3022 966 \n", "Q 3022 478 2636 193 \n", "Q 2250 -91 1575 -91 \n", "Q 1294 -91 989 -36 \n", "Q 684 19 347 128 \n", "L 347 722 \n", "Q 666 556 975 473 \n", "Q 1284 391 1588 391 \n", "Q 1994 391 2212 530 \n", "Q 2431 669 2431 922 \n", "Q 2431 1156 2273 1281 \n", "Q 2116 1406 1581 1522 \n", "L 1381 1569 \n", "Q 847 1681 609 1914 \n", "Q 372 2147 372 2553 \n", "Q 372 3047 722 3315 \n", "Q 1072 3584 1716 3584 \n", "Q 2034 3584 2315 3537 \n", "Q 2597 3491 2834 3397 \n", "z\n", "\" id=\"DejaVuSans-73\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 4122 4306 \n", "L 4122 3641 \n", "Q 3803 3938 3442 4084 \n", "Q 3081 4231 2675 4231 \n", "Q 1875 4231 1450 3742 \n", "Q 1025 3253 1025 2328 \n", "Q 1025 1406 1450 917 \n", "Q 1875 428 2675 428 \n", "Q 3081 428 3442 575 \n", "Q 3803 722 4122 1019 \n", "L 4122 359 \n", "Q 3791 134 3420 21 \n", "Q 3050 -91 2638 -91 \n", "Q 1578 -91 968 557 \n", "Q 359 1206 359 2328 \n", "Q 359 3453 968 4101 \n", "Q 1578 4750 2638 4750 \n", "Q 3056 4750 3426 4639 \n", "Q 3797 4528 4122 4306 \n", "z\n", "\" id=\"DejaVuSans-43\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 628 4666 \n", "L 1259 4666 \n", "L 1259 0 \n", "L 628 0 \n", "L 628 4666 \n", "z\n", "\" id=\"DejaVuSans-49\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 628 4666 \n", "L 3309 4666 \n", "L 3309 4134 \n", "L 1259 4134 \n", "L 1259 2759 \n", "L 3109 2759 \n", "L 3109 2228 \n", "L 1259 2228 \n", "L 1259 0 \n", "L 628 0 \n", "L 628 4666 \n", "z\n", "\" id=\"DejaVuSans-46\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 2841 2188 \n", "Q 3044 2119 3236 1894 \n", "Q 3428 1669 3622 1275 \n", "L 4263 0 \n", "L 3584 0 \n", "L 2988 1197 \n", "Q 2756 1666 2539 1819 \n", "Q 2322 1972 1947 1972 \n", "L 1259 1972 \n", "L 1259 0 \n", "L 628 0 \n", "L 628 4666 \n", "L 2053 4666 \n", "Q 2853 4666 3247 4331 \n", "Q 3641 3997 3641 3322 \n", "Q 3641 2881 3436 2590 \n", "Q 3231 2300 2841 2188 \n", "z\n", "M 1259 4147 \n", "L 1259 2491 \n", "L 2053 2491 \n", "Q 2509 2491 2742 2702 \n", "Q 2975 2913 2975 3322 \n", "Q 2975 3731 2742 3939 \n", "Q 2509 4147 2053 4147 \n", "L 1259 4147 \n", "z\n", "\" id=\"DejaVuSans-52\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 794 531 \n", "L 1825 531 \n", "L 1825 4091 \n", "L 703 3866 \n", "L 703 4441 \n", "L 1819 4666 \n", "L 2450 4666 \n", "L 2450 531 \n", "L 3481 531 \n", "L 3481 0 \n", "L 794 0 \n", "L 794 531 \n", "z\n", "\" id=\"DejaVuSans-31\" transform=\"scale(0.015625)\"/>\n", "      <path d=\"M 2034 4250 \n", "Q 1547 4250 1301 3770 \n", "Q 1056 3291 1056 2328 \n", "Q 1056 1369 1301 889 \n", "Q 1547 409 2034 409 \n", "Q 2525 409 2770 889 \n", "Q 3016 1369 3016 2328 \n", "Q 3016 3291 2770 3770 \n", "Q 2525 4250 2034 4250 \n", "z\n", "M 2034 4750 \n", "Q 2819 4750 3233 4129 \n", "Q 3647 3509 3647 2328 \n", "Q 3647 1150 3233 529 \n", "Q 2819 -91 2034 -91 \n", "Q 1250 -91 836 529 \n", "Q 422 1150 422 2328 \n", "Q 422 3509 836 4129 \n", "Q 1250 4750 2034 4750 \n", "z\n", "\" id=\"DejaVuSans-30\" transform=\"scale(0.015625)\"/>\n", "     </defs>\n", "     <use xlink:href=\"#DejaVuSans-41\"/>\n", "     <use x=\"68.408203\" xlink:href=\"#DejaVuSans-75\"/>\n", "     <use x=\"131.787109\" xlink:href=\"#DejaVuSans-67\"/>\n", "     <use x=\"195.263672\" xlink:href=\"#DejaVuSans-6d\"/>\n", "     <use x=\"292.675781\" xlink:href=\"#DejaVuSans-65\"/>\n", "     <use x=\"354.199219\" xlink:href=\"#DejaVuSans-6e\"/>\n", "     <use x=\"417.578125\" xlink:href=\"#DejaVuSans-74\"/>\n", "     <use x=\"456.787109\" xlink:href=\"#DejaVuSans-61\"/>\n", "     <use x=\"518.066406\" xlink:href=\"#DejaVuSans-74\"/>\n", "     <use x=\"557.275391\" xlink:href=\"#DejaVuSans-69\"/>\n", "     <use x=\"585.058594\" xlink:href=\"#DejaVuSans-6f\"/>\n", "     <use x=\"646.240234\" xlink:href=\"#DejaVuSans-6e\"/>\n", "     <use x=\"709.619141\" xlink:href=\"#DejaVuSans-20\"/>\n", "     <use x=\"741.40625\" xlink:href=\"#DejaVuSans-65\"/>\n", "     <use x=\"801.179688\" xlink:href=\"#DejaVuSans-78\"/>\n", "     <use x=\"860.359375\" xlink:href=\"#DejaVuSans-61\"/>\n", "     <use x=\"921.638672\" xlink:href=\"#DejaVuSans-6d\"/>\n", "     <use x=\"1019.050781\" xlink:href=\"#DejaVuSans-70\"/>\n", "     <use x=\"1082.527344\" xlink:href=\"#DejaVuSans-6c\"/>\n", "     <use x=\"1110.310547\" xlink:href=\"#DejaVuSans-65\"/>\n", "     <use x=\"1171.833984\" xlink:href=\"#DejaVuSans-73\"/>\n", "     <use x=\"1223.933594\" xlink:href=\"#DejaVuSans-20\"/>\n", "     <use x=\"1255.720703\" xlink:href=\"#DejaVuSans-6f\"/>\n", "     <use x=\"1316.902344\" xlink:href=\"#DejaVuSans-6e\"/>\n", "     <use x=\"1380.28125\" xlink:href=\"#DejaVuSans-20\"/>\n", "     <use x=\"1412.068359\" xlink:href=\"#DejaVuSans-43\"/>\n", "     <use x=\"1481.892578\" xlink:href=\"#DejaVuSans-49\"/>\n", "     <use x=\"1511.384766\" xlink:href=\"#DejaVuSans-46\"/>\n", "     <use x=\"1559.779297\" xlink:href=\"#DejaVuSans-41\"/>\n", "     <use x=\"1628.1875\" xlink:href=\"#DejaVuSans-52\"/>\n", "     <use x=\"1697.669922\" xlink:href=\"#DejaVuSans-31\"/>\n", "     <use x=\"1761.292969\" xlink:href=\"#DejaVuSans-30\"/>\n", "    </g>\n", "   </g>\n", "  </g>\n", " </g>\n", " <defs>\n", "  <clipPath id=\"pa6ef972376\">\n", "   <rect height=\"226.434783\" width=\"446.4\" x=\"7.2\" y=\"22.318125\"/>\n", "  </clipPath>\n", " </defs>\n", "</svg>\n"], "text/plain": ["<Figure size 576x576 with 1 Axes>"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["NUM_IMAGES = 4\n", "images = [train_dataset[idx][0] for idx in range(NUM_IMAGES)]\n", "orig_images = [Image.fromarray(train_dataset.data[idx]) for idx in range(NUM_IMAGES)]\n", "orig_images = [test_transform(img) for img in orig_images]\n", "\n", "img_grid = torchvision.utils.make_grid(torch.stack(images + orig_images, dim=0), nrow=4, normalize=True, pad_value=0.5)\n", "img_grid = img_grid.permute(1, 2, 0)\n", "\n", "plt.figure(figsize=(8, 8))\n", "plt.title(\"Augmentation examples on CIFAR10\")\n", "plt.imshow(img_grid)\n", "plt.axis(\"off\")\n", "plt.show()\n", "plt.close()"]}, {"cell_type": "markdown", "id": "f3aa3360", "metadata": {"papermill": {"duration": 0.035548, "end_time": "2021-09-16T12:38:07.331113", "exception": false, "start_time": "2021-09-16T12:38:07.295565", "status": "completed"}, "tags": []}, "source": ["## PyTorch Lightning\n", "\n", "In this notebook and in many following ones, we will make use of the library [PyTorch Lightning](https://www.pytorchlightning.ai/).\n", "PyTorch Lightning is a framework that simplifies your code needed to train, evaluate, and test a model in PyTorch.\n", "It also handles logging into [TensorBoard](https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html), a visualization toolkit for ML experiments, and saving model checkpoints automatically with minimal code overhead from our side.\n", "This is extremely helpful for us as we want to focus on implementing different model architectures and spend little time on other code overhead.\n", "Note that at the time of writing/teaching, the framework has been released in version 1.3.\n", "Future versions might have a slightly changed interface and thus might not work perfectly with the code (we will try to keep it up-to-date as much as possible).\n", "\n", "Now, we will take the first step in PyTorch Lightning, and continue to explore the framework in our other tutorials.\n", "PyTorch Lightning comes with a lot of useful functions, such as one for setting the seed as we have seen before:"]}, {"cell_type": "code", "execution_count": 9, "id": "15c78406", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:07.407358Z", "iopub.status.busy": "2021-09-16T12:38:07.406885Z", "iopub.status.idle": "2021-09-16T12:38:07.410861Z", "shell.execute_reply": "2021-09-16T12:38:07.410462Z"}, "papermill": {"duration": 0.043657, "end_time": "2021-09-16T12:38:07.410965", "exception": false, "start_time": "2021-09-16T12:38:07.367308", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"data": {"text/plain": ["42"]}, "execution_count": 9, "metadata": {}, "output_type": "execute_result"}], "source": ["# Setting the seed\n", "pl.seed_everything(42)"]}, {"cell_type": "markdown", "id": "3a22dbea", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.037566, "end_time": "2021-09-16T12:38:07.485219", "exception": false, "start_time": "2021-09-16T12:38:07.447653", "status": "completed"}, "tags": []}, "source": ["Thus, in the future, we don't have to define our own `set_seed` function anymore.\n", "\n", "In PyTorch Lightning, we define `pl.LightningModule`'s (inheriting from `torch.nn.Module`) that organize our code into 5 main sections:\n", "\n", "1. Initialization (`__init__`), where we create all necessary parameters/models\n", "2. Optimizers (`configure_optimizers`) where we create the optimizers, learning rate scheduler, etc.\n", "3.\n", "Training loop (`training_step`) where we only have to define the loss calculation for a single batch (the loop of optimizer.zero_grad(), loss.backward() and optimizer.step(), as well as any logging/saving operation, is done in the background)\n", "4.\n", "Validation loop (`validation_step`) where similarly to the training, we only have to define what should happen per step\n", "5. Test loop (`test_step`) which is the same as validation, only on a test set.\n", "\n", "Therefore, we don't abstract the PyTorch code, but rather organize it and define some default operations that are commonly used.\n", "If you need to change something else in your training/validation/test loop, there are many possible functions you can overwrite (see the [docs](https://pytorch-lightning.readthedocs.io/en/stable/lightning_module.html) for details).\n", "\n", "Now we can look at an example of how a Lightning Module for training a CNN looks like:"]}, {"cell_type": "code", "execution_count": 10, "id": "6a9e9ab3", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:07.571798Z", "iopub.status.busy": "2021-09-16T12:38:07.563890Z", "iopub.status.idle": "2021-09-16T12:38:07.573850Z", "shell.execute_reply": "2021-09-16T12:38:07.573331Z"}, "papermill": {"duration": 0.048538, "end_time": "2021-09-16T12:38:07.573956", "exception": false, "start_time": "2021-09-16T12:38:07.525418", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class CIFARModule(pl.LightningModule):\n", "    def __init__(self, model_name, model_hparams, optimizer_name, optimizer_hparams):\n", "        \"\"\"\n", "        Inputs:\n", "            model_name - Name of the model/CNN to run. Used for creating the model (see function below)\n", "            model_hparams - Hyperparameters for the model, as dictionary.\n", "            optimizer_name - Name of the optimizer to use. Currently supported: Adam, SGD\n", "            optimizer_hparams - Hyperparameters for the optimizer, as dictionary. This includes learning rate, weight decay, etc.\n", "        \"\"\"\n", "        super().__init__()\n", "        # Exports the hyperparameters to a YAML file, and create \"self.hparams\" namespace\n", "        self.save_hyperparameters()\n", "        # Create model\n", "        self.model = create_model(model_name, model_hparams)\n", "        # Create loss module\n", "        self.loss_module = nn.CrossEntropyLoss()\n", "        # Example input for visualizing the graph in Tensorboard\n", "        self.example_input_array = torch.zeros((1, 3, 32, 32), dtype=torch.float32)\n", "\n", "    def forward(self, imgs):\n", "        # Forward function that is run when visualizing the graph\n", "        return self.model(imgs)\n", "\n", "    def configure_optimizers(self):\n", "        # We will support Adam or SGD as optimizers.\n", "        if self.hparams.optimizer_name == \"Adam\":\n", "            # AdamW is Adam with a correct implementation of weight decay (see here\n", "            # for details: https://arxiv.org/pdf/1711.05101.pdf)\n", "            optimizer = optim.AdamW(self.parameters(), **self.hparams.optimizer_hparams)\n", "        elif self.hparams.optimizer_name == \"SGD\":\n", "            optimizer = optim.SGD(self.parameters(), **self.hparams.optimizer_hparams)\n", "        else:\n", "            assert False, f'Unknown optimizer: \"{self.hparams.optimizer_name}\"'\n", "\n", "        # We will reduce the learning rate by 0.1 after 100 and 150 epochs\n", "        scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[100, 150], gamma=0.1)\n", "        return [optimizer], [scheduler]\n", "\n", "    def training_step(self, batch, batch_idx):\n", "        # \"batch\" is the output of the training data loader.\n", "        imgs, labels = batch\n", "        preds = self.model(imgs)\n", "        loss = self.loss_module(preds, labels)\n", "        acc = (preds.argmax(dim=-1) == labels).float().mean()\n", "\n", "        # Logs the accuracy per epoch to tensorboard (weighted average over batches)\n", "        self.log(\"train_acc\", acc, on_step=False, on_epoch=True)\n", "        self.log(\"train_loss\", loss)\n", "        return loss  # Return tensor to call \".backward\" on\n", "\n", "    def validation_step(self, batch, batch_idx):\n", "        imgs, labels = batch\n", "        preds = self.model(imgs).argmax(dim=-1)\n", "        acc = (labels == preds).float().mean()\n", "        # By default logs it per epoch (weighted average over batches)\n", "        self.log(\"val_acc\", acc)\n", "\n", "    def test_step(self, batch, batch_idx):\n", "        imgs, labels = batch\n", "        preds = self.model(imgs).argmax(dim=-1)\n", "        acc = (labels == preds).float().mean()\n", "        # By default logs it per epoch (weighted average over batches), and returns it afterwards\n", "        self.log(\"test_acc\", acc)"]}, {"cell_type": "markdown", "id": "072c2427", "metadata": {"papermill": {"duration": 0.036546, "end_time": "2021-09-16T12:38:07.647180", "exception": false, "start_time": "2021-09-16T12:38:07.610634", "status": "completed"}, "tags": []}, "source": ["We see that the code is organized and clear, which helps if someone else tries to understand your code.\n", "\n", "Another important part of PyTorch Lightning is the concept of callbacks.\n", "Callbacks are self-contained functions that contain the non-essential logic of your Lightning Module.\n", "They are usually called after finishing a training epoch, but can also influence other parts of your training loop.\n", "For instance, we will use the following two pre-defined callbacks: `LearningRateMonitor` and `ModelCheckpoint`.\n", "The learning rate monitor adds the current learning rate to our TensorBoard, which helps to verify that our learning rate scheduler works correctly.\n", "The model checkpoint callback allows you to customize the saving routine of your checkpoints.\n", "For instance, how many checkpoints to keep, when to save, which metric to look out for, etc.\n", "We import them below:"]}, {"cell_type": "code", "execution_count": 11, "id": "d10f0a35", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:07.723480Z", "iopub.status.busy": "2021-09-16T12:38:07.722634Z", "iopub.status.idle": "2021-09-16T12:38:07.725280Z", "shell.execute_reply": "2021-09-16T12:38:07.725656Z"}, "papermill": {"duration": 0.042432, "end_time": "2021-09-16T12:38:07.725796", "exception": false, "start_time": "2021-09-16T12:38:07.683364", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Callbacks"]}, {"cell_type": "markdown", "id": "b0f635a3", "metadata": {"papermill": {"duration": 0.037874, "end_time": "2021-09-16T12:38:07.800445", "exception": false, "start_time": "2021-09-16T12:38:07.762571", "status": "completed"}, "tags": []}, "source": ["To allow running multiple different models with the same Lightning module, we define a function below that maps a model name to the model class.\n", "At this stage, the dictionary `model_dict` is empty, but we will fill it throughout the notebook with our new models."]}, {"cell_type": "code", "execution_count": 12, "id": "94546ff5", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:07.879290Z", "iopub.status.busy": "2021-09-16T12:38:07.878824Z", "iopub.status.idle": "2021-09-16T12:38:07.880833Z", "shell.execute_reply": "2021-09-16T12:38:07.880411Z"}, "papermill": {"duration": 0.043613, "end_time": "2021-09-16T12:38:07.880937", "exception": false, "start_time": "2021-09-16T12:38:07.837324", "status": "completed"}, "tags": []}, "outputs": [], "source": ["model_dict = {}\n", "\n", "\n", "def create_model(model_name, model_hparams):\n", "    if model_name in model_dict:\n", "        return model_dict[model_name](**model_hparams)\n", "    else:\n", "        assert False, f'Unknown model name \"{model_name}\". Available models are: {str(model_dict.keys())}'"]}, {"cell_type": "markdown", "id": "52265c22", "metadata": {"papermill": {"duration": 0.038412, "end_time": "2021-09-16T12:38:07.956958", "exception": false, "start_time": "2021-09-16T12:38:07.918546", "status": "completed"}, "tags": []}, "source": ["Similarly, to use the activation function as another hyperparameter in\n", "our model, we define a \"name to function\" dict below:"]}, {"cell_type": "code", "execution_count": 13, "id": "1d0fb575", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:08.041628Z", "iopub.status.busy": "2021-09-16T12:38:08.041166Z", "iopub.status.idle": "2021-09-16T12:38:08.042824Z", "shell.execute_reply": "2021-09-16T12:38:08.043197Z"}, "papermill": {"duration": 0.041698, "end_time": "2021-09-16T12:38:08.043313", "exception": false, "start_time": "2021-09-16T12:38:08.001615", "status": "completed"}, "tags": []}, "outputs": [], "source": ["act_fn_by_name = {\"tanh\": nn.Tanh, \"relu\": nn.ReLU, \"leakyrelu\": nn.LeakyReLU, \"gelu\": nn.GELU}"]}, {"cell_type": "markdown", "id": "1fdb8c00", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.036928, "end_time": "2021-09-16T12:38:08.118848", "exception": false, "start_time": "2021-09-16T12:38:08.081920", "status": "completed"}, "tags": []}, "source": ["If we pass the classes or objects directly as an argument to the Lightning module, we couldn't take advantage of PyTorch Lightning's automatically hyperparameter saving and loading.\n", "\n", "Besides the Lightning module, the second most important module in PyTorch Lightning is the `Trainer`.\n", "The trainer is responsible to execute the training steps defined in the Lightning module and completes the framework.\n", "Similar to the Lightning module, you can override any key part that you don't want to be automated, but the default settings are often the best practice to do.\n", "For a full overview, see the [documentation](https://pytorch-lightning.readthedocs.io/en/stable/trainer.html).\n", "The most important functions we use below are:\n", "\n", "* `trainer.fit`: Takes as input a lightning module, a training dataset, and an (optional) validation dataset.\n", "This function trains the given module on the training dataset with occasional validation (default once per epoch, can be changed)\n", "* `trainer.test`: Takes as input a model and a dataset on which we want to test.\n", "It returns the test metric on the dataset.\n", "\n", "For training and testing, we don't have to worry about things like setting the model to eval mode (`model.eval()`) as this is all done automatically.\n", "See below how we define a training function for our models:"]}, {"cell_type": "code", "execution_count": 14, "id": "a4803d12", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:08.201492Z", "iopub.status.busy": "2021-09-16T12:38:08.200952Z", "iopub.status.idle": "2021-09-16T12:38:08.203083Z", "shell.execute_reply": "2021-09-16T12:38:08.202618Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.048039, "end_time": "2021-09-16T12:38:08.203183", "exception": false, "start_time": "2021-09-16T12:38:08.155144", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def train_model(model_name, save_name=None, **kwargs):\n", "    \"\"\"\n", "    Inputs:\n", "        model_name - Name of the model you want to run. Is used to look up the class in \"model_dict\"\n", "        save_name (optional) - If specified, this name will be used for creating the checkpoint and logging directory.\n", "    \"\"\"\n", "    if save_name is None:\n", "        save_name = model_name\n", "\n", "    # Create a PyTorch Lightning trainer with the generation callback\n", "    trainer = pl.Trainer(\n", "        default_root_dir=os.path.join(CHECKPOINT_PATH, save_name),  # Where to save models\n", "        # We run on a single GPU (if possible)\n", "        gpus=1 if str(device) == \"cuda:0\" else 0,\n", "        # How many epochs to train for if no patience is set\n", "        max_epochs=180,\n", "        callbacks=[\n", "            ModelCheckpoint(\n", "                save_weights_only=True, mode=\"max\", monitor=\"val_acc\"\n", "            ),  # Save the best checkpoint based on the maximum val_acc recorded. Saves only weights and not optimizer\n", "            LearningRateMonitor(\"epoch\"),\n", "        ],  # Log learning rate every epoch\n", "        progress_bar_refresh_rate=1,\n", "    )  # In case your notebook crashes due to the progress bar, consider increasing the refresh rate\n", "    trainer.logger._log_graph = True  # If True, we plot the computation graph in tensorboard\n", "    trainer.logger._default_hp_metric = None  # Optional logging argument that we don't need\n", "\n", "    # Check whether pretrained model exists. If yes, load it and skip training\n", "    pretrained_filename = os.path.join(CHECKPOINT_PATH, save_name + \".ckpt\")\n", "    if os.path.isfile(pretrained_filename):\n", "        print(f\"Found pretrained model at {pretrained_filename}, loading...\")\n", "        # Automatically loads the model with the saved hyperparameters\n", "        model = CIFARModule.load_from_checkpoint(pretrained_filename)\n", "    else:\n", "        pl.seed_everything(42)  # To be reproducable\n", "        model = CIFARModule(model_name=model_name, **kwargs)\n", "        trainer.fit(model, train_loader, val_loader)\n", "        model = CIFARModule.load_from_checkpoint(\n", "            trainer.checkpoint_callback.best_model_path\n", "        )  # Load best checkpoint after training\n", "\n", "    # Test best model on validation and test set\n", "    val_result = trainer.test(model, test_dataloaders=val_loader, verbose=False)\n", "    test_result = trainer.test(model, test_dataloaders=test_loader, verbose=False)\n", "    result = {\"test\": test_result[0][\"test_acc\"], \"val\": val_result[0][\"test_acc\"]}\n", "\n", "    return model, result"]}, {"cell_type": "markdown", "id": "0ea49e7a", "metadata": {"papermill": {"duration": 0.036639, "end_time": "2021-09-16T12:38:08.282179", "exception": false, "start_time": "2021-09-16T12:38:08.245540", "status": "completed"}, "tags": []}, "source": ["Finally, we can focus on the Convolutional Neural Networks we want to\n", "implement today: GoogleNet, ResNet, and DenseNet."]}, {"cell_type": "markdown", "id": "c1c753ce", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.038185, "end_time": "2021-09-16T12:38:08.357713", "exception": false, "start_time": "2021-09-16T12:38:08.319528", "status": "completed"}, "tags": []}, "source": ["## Inception\n", "\n", "<div class=\"center-wrapper\"><div class=\"video-wrapper\"><iframe src=\"https://www.youtube.com/embed/9yRXqYJDHr4\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></div></div>\n", "\n", "The [GoogleNet](https://arxiv.org/abs/1409.4842), proposed in 2014, won the ImageNet Challenge because of its usage of the Inception modules.\n", "In general, we will mainly focus on the concept of Inception in this tutorial instead of the specifics of the GoogleNet, as based on Inception, there have been many follow-up works ([Inception-v2](https://arxiv.org/abs/1512.00567), [Inception-v3](https://arxiv.org/abs/1512.00567), [Inception-v4](https://arxiv.org/abs/1602.07261), [Inception-ResNet](https://arxiv.org/abs/1602.07261),...).\n", "The follow-up works mainly focus on increasing efficiency and enabling very deep Inception networks.\n", "However, for a fundamental understanding, it is sufficient to look at the original Inception block.\n", "\n", "An Inception block applies four convolution blocks separately on the same feature map: a 1x1, 3x3, and 5x5 convolution, and a max pool operation.\n", "This allows the network to look at the same data with different receptive fields.\n", "Of course, learning only 5x5 convolution would be theoretically more powerful.\n", "However, this is not only more computation and memory heavy but also tends to overfit much easier.\n", "The overall inception block looks like below (figure credit - [Szegedy et al. ](https://arxiv.org/abs/1409.4842)):\n", "\n", "<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/inception_block.svg\" style=\"display: block; margin-left: auto; margin-right: auto;\" width=\"500px\"/></center>\n", "\n", "The additional 1x1 convolutions before the 3x3 and 5x5 convolutions are used for dimensionality reduction.\n", "This is especially crucial as the feature maps of all branches are merged afterward, and we don't want any explosion of feature size.\n", "As 5x5 convolutions are 25 times more expensive than 1x1 convolutions, we can save a lot of computation and parameters by reducing the dimensionality before the large convolutions.\n", "\n", "We can now try to implement the Inception Block ourselves:"]}, {"cell_type": "code", "execution_count": 15, "id": "d37d1409", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:08.440831Z", "iopub.status.busy": "2021-09-16T12:38:08.440351Z", "iopub.status.idle": "2021-09-16T12:38:08.442719Z", "shell.execute_reply": "2021-09-16T12:38:08.442257Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.046514, "end_time": "2021-09-16T12:38:08.442822", "exception": false, "start_time": "2021-09-16T12:38:08.396308", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class InceptionBlock(nn.Module):\n", "    def __init__(self, c_in, c_red: dict, c_out: dict, act_fn):\n", "        \"\"\"\n", "        Inputs:\n", "            c_in - Number of input feature maps from the previous layers\n", "            c_red - Dictionary with keys \"3x3\" and \"5x5\" specifying the output of the dimensionality reducing 1x1 convolutions\n", "            c_out - Dictionary with keys \"1x1\", \"3x3\", \"5x5\", and \"max\"\n", "            act_fn - Activation class constructor (e.g. nn.ReLU)\n", "        \"\"\"\n", "        super().__init__()\n", "\n", "        # 1x1 convolution branch\n", "        self.conv_1x1 = nn.Sequential(\n", "            nn.Conv2d(c_in, c_out[\"1x1\"], kernel_size=1), nn.BatchNorm2d(c_out[\"1x1\"]), act_fn()\n", "        )\n", "\n", "        # 3x3 convolution branch\n", "        self.conv_3x3 = nn.Sequential(\n", "            nn.Conv2d(c_in, c_red[\"3x3\"], kernel_size=1),\n", "            nn.BatchNorm2d(c_red[\"3x3\"]),\n", "            act_fn(),\n", "            nn.Conv2d(c_red[\"3x3\"], c_out[\"3x3\"], kernel_size=3, padding=1),\n", "            nn.BatchNorm2d(c_out[\"3x3\"]),\n", "            act_fn(),\n", "        )\n", "\n", "        # 5x5 convolution branch\n", "        self.conv_5x5 = nn.Sequential(\n", "            nn.Conv2d(c_in, c_red[\"5x5\"], kernel_size=1),\n", "            nn.BatchNorm2d(c_red[\"5x5\"]),\n", "            act_fn(),\n", "            nn.Conv2d(c_red[\"5x5\"], c_out[\"5x5\"], kernel_size=5, padding=2),\n", "            nn.BatchNorm2d(c_out[\"5x5\"]),\n", "            act_fn(),\n", "        )\n", "\n", "        # Max-pool branch\n", "        self.max_pool = nn.Sequential(\n", "            nn.MaxPool2d(kernel_size=3, padding=1, stride=1),\n", "            nn.Conv2d(c_in, c_out[\"max\"], kernel_size=1),\n", "            nn.BatchNorm2d(c_out[\"max\"]),\n", "            act_fn(),\n", "        )\n", "\n", "    def forward(self, x):\n", "        x_1x1 = self.conv_1x1(x)\n", "        x_3x3 = self.conv_3x3(x)\n", "        x_5x5 = self.conv_5x5(x)\n", "        x_max = self.max_pool(x)\n", "        x_out = torch.cat([x_1x1, x_3x3, x_5x5, x_max], dim=1)\n", "        return x_out"]}, {"cell_type": "markdown", "id": "9c98368e", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.040994, "end_time": "2021-09-16T12:38:08.520437", "exception": false, "start_time": "2021-09-16T12:38:08.479443", "status": "completed"}, "tags": []}, "source": ["The GoogleNet architecture consists of stacking multiple Inception blocks with occasional max pooling to reduce the height and width of the feature maps.\n", "The original GoogleNet was designed for image sizes of ImageNet (224x224 pixels) and had almost 7 million parameters.\n", "As we train on CIFAR10 with image sizes of 32x32, we don't require such a heavy architecture, and instead, apply a reduced version.\n", "The number of channels for dimensionality reduction and output per filter (1x1, 3x3, 5x5, and max pooling) need to be manually specified and can be changed if interested.\n", "The general intuition is to have the most filters for the 3x3\n", "convolutions, as they are powerful enough to take the context into\n", "account while requiring almost a third of the parameters of the 5x5\n", "convolution."]}, {"cell_type": "code", "execution_count": 16, "id": "97701a20", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:08.621403Z", "iopub.status.busy": "2021-09-16T12:38:08.611472Z", "iopub.status.idle": "2021-09-16T12:38:08.623000Z", "shell.execute_reply": "2021-09-16T12:38:08.623380Z"}, "papermill": {"duration": 0.058103, "end_time": "2021-09-16T12:38:08.623497", "exception": false, "start_time": "2021-09-16T12:38:08.565394", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class GoogleNet(nn.Module):\n", "    def __init__(self, num_classes=10, act_fn_name=\"relu\", **kwargs):\n", "        super().__init__()\n", "        self.hparams = SimpleNamespace(\n", "            num_classes=num_classes, act_fn_name=act_fn_name, act_fn=act_fn_by_name[act_fn_name]\n", "        )\n", "        self._create_network()\n", "        self._init_params()\n", "\n", "    def _create_network(self):\n", "        # A first convolution on the original image to scale up the channel size\n", "        self.input_net = nn.Sequential(\n", "            nn.Conv2d(3, 64, kernel_size=3, padding=1), nn.BatchNorm2d(64), self.hparams.act_fn()\n", "        )\n", "        # Stacking inception blocks\n", "        self.inception_blocks = nn.Sequential(\n", "            InceptionBlock(\n", "                64,\n", "                c_red={\"3x3\": 32, \"5x5\": 16},\n", "                c_out={\"1x1\": 16, \"3x3\": 32, \"5x5\": 8, \"max\": 8},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "            InceptionBlock(\n", "                64,\n", "                c_red={\"3x3\": 32, \"5x5\": 16},\n", "                c_out={\"1x1\": 24, \"3x3\": 48, \"5x5\": 12, \"max\": 12},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "            nn.MaxPool2d(3, stride=2, padding=1),  # 32x32 => 16x16\n", "            InceptionBlock(\n", "                96,\n", "                c_red={\"3x3\": 32, \"5x5\": 16},\n", "                c_out={\"1x1\": 24, \"3x3\": 48, \"5x5\": 12, \"max\": 12},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "            InceptionBlock(\n", "                96,\n", "                c_red={\"3x3\": 32, \"5x5\": 16},\n", "                c_out={\"1x1\": 16, \"3x3\": 48, \"5x5\": 16, \"max\": 16},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "            InceptionBlock(\n", "                96,\n", "                c_red={\"3x3\": 32, \"5x5\": 16},\n", "                c_out={\"1x1\": 16, \"3x3\": 48, \"5x5\": 16, \"max\": 16},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "            InceptionBlock(\n", "                96,\n", "                c_red={\"3x3\": 32, \"5x5\": 16},\n", "                c_out={\"1x1\": 32, \"3x3\": 48, \"5x5\": 24, \"max\": 24},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "            nn.MaxPool2d(3, stride=2, padding=1),  # 16x16 => 8x8\n", "            InceptionBlock(\n", "                128,\n", "                c_red={\"3x3\": 48, \"5x5\": 16},\n", "                c_out={\"1x1\": 32, \"3x3\": 64, \"5x5\": 16, \"max\": 16},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "            InceptionBlock(\n", "                128,\n", "                c_red={\"3x3\": 48, \"5x5\": 16},\n", "                c_out={\"1x1\": 32, \"3x3\": 64, \"5x5\": 16, \"max\": 16},\n", "                act_fn=self.hparams.act_fn,\n", "            ),\n", "        )\n", "        # Mapping to classification output\n", "        self.output_net = nn.Sequential(\n", "            nn.AdaptiveAvgPool2d((1, 1)), nn.Flatten(), nn.Linear(128, self.hparams.num_classes)\n", "        )\n", "\n", "    def _init_params(self):\n", "        # Based on our discussion in Tutorial 4, we should initialize the\n", "        # convolutions according to the activation function\n", "        for m in self.modules():\n", "            if isinstance(m, nn.Conv2d):\n", "                nn.init.kaiming_normal_(m.weight, nonlinearity=self.hparams.act_fn_name)\n", "            elif isinstance(m, nn.BatchNorm2d):\n", "                nn.init.constant_(m.weight, 1)\n", "                nn.init.constant_(m.bias, 0)\n", "\n", "    def forward(self, x):\n", "        x = self.input_net(x)\n", "        x = self.inception_blocks(x)\n", "        x = self.output_net(x)\n", "        return x"]}, {"cell_type": "markdown", "id": "6f8e905b", "metadata": {"papermill": {"duration": 0.039125, "end_time": "2021-09-16T12:38:08.699847", "exception": false, "start_time": "2021-09-16T12:38:08.660722", "status": "completed"}, "tags": []}, "source": ["Now, we can integrate our model to the model dictionary we defined above:"]}, {"cell_type": "code", "execution_count": 17, "id": "4b1f373a", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:08.776825Z", "iopub.status.busy": "2021-09-16T12:38:08.776344Z", "iopub.status.idle": "2021-09-16T12:38:08.778452Z", "shell.execute_reply": "2021-09-16T12:38:08.778057Z"}, "papermill": {"duration": 0.042016, "end_time": "2021-09-16T12:38:08.778552", "exception": false, "start_time": "2021-09-16T12:38:08.736536", "status": "completed"}, "tags": []}, "outputs": [], "source": ["model_dict[\"GoogleNet\"] = GoogleNet"]}, {"cell_type": "markdown", "id": "c5d1a0df", "metadata": {"papermill": {"duration": 0.036756, "end_time": "2021-09-16T12:38:08.851721", "exception": false, "start_time": "2021-09-16T12:38:08.814965", "status": "completed"}, "tags": []}, "source": ["The training of the model is handled by PyTorch Lightning, and we just have to define the command to start.\n", "Note that we train for almost 200 epochs, which takes about an hour on Lisa's default GPUs (GTX1080Ti).\n", "We would recommend using the saved models and train your own model if you are interested."]}, {"cell_type": "code", "execution_count": 18, "id": "8fd80188", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:08.931165Z", "iopub.status.busy": "2021-09-16T12:38:08.930697Z", "iopub.status.idle": "2021-09-16T12:38:13.663153Z", "shell.execute_reply": "2021-09-16T12:38:13.663545Z"}, "papermill": {"duration": 4.775246, "end_time": "2021-09-16T12:38:13.663690", "exception": false, "start_time": "2021-09-16T12:38:08.888444", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["/home/AzDevOps_azpcontainer/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py:678: LightningDeprecationWarning: `trainer.test(test_dataloaders)` is deprecated in v1.4 and will be removed in v1.6. Use `trainer.test(dataloaders)` instead.\n", "  rank_zero_deprecation(\n", "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torch/_jit_internal.py:603: LightningDeprecationWarning: The `LightningModule.datamodule` property is deprecated in v1.3 and will be removed in v1.5. Access the datamodule through using `self.trainer.datamodule` instead.\n", "  if hasattr(mod, name):\n", "/usr/local/lib/python3.9/dist-packages/torch/_jit_internal.py:603: LightningDeprecationWarning: The `LightningModule.loaded_optimizer_states_dict` property is deprecated in v1.4 and will be removed in v1.6.\n", "  if hasattr(mod, name):\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model at saved_models/ConvNets/GoogleNet.ckpt, loading...\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "7ece9c5c517e4dfe9131a5a78db598a5", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "a03e388e9a8542fdb879274e69b9bbe4", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}], "source": ["googlenet_model, googlenet_results = train_model(\n", "    model_name=\"GoogleNet\",\n", "    model_hparams={\"num_classes\": 10, \"act_fn_name\": \"relu\"},\n", "    optimizer_name=\"Adam\",\n", "    optimizer_hparams={\"lr\": 1e-3, \"weight_decay\": 1e-4},\n", ")"]}, {"cell_type": "markdown", "id": "bfa4d89e", "metadata": {"papermill": {"duration": 0.039461, "end_time": "2021-09-16T12:38:13.743264", "exception": false, "start_time": "2021-09-16T12:38:13.703803", "status": "completed"}, "tags": []}, "source": ["We will compare the results later in the notebooks, but we can already print them here for a first glance:"]}, {"cell_type": "code", "execution_count": 19, "id": "bef04499", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:13.827147Z", "iopub.status.busy": "2021-09-16T12:38:13.826608Z", "iopub.status.idle": "2021-09-16T12:38:13.829247Z", "shell.execute_reply": "2021-09-16T12:38:13.828807Z"}, "papermill": {"duration": 0.046582, "end_time": "2021-09-16T12:38:13.829351", "exception": false, "start_time": "2021-09-16T12:38:13.782769", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["GoogleNet Results {'test': 0.8970000147819519, 'val': 0.9039999842643738}\n"]}], "source": ["print(\"GoogleNet Results\", googlenet_results)"]}, {"cell_type": "markdown", "id": "61c08e31", "metadata": {"papermill": {"duration": 0.039731, "end_time": "2021-09-16T12:38:13.908262", "exception": false, "start_time": "2021-09-16T12:38:13.868531", "status": "completed"}, "tags": []}, "source": ["### Tensorboard log\n", "\n", "A nice extra of PyTorch Lightning is the automatic logging into TensorBoard.\n", "To give you a better intuition of what TensorBoard can be used, we can look at the board that PyTorch Lightning has been generated when training the GoogleNet.\n", "TensorBoard provides an inline functionality for Jupyter notebooks, and we use it here:"]}, {"cell_type": "code", "execution_count": 20, "id": "7784f24f", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:13.991276Z", "iopub.status.busy": "2021-09-16T12:38:13.990813Z", "iopub.status.idle": "2021-09-16T12:38:13.992872Z", "shell.execute_reply": "2021-09-16T12:38:13.992472Z"}, "papermill": {"duration": 0.043809, "end_time": "2021-09-16T12:38:13.992971", "exception": false, "start_time": "2021-09-16T12:38:13.949162", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Import tensorboard\n", "# %load_ext tensorboard"]}, {"cell_type": "code", "execution_count": 21, "id": "c42a23d6", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:14.074171Z", "iopub.status.busy": "2021-09-16T12:38:14.073678Z", "iopub.status.idle": "2021-09-16T12:38:14.075314Z", "shell.execute_reply": "2021-09-16T12:38:14.075688Z"}, "papermill": {"duration": 0.043657, "end_time": "2021-09-16T12:38:14.075800", "exception": false, "start_time": "2021-09-16T12:38:14.032143", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Opens tensorboard in notebook. Adjust the path to your CHECKPOINT_PATH!\n", "# %tensorboard --logdir ../saved_models/tutorial5/tensorboards/GoogleNet/"]}, {"cell_type": "markdown", "id": "9f88a3e4", "metadata": {"papermill": {"duration": 0.039853, "end_time": "2021-09-16T12:38:14.155403", "exception": false, "start_time": "2021-09-16T12:38:14.115550", "status": "completed"}, "tags": []}, "source": ["<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/tensorboard_screenshot_GoogleNet.png\" width=\"1000px\"></center>\n", "\n", "TensorBoard is organized in multiple tabs.\n", "The main tab is the scalar tab where we can log the development of single numbers.\n", "For example, we have plotted the training loss, accuracy, learning rate, etc.\n", "If we look at the training or validation accuracy, we can really see the impact of using a learning rate scheduler.\n", "Reducing the learning rate gives our model a nice increase in training performance.\n", "Similarly, when looking at the training loss, we see a sudden decrease at this point.\n", "However, the high numbers on the training set compared to validation indicate that our model was overfitting which is inevitable for such large networks.\n", "\n", "Another interesting tab in TensorBoard is the graph tab.\n", "It shows us the network architecture organized by building blocks from the input to the output.\n", "It basically shows the operations taken in the forward step of `CIFARModule`.\n", "Double-click on a module to open it.\n", "Feel free to explore the architecture from a different perspective.\n", "The graph visualization can often help you to validate that your model\n", "is actually doing what it is supposed to do, and you don't miss any\n", "layers in the computation graph."]}, {"cell_type": "markdown", "id": "21cc4562", "metadata": {"papermill": {"duration": 0.039401, "end_time": "2021-09-16T12:38:14.234907", "exception": false, "start_time": "2021-09-16T12:38:14.195506", "status": "completed"}, "tags": []}, "source": ["## ResNet\n", "\n", "The [ResNet](https://arxiv.org/abs/1512.03385) paper is one of the  [most cited AI papers](https://www.natureindex.com/news-blog/google-scholar-reveals-most-influential-papers-research-citations-twenty-twenty), and has been the foundation for neural networks with more than 1,000 layers.\n", "Despite its simplicity, the idea of residual connections is highly effective as it supports stable gradient propagation through the network.\n", "Instead of modeling $x_{l+1}=F(x_{l})$, we model $x_{l+1}=x_{l}+F(x_{l})$ where $F$ is a non-linear mapping (usually a sequence of NN modules likes convolutions, activation functions, and normalizations).\n", "If we do backpropagation on such residual connections, we obtain:\n", "\n", "$$\\frac{\\partial x_{l+1}}{\\partial x_{l}} = \\mathbf{I} + \\frac{\\partial F(x_{l})}{\\partial x_{l}}$$\n", "\n", "The bias towards the identity matrix guarantees a stable gradient propagation being less effected by $F$ itself.\n", "There have been many variants of ResNet proposed, which mostly concern the function $F$, or operations applied on the sum.\n", "In this tutorial, we look at two of them: the original ResNet block, and the [Pre-Activation ResNet block](https://arxiv.org/abs/1603.05027).\n", "We visually compare the blocks below (figure credit - [He et al. ](https://arxiv.org/abs/1603.05027)):\n", "\n", "<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/resnet_block.svg\" style=\"display: block; margin-left: auto; margin-right: auto;\" width=\"300px\"/></center>\n", "\n", "The original ResNet block applies a non-linear activation function, usually ReLU, after the skip connection.\n", "In contrast, the pre-activation ResNet block applies the non-linearity at the beginning of $F$.\n", "Both have their advantages and disadvantages.\n", "For very deep network, however, the pre-activation ResNet has shown to perform better as the gradient flow is guaranteed to have the identity matrix as calculated above, and is not harmed by any non-linear activation applied to it.\n", "For comparison, in this notebook, we implement both ResNet types as shallow networks.\n", "\n", "Let's start with the original ResNet block.\n", "The visualization above already shows what layers are included in $F$.\n", "One special case we have to handle is when we want to reduce the image dimensions in terms of width and height.\n", "The basic ResNet block requires $F(x_{l})$ to be of the same shape as $x_{l}$.\n", "Thus, we need to change the dimensionality of $x_{l}$ as well before adding to $F(x_{l})$.\n", "The original implementation used an identity mapping with stride 2 and padded additional feature dimensions with 0.\n", "However, the more common implementation is to use a 1x1 convolution with stride 2 as it allows us to change the feature dimensionality while being efficient in parameter and computation cost.\n", "The code for the ResNet block is relatively simple, and shown below:"]}, {"cell_type": "code", "execution_count": 22, "id": "b65250fe", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:14.320751Z", "iopub.status.busy": "2021-09-16T12:38:14.320273Z", "iopub.status.idle": "2021-09-16T12:38:14.322327Z", "shell.execute_reply": "2021-09-16T12:38:14.321920Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.047958, "end_time": "2021-09-16T12:38:14.322431", "exception": false, "start_time": "2021-09-16T12:38:14.274473", "status": "completed"}, "tags": []}, "outputs": [], "source": ["\n", "\n", "class ResNetBlock(nn.Module):\n", "    def __init__(self, c_in, act_fn, subsample=False, c_out=-1):\n", "        \"\"\"\n", "        Inputs:\n", "            c_in - Number of input features\n", "            act_fn - Activation class constructor (e.g. nn.ReLU)\n", "            subsample - If True, we want to apply a stride inside the block and reduce the output shape by 2 in height and width\n", "            c_out - Number of output features. Note that this is only relevant if subsample is True, as otherwise, c_out = c_in\n", "        \"\"\"\n", "        super().__init__()\n", "        if not subsample:\n", "            c_out = c_in\n", "\n", "        # Network representing F\n", "        self.net = nn.Sequential(\n", "            nn.Conv2d(\n", "                c_in, c_out, kernel_size=3, padding=1, stride=1 if not subsample else 2, bias=False\n", "            ),  # No bias needed as the Batch Norm handles it\n", "            nn.BatchNorm2d(c_out),\n", "            act_fn(),\n", "            nn.Conv2d(c_out, c_out, kernel_size=3, padding=1, bias=False),\n", "            nn.BatchNorm2d(c_out),\n", "        )\n", "\n", "        # 1x1 convolution with stride 2 means we take the upper left value, and transform it to new output size\n", "        self.downsample = nn.Conv2d(c_in, c_out, kernel_size=1, stride=2) if subsample else None\n", "        self.act_fn = act_fn()\n", "\n", "    def forward(self, x):\n", "        z = self.net(x)\n", "        if self.downsample is not None:\n", "            x = self.downsample(x)\n", "        out = z + x\n", "        out = self.act_fn(out)\n", "        return out"]}, {"cell_type": "markdown", "id": "ed4bc78f", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.040026, "end_time": "2021-09-16T12:38:14.404277", "exception": false, "start_time": "2021-09-16T12:38:14.364251", "status": "completed"}, "tags": []}, "source": ["The second block we implement is the pre-activation ResNet block.\n", "For this, we have to change the order of layer in `self.net`, and do not apply an activation function on the output.\n", "Additionally, the downsampling operation has to apply a non-linearity as well as the input, $x_l$, has not been processed by a non-linearity yet.\n", "Hence, the block looks as follows:"]}, {"cell_type": "code", "execution_count": 23, "id": "2baaee82", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:14.492667Z", "iopub.status.busy": "2021-09-16T12:38:14.492187Z", "iopub.status.idle": "2021-09-16T12:38:14.493873Z", "shell.execute_reply": "2021-09-16T12:38:14.494251Z"}, "papermill": {"duration": 0.048396, "end_time": "2021-09-16T12:38:14.494372", "exception": false, "start_time": "2021-09-16T12:38:14.445976", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class PreActResNetBlock(nn.Module):\n", "    def __init__(self, c_in, act_fn, subsample=False, c_out=-1):\n", "        \"\"\"\n", "        Inputs:\n", "            c_in - Number of input features\n", "            act_fn - Activation class constructor (e.g. nn.ReLU)\n", "            subsample - If True, we want to apply a stride inside the block and reduce the output shape by 2 in height and width\n", "            c_out - Number of output features. Note that this is only relevant if subsample is True, as otherwise, c_out = c_in\n", "        \"\"\"\n", "        super().__init__()\n", "        if not subsample:\n", "            c_out = c_in\n", "\n", "        # Network representing F\n", "        self.net = nn.Sequential(\n", "            nn.BatchNorm2d(c_in),\n", "            act_fn(),\n", "            nn.Conv2d(c_in, c_out, kernel_size=3, padding=1, stride=1 if not subsample else 2, bias=False),\n", "            nn.BatchNorm2d(c_out),\n", "            act_fn(),\n", "            nn.Conv2d(c_out, c_out, kernel_size=3, padding=1, bias=False),\n", "        )\n", "\n", "        # 1x1 convolution needs to apply non-linearity as well as not done on skip connection\n", "        self.downsample = (\n", "            nn.Sequential(nn.BatchNorm2d(c_in), act_fn(), nn.Conv2d(c_in, c_out, kernel_size=1, stride=2, bias=False))\n", "            if subsample\n", "            else None\n", "        )\n", "\n", "    def forward(self, x):\n", "        z = self.net(x)\n", "        if self.downsample is not None:\n", "            x = self.downsample(x)\n", "        out = z + x\n", "        return out"]}, {"cell_type": "markdown", "id": "b35e108e", "metadata": {"papermill": {"duration": 0.039781, "end_time": "2021-09-16T12:38:14.573781", "exception": false, "start_time": "2021-09-16T12:38:14.534000", "status": "completed"}, "tags": []}, "source": ["Similarly to the model selection, we define a dictionary to create a mapping from string to block class.\n", "We will use the string name as hyperparameter value in our model to choose between the ResNet blocks.\n", "Feel free to implement any other ResNet block type and add it here as well."]}, {"cell_type": "code", "execution_count": 24, "id": "14220bec", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:14.656954Z", "iopub.status.busy": "2021-09-16T12:38:14.656486Z", "iopub.status.idle": "2021-09-16T12:38:14.658586Z", "shell.execute_reply": "2021-09-16T12:38:14.658188Z"}, "papermill": {"duration": 0.044864, "end_time": "2021-09-16T12:38:14.658692", "exception": false, "start_time": "2021-09-16T12:38:14.613828", "status": "completed"}, "tags": []}, "outputs": [], "source": ["resnet_blocks_by_name = {\"ResNetBlock\": ResNetBlock, \"PreActResNetBlock\": PreActResNetBlock}"]}, {"cell_type": "markdown", "id": "cda408f9", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.039519, "end_time": "2021-09-16T12:38:14.738251", "exception": false, "start_time": "2021-09-16T12:38:14.698732", "status": "completed"}, "tags": []}, "source": ["The overall ResNet architecture consists of stacking multiple ResNet blocks, of which some are downsampling the input.\n", "When talking about ResNet blocks in the whole network, we usually group them by the same output shape.\n", "Hence, if we say the ResNet has `[3,3,3]` blocks, it means that we have 3 times a group of 3 ResNet blocks, where a subsampling is taking place in the fourth and seventh block.\n", "The ResNet with `[3,3,3]` blocks on CIFAR10 is visualized below.\n", "\n", "<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/resnet_notation.svg\" width=\"500px\"></center>\n", "\n", "The three groups operate on the resolutions $32\\times32$, $16\\times16$ and $8\\times8$ respectively.\n", "The blocks in orange denote ResNet blocks with downsampling.\n", "The same notation is used by many other implementations such as in the [torchvision library](https://pytorch.org/docs/stable/_modules/torchvision/models/resnet.html#resnet18) from PyTorch.\n", "Thus, our code looks as follows:"]}, {"cell_type": "code", "execution_count": 25, "id": "fe13b373", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:14.828391Z", "iopub.status.busy": "2021-09-16T12:38:14.825437Z", "iopub.status.idle": "2021-09-16T12:38:14.830444Z", "shell.execute_reply": "2021-09-16T12:38:14.830005Z"}, "papermill": {"duration": 0.052672, "end_time": "2021-09-16T12:38:14.830553", "exception": false, "start_time": "2021-09-16T12:38:14.777881", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ResNet(nn.Module):\n", "    def __init__(\n", "        self,\n", "        num_classes=10,\n", "        num_blocks=[3, 3, 3],\n", "        c_hidden=[16, 32, 64],\n", "        act_fn_name=\"relu\",\n", "        block_name=\"ResNetBlock\",\n", "        **kwargs,\n", "    ):\n", "        \"\"\"\n", "        Inputs:\n", "            num_classes - Number of classification outputs (10 for CIFAR10)\n", "            num_blocks - List with the number of ResNet blocks to use. The first block of each group uses downsampling, except the first.\n", "            c_hidden - List with the hidden dimensionalities in the different blocks. Usually multiplied by 2 the deeper we go.\n", "            act_fn_name - Name of the activation function to use, looked up in \"act_fn_by_name\"\n", "            block_name - Name of the ResNet block, looked up in \"resnet_blocks_by_name\"\n", "        \"\"\"\n", "        super().__init__()\n", "        assert block_name in resnet_blocks_by_name\n", "        self.hparams = SimpleNamespace(\n", "            num_classes=num_classes,\n", "            c_hidden=c_hidden,\n", "            num_blocks=num_blocks,\n", "            act_fn_name=act_fn_name,\n", "            act_fn=act_fn_by_name[act_fn_name],\n", "            block_class=resnet_blocks_by_name[block_name],\n", "        )\n", "        self._create_network()\n", "        self._init_params()\n", "\n", "    def _create_network(self):\n", "        c_hidden = self.hparams.c_hidden\n", "\n", "        # A first convolution on the original image to scale up the channel size\n", "        if self.hparams.block_class == PreActResNetBlock:  # => Don't apply non-linearity on output\n", "            self.input_net = nn.Sequential(nn.Conv2d(3, c_hidden[0], kernel_size=3, padding=1, bias=False))\n", "        else:\n", "            self.input_net = nn.Sequential(\n", "                nn.Conv2d(3, c_hidden[0], kernel_size=3, padding=1, bias=False),\n", "                nn.BatchNorm2d(c_hidden[0]),\n", "                self.hparams.act_fn(),\n", "            )\n", "\n", "        # Creating the ResNet blocks\n", "        blocks = []\n", "        for block_idx, block_count in enumerate(self.hparams.num_blocks):\n", "            for bc in range(block_count):\n", "                # Subsample the first block of each group, except the very first one.\n", "                subsample = bc == 0 and block_idx > 0\n", "                blocks.append(\n", "                    self.hparams.block_class(\n", "                        c_in=c_hidden[block_idx if not subsample else (block_idx - 1)],\n", "                        act_fn=self.hparams.act_fn,\n", "                        subsample=subsample,\n", "                        c_out=c_hidden[block_idx],\n", "                    )\n", "                )\n", "        self.blocks = nn.Sequential(*blocks)\n", "\n", "        # Mapping to classification output\n", "        self.output_net = nn.Sequential(\n", "            nn.AdaptiveAvgPool2d((1, 1)), nn.Flatten(), nn.Linear(c_hidden[-1], self.hparams.num_classes)\n", "        )\n", "\n", "    def _init_params(self):\n", "        # Based on our discussion in Tutorial 4, we should initialize the convolutions according to the activation function\n", "        # Fan-out focuses on the gradient distribution, and is commonly used in ResNets\n", "        for m in self.modules():\n", "            if isinstance(m, nn.Conv2d):\n", "                nn.init.kaiming_normal_(m.weight, mode=\"fan_out\", nonlinearity=self.hparams.act_fn_name)\n", "            elif isinstance(m, nn.BatchNorm2d):\n", "                nn.init.constant_(m.weight, 1)\n", "                nn.init.constant_(m.bias, 0)\n", "\n", "    def forward(self, x):\n", "        x = self.input_net(x)\n", "        x = self.blocks(x)\n", "        x = self.output_net(x)\n", "        return x"]}, {"cell_type": "markdown", "id": "9476ca46", "metadata": {"papermill": {"duration": 0.039784, "end_time": "2021-09-16T12:38:14.909939", "exception": false, "start_time": "2021-09-16T12:38:14.870155", "status": "completed"}, "tags": []}, "source": ["We also need to add the new ResNet class to our model dictionary:"]}, {"cell_type": "code", "execution_count": 26, "id": "8880ae3b", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:14.993955Z", "iopub.status.busy": "2021-09-16T12:38:14.993453Z", "iopub.status.idle": "2021-09-16T12:38:14.995456Z", "shell.execute_reply": "2021-09-16T12:38:14.994995Z"}, "papermill": {"duration": 0.044259, "end_time": "2021-09-16T12:38:14.995555", "exception": false, "start_time": "2021-09-16T12:38:14.951296", "status": "completed"}, "tags": []}, "outputs": [], "source": ["model_dict[\"ResNet\"] = ResNet"]}, {"cell_type": "markdown", "id": "aa0196ed", "metadata": {"papermill": {"duration": 0.039836, "end_time": "2021-09-16T12:38:15.075242", "exception": false, "start_time": "2021-09-16T12:38:15.035406", "status": "completed"}, "tags": []}, "source": ["Finally, we can train our ResNet models.\n", "One difference to the GoogleNet training is that we explicitly use SGD with Momentum as optimizer instead of Adam.\n", "Adam often leads to a slightly worse accuracy on plain, shallow ResNets.\n", "It is not 100% clear why Adam performs worse in this context, but one possible explanation is related to ResNet's loss surface.\n", "ResNet has been shown to produce smoother loss surfaces than networks without skip connection (see [Li et al., 2018](https://arxiv.org/pdf/1712.09913.pdf) for details).\n", "A possible visualization of the loss surface with/out skip connections is below (figure credit - [Li et al. ](https://arxiv.org/pdf/1712.09913.pdf)):\n", "\n", "<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/resnet_loss_surface.png\" style=\"display: block; margin-left: auto; margin-right: auto;\" width=\"600px\"/></center>\n", "\n", "The $x$ and $y$ axis shows a projection of the parameter space, and the $z$ axis shows the loss values achieved by different parameter values.\n", "On smooth surfaces like the one on the right, we might not require an adaptive learning rate as Adam provides.\n", "Instead, Adam can get stuck in local optima while SGD finds the wider minima that tend to generalize better.\n", "However, to answer this question in detail, we would need an extra tutorial because it is not easy to answer.\n", "For now, we conclude: for ResNet architectures, consider the optimizer to be an important hyperparameter, and try training with both Adam and SGD.\n", "Let's train the model below with SGD:"]}, {"cell_type": "code", "execution_count": 27, "id": "2397c99d", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:15.160927Z", "iopub.status.busy": "2021-09-16T12:38:15.160458Z", "iopub.status.idle": "2021-09-16T12:38:17.369055Z", "shell.execute_reply": "2021-09-16T12:38:17.368634Z"}, "papermill": {"duration": 2.253669, "end_time": "2021-09-16T12:38:17.369170", "exception": false, "start_time": "2021-09-16T12:38:15.115501", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model at saved_models/ConvNets/ResNet.ckpt, loading...\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "208111d43b2547a599fdcd62ee551ad4", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "33eb164d6f8446399823b259486ff7ba", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}], "source": ["resnet_model, resnet_results = train_model(\n", "    model_name=\"ResNet\",\n", "    model_hparams={\"num_classes\": 10, \"c_hidden\": [16, 32, 64], \"num_blocks\": [3, 3, 3], \"act_fn_name\": \"relu\"},\n", "    optimizer_name=\"SGD\",\n", "    optimizer_hparams={\"lr\": 0.1, \"momentum\": 0.9, \"weight_decay\": 1e-4},\n", ")"]}, {"cell_type": "markdown", "id": "ee817a2a", "metadata": {"papermill": {"duration": 0.042771, "end_time": "2021-09-16T12:38:17.455319", "exception": false, "start_time": "2021-09-16T12:38:17.412548", "status": "completed"}, "tags": []}, "source": ["Let's also train the pre-activation ResNet as comparison:"]}, {"cell_type": "code", "execution_count": 28, "id": "9210f4ff", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:17.551345Z", "iopub.status.busy": "2021-09-16T12:38:17.545018Z", "iopub.status.idle": "2021-09-16T12:38:19.848678Z", "shell.execute_reply": "2021-09-16T12:38:19.849190Z"}, "papermill": {"duration": 2.350494, "end_time": "2021-09-16T12:38:19.849340", "exception": false, "start_time": "2021-09-16T12:38:17.498846", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model at saved_models/ConvNets/ResNetPreAct.ckpt, loading...\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "de6156dcb2674899bbaa068149530b80", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "96ad9a476be744bc9723a9fbf21c47a8", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}], "source": ["resnetpreact_model, resnetpreact_results = train_model(\n", "    model_name=\"ResNet\",\n", "    model_hparams={\n", "        \"num_classes\": 10,\n", "        \"c_hidden\": [16, 32, 64],\n", "        \"num_blocks\": [3, 3, 3],\n", "        \"act_fn_name\": \"relu\",\n", "        \"block_name\": \"PreActResNetBlock\",\n", "    },\n", "    optimizer_name=\"SGD\",\n", "    optimizer_hparams={\"lr\": 0.1, \"momentum\": 0.9, \"weight_decay\": 1e-4},\n", "    save_name=\"ResNetPreAct\",\n", ")"]}, {"cell_type": "markdown", "id": "d6c28e56", "metadata": {"papermill": {"duration": 0.043574, "end_time": "2021-09-16T12:38:19.939055", "exception": false, "start_time": "2021-09-16T12:38:19.895481", "status": "completed"}, "tags": []}, "source": ["### Tensorboard log\n", "\n", "Similarly to our GoogleNet model, we also have a TensorBoard log for the ResNet model. We can open it below."]}, {"cell_type": "code", "execution_count": 29, "id": "03e971e8", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:20.032044Z", "iopub.status.busy": "2021-09-16T12:38:20.031563Z", "iopub.status.idle": "2021-09-16T12:38:20.033489Z", "shell.execute_reply": "2021-09-16T12:38:20.033078Z"}, "papermill": {"duration": 0.050816, "end_time": "2021-09-16T12:38:20.033652", "exception": false, "start_time": "2021-09-16T12:38:19.982836", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Opens tensorboard in notebook. Adjust the path to your CHECKPOINT_PATH! Feel free to change \"ResNet\" to \"ResNetPreAct\"\n", "# %tensorboard --logdir ../saved_models/tutorial5/tensorboards/ResNet/"]}, {"cell_type": "markdown", "id": "51793e5b", "metadata": {"papermill": {"duration": 0.043746, "end_time": "2021-09-16T12:38:20.123623", "exception": false, "start_time": "2021-09-16T12:38:20.079877", "status": "completed"}, "tags": []}, "source": ["<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/tensorboard_screenshot_ResNet.png\" width=\"1000px\"></center>\n", "\n", "Feel free to explore the TensorBoard yourself, including the computation graph.\n", "In general, we can see that with SGD, the ResNet has a higher training loss than the GoogleNet in the first stage of the training.\n", "After reducing the learning rate however, the model achieves even higher validation accuracies.\n", "We compare the precise scores at the end of the notebook."]}, {"cell_type": "markdown", "id": "5bd1f30b", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.043972, "end_time": "2021-09-16T12:38:20.211663", "exception": false, "start_time": "2021-09-16T12:38:20.167691", "status": "completed"}, "tags": []}, "source": ["## DenseNet\n", "\n", "<div class=\"center-wrapper\"><div class=\"video-wrapper\"><iframe src=\"https://www.youtube.com/embed/ELEqNwv9vkE\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen></iframe></div></div>\n", "\n", "[DenseNet](https://arxiv.org/abs/1608.06993) is another architecture for enabling very deep neural networks and takes a slightly different perspective on residual connections.\n", "Instead of modeling the difference between layers, DenseNet considers residual connections as a possible way to reuse features across layers, removing any necessity to learn redundant feature maps.\n", "If we go deeper into the network, the model learns abstract features to recognize patterns.\n", "However, some complex patterns consist of a combination of abstract features (e.g. hand, face, etc.\n", "), and low-level features (e.g. edges, basic color, etc.).\n", "To find these low-level features in the deep layers, standard CNNs have to learn copy such feature maps, which wastes a lot of parameter complexity.\n", "DenseNet provides an efficient way of reusing features by having each convolution depends on all previous input features, but add only a small amount of filters to it.\n", "See the figure below for an illustration (figure credit - [Hu et al. ](https://arxiv.org/abs/1608.06993)):\n", "\n", "<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/densenet_block.svg\" style=\"display: block; margin-left: auto; margin-right: auto;\" width=\"500px\"/></center>\n", "\n", "The last layer, called the transition layer, is responsible for reducing the dimensionality of the feature maps in height, width, and channel size.\n", "Although those technically break the identity backpropagation, there are only a few in a network so that it doesn't affect the gradient flow much.\n", "\n", "We split the implementation of the layers in DenseNet into three parts: a `DenseLayer`, and a `DenseBlock`, and a `TransitionLayer`.\n", "The module `DenseLayer` implements a single layer inside a dense block.\n", "It applies a 1x1 convolution for dimensionality reduction with a subsequential 3x3 convolution.\n", "The output channels are concatenated to the originals and returned.\n", "Note that we apply the Batch Normalization as the first layer of each block.\n", "This allows slightly different activations for the same features to different layers, depending on what is needed.\n", "Overall, we can implement it as follows:"]}, {"cell_type": "code", "execution_count": 30, "id": "2e3e3885", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:20.306952Z", "iopub.status.busy": "2021-09-16T12:38:20.306470Z", "iopub.status.idle": "2021-09-16T12:38:20.308495Z", "shell.execute_reply": "2021-09-16T12:38:20.308013Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.052369, "end_time": "2021-09-16T12:38:20.308599", "exception": false, "start_time": "2021-09-16T12:38:20.256230", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class DenseLayer(nn.Module):\n", "    def __init__(self, c_in, bn_size, growth_rate, act_fn):\n", "        \"\"\"\n", "        Inputs:\n", "            c_in - Number of input channels\n", "            bn_size - Bottleneck size (factor of growth rate) for the output of the 1x1 convolution. Typically between 2 and 4.\n", "            growth_rate - Number of output channels of the 3x3 convolution\n", "            act_fn - Activation class constructor (e.g. nn.ReLU)\n", "        \"\"\"\n", "        super().__init__()\n", "        self.net = nn.Sequential(\n", "            nn.BatchNorm2d(c_in),\n", "            act_fn(),\n", "            nn.Conv2d(c_in, bn_size * growth_rate, kernel_size=1, bias=False),\n", "            nn.BatchNorm2d(bn_size * growth_rate),\n", "            act_fn(),\n", "            nn.Conv2d(bn_size * growth_rate, growth_rate, kernel_size=3, padding=1, bias=False),\n", "        )\n", "\n", "    def forward(self, x):\n", "        out = self.net(x)\n", "        out = torch.cat([out, x], dim=1)\n", "        return out"]}, {"cell_type": "markdown", "id": "843bc4ba", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.044366, "end_time": "2021-09-16T12:38:20.396993", "exception": false, "start_time": "2021-09-16T12:38:20.352627", "status": "completed"}, "tags": []}, "source": ["The module `DenseBlock` summarizes multiple dense layers applied in sequence.\n", "Each dense layer takes as input the original input concatenated with all previous layers' feature maps:"]}, {"cell_type": "code", "execution_count": 31, "id": "58fe12f0", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:20.490813Z", "iopub.status.busy": "2021-09-16T12:38:20.490330Z", "iopub.status.idle": "2021-09-16T12:38:20.492386Z", "shell.execute_reply": "2021-09-16T12:38:20.491984Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.050259, "end_time": "2021-09-16T12:38:20.492487", "exception": false, "start_time": "2021-09-16T12:38:20.442228", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class DenseBlock(nn.Module):\n", "    def __init__(self, c_in, num_layers, bn_size, growth_rate, act_fn):\n", "        \"\"\"\n", "        Inputs:\n", "            c_in - Number of input channels\n", "            num_layers - Number of dense layers to apply in the block\n", "            bn_size - Bottleneck size to use in the dense layers\n", "            growth_rate - Growth rate to use in the dense layers\n", "            act_fn - Activation function to use in the dense layers\n", "        \"\"\"\n", "        super().__init__()\n", "        layers = []\n", "        for layer_idx in range(num_layers):\n", "            # Input channels are original plus the feature maps from previous layers\n", "            layer_c_in = c_in + layer_idx * growth_rate\n", "            layers.append(DenseLayer(c_in=layer_c_in, bn_size=bn_size, growth_rate=growth_rate, act_fn=act_fn))\n", "        self.block = nn.Sequential(*layers)\n", "\n", "    def forward(self, x):\n", "        out = self.block(x)\n", "        return out"]}, {"cell_type": "markdown", "id": "ab2ac4c2", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.045107, "end_time": "2021-09-16T12:38:20.581394", "exception": false, "start_time": "2021-09-16T12:38:20.536287", "status": "completed"}, "tags": []}, "source": ["Finally, the `TransitionLayer` takes as input the final output of a dense block and reduces its channel dimensionality using a 1x1 convolution.\n", "To reduce the height and width dimension, we take a slightly different approach than in ResNet and apply an average pooling with kernel size 2 and stride 2.\n", "This is because we don't have an additional connection to the output that would consider the full 2x2 patch instead of a single value.\n", "Besides, it is more parameter efficient than using a 3x3 convolution with stride 2.\n", "Thus, the layer is implemented as follows:"]}, {"cell_type": "code", "execution_count": 32, "id": "851d8ece", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:20.676384Z", "iopub.status.busy": "2021-09-16T12:38:20.675900Z", "iopub.status.idle": "2021-09-16T12:38:20.677944Z", "shell.execute_reply": "2021-09-16T12:38:20.677383Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.052693, "end_time": "2021-09-16T12:38:20.678048", "exception": false, "start_time": "2021-09-16T12:38:20.625355", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class TransitionLayer(nn.Module):\n", "    def __init__(self, c_in, c_out, act_fn):\n", "        super().__init__()\n", "        self.transition = nn.Sequential(\n", "            nn.BatchNorm2d(c_in),\n", "            act_fn(),\n", "            nn.Conv2d(c_in, c_out, kernel_size=1, bias=False),\n", "            nn.AvgPool2d(kernel_size=2, stride=2),  # Average the output for each 2x2 pixel group\n", "        )\n", "\n", "    def forward(self, x):\n", "        return self.transition(x)"]}, {"cell_type": "markdown", "id": "2bd1b0a3", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.044056, "end_time": "2021-09-16T12:38:20.768481", "exception": false, "start_time": "2021-09-16T12:38:20.724425", "status": "completed"}, "tags": []}, "source": ["Now we can put everything together and create our DenseNet.\n", "To specify the number of layers, we use a similar notation as in ResNets and pass on a list of ints representing the number of layers per block.\n", "After each dense block except the last one, we apply a transition layer to reduce the dimensionality by 2."]}, {"cell_type": "code", "execution_count": 33, "id": "43aa347f", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:20.867695Z", "iopub.status.busy": "2021-09-16T12:38:20.865523Z", "iopub.status.idle": "2021-09-16T12:38:20.869690Z", "shell.execute_reply": "2021-09-16T12:38:20.869216Z"}, "papermill": {"duration": 0.056811, "end_time": "2021-09-16T12:38:20.869813", "exception": false, "start_time": "2021-09-16T12:38:20.813002", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class DenseNet(nn.Module):\n", "    def __init__(\n", "        self, num_classes=10, num_layers=[6, 6, 6, 6], bn_size=2, growth_rate=16, act_fn_name=\"relu\", **kwargs\n", "    ):\n", "        super().__init__()\n", "        self.hparams = SimpleNamespace(\n", "            num_classes=num_classes,\n", "            num_layers=num_layers,\n", "            bn_size=bn_size,\n", "            growth_rate=growth_rate,\n", "            act_fn_name=act_fn_name,\n", "            act_fn=act_fn_by_name[act_fn_name],\n", "        )\n", "        self._create_network()\n", "        self._init_params()\n", "\n", "    def _create_network(self):\n", "        c_hidden = self.hparams.growth_rate * self.hparams.bn_size  # The start number of hidden channels\n", "\n", "        # A first convolution on the original image to scale up the channel size\n", "        self.input_net = nn.Sequential(\n", "            # No batch norm or activation function as done inside the Dense layers\n", "            nn.Conv2d(3, c_hidden, kernel_size=3, padding=1)\n", "        )\n", "\n", "        # Creating the dense blocks, eventually including transition layers\n", "        blocks = []\n", "        for block_idx, num_layers in enumerate(self.hparams.num_layers):\n", "            blocks.append(\n", "                DenseBlock(\n", "                    c_in=c_hidden,\n", "                    num_layers=num_layers,\n", "                    bn_size=self.hparams.bn_size,\n", "                    growth_rate=self.hparams.growth_rate,\n", "                    act_fn=self.hparams.act_fn,\n", "                )\n", "            )\n", "            c_hidden = c_hidden + num_layers * self.hparams.growth_rate  # Overall output of the dense block\n", "            if block_idx < len(self.hparams.num_layers) - 1:  # Don't apply transition layer on last block\n", "                blocks.append(TransitionLayer(c_in=c_hidden, c_out=c_hidden // 2, act_fn=self.hparams.act_fn))\n", "                c_hidden = c_hidden // 2\n", "\n", "        self.blocks = nn.Sequential(*blocks)\n", "\n", "        # Mapping to classification output\n", "        self.output_net = nn.Sequential(\n", "            nn.BatchNorm2d(c_hidden),  # The features have not passed a non-linearity until here.\n", "            self.hparams.act_fn(),\n", "            nn.AdaptiveAvgPool2d((1, 1)),\n", "            nn.Flatten(),\n", "            nn.Linear(c_hidden, self.hparams.num_classes),\n", "        )\n", "\n", "    def _init_params(self):\n", "        # Based on our discussion in Tutorial 4, we should initialize the\n", "        # convolutions according to the activation function\n", "        for m in self.modules():\n", "            if isinstance(m, nn.Conv2d):\n", "                nn.init.kaiming_normal_(m.weight, nonlinearity=self.hparams.act_fn_name)\n", "            elif isinstance(m, nn.BatchNorm2d):\n", "                nn.init.constant_(m.weight, 1)\n", "                nn.init.constant_(m.bias, 0)\n", "\n", "    def forward(self, x):\n", "        x = self.input_net(x)\n", "        x = self.blocks(x)\n", "        x = self.output_net(x)\n", "        return x"]}, {"cell_type": "markdown", "id": "e313031d", "metadata": {"papermill": {"duration": 0.044311, "end_time": "2021-09-16T12:38:20.957869", "exception": false, "start_time": "2021-09-16T12:38:20.913558", "status": "completed"}, "tags": []}, "source": ["Let's also add the DenseNet to our model dictionary:"]}, {"cell_type": "code", "execution_count": 34, "id": "68ac8480", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:21.050044Z", "iopub.status.busy": "2021-09-16T12:38:21.049548Z", "iopub.status.idle": "2021-09-16T12:38:21.051561Z", "shell.execute_reply": "2021-09-16T12:38:21.051090Z"}, "papermill": {"duration": 0.049469, "end_time": "2021-09-16T12:38:21.051663", "exception": false, "start_time": "2021-09-16T12:38:21.002194", "status": "completed"}, "tags": []}, "outputs": [], "source": ["model_dict[\"DenseNet\"] = DenseNet"]}, {"cell_type": "markdown", "id": "543070b1", "metadata": {"papermill": {"duration": 0.044358, "end_time": "2021-09-16T12:38:21.140905", "exception": false, "start_time": "2021-09-16T12:38:21.096547", "status": "completed"}, "tags": []}, "source": ["Lastly, we train our network.\n", "In contrast to ResNet, DenseNet does not show any issues with Adam, and hence we train it with this optimizer.\n", "The other hyperparameters are chosen to result in a network with a similar parameter size as the ResNet and GoogleNet.\n", "Commonly, when designing very deep networks, DenseNet is more parameter\n", "efficient than ResNet while achieving a similar or even better\n", "performance."]}, {"cell_type": "code", "execution_count": 35, "id": "4140be79", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:21.234911Z", "iopub.status.busy": "2021-09-16T12:38:21.234434Z", "iopub.status.idle": "2021-09-16T12:38:26.276566Z", "shell.execute_reply": "2021-09-16T12:38:26.276955Z"}, "papermill": {"duration": 5.091682, "end_time": "2021-09-16T12:38:26.277090", "exception": false, "start_time": "2021-09-16T12:38:21.185408", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model at saved_models/ConvNets/DenseNet.ckpt, loading...\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "00d2536fa8f84cb682fda61b782facff", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "62f007cb989544e1888350cd397dd348", "version_major": 2, "version_minor": 0}, "text/plain": ["Testing: 0it [00:00, ?it/s]"]}, "metadata": {}, "output_type": "display_data"}], "source": ["densenet_model, densenet_results = train_model(\n", "    model_name=\"DenseNet\",\n", "    model_hparams={\n", "        \"num_classes\": 10,\n", "        \"num_layers\": [6, 6, 6, 6],\n", "        \"bn_size\": 2,\n", "        \"growth_rate\": 16,\n", "        \"act_fn_name\": \"relu\",\n", "    },\n", "    optimizer_name=\"Adam\",\n", "    optimizer_hparams={\"lr\": 1e-3, \"weight_decay\": 1e-4},\n", ")"]}, {"cell_type": "markdown", "id": "3501b278", "metadata": {"papermill": {"duration": 0.046278, "end_time": "2021-09-16T12:38:26.372217", "exception": false, "start_time": "2021-09-16T12:38:26.325939", "status": "completed"}, "tags": []}, "source": ["### Tensorboard log\n", "\n", "Finally, we also have another TensorBoard for the DenseNet training. We take a look at it below:"]}, {"cell_type": "code", "execution_count": 36, "id": "d80d2fb6", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:26.467681Z", "iopub.status.busy": "2021-09-16T12:38:26.466746Z", "iopub.status.idle": "2021-09-16T12:38:26.469713Z", "shell.execute_reply": "2021-09-16T12:38:26.469310Z"}, "papermill": {"duration": 0.051752, "end_time": "2021-09-16T12:38:26.470001", "exception": false, "start_time": "2021-09-16T12:38:26.418249", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Opens tensorboard in notebook. Adjust the path to your CHECKPOINT_PATH! Feel free to change \"ResNet\" to \"ResNetPreAct\"\n", "# %tensorboard --logdir ../saved_models/tutorial5/tensorboards/DenseNet/"]}, {"cell_type": "markdown", "id": "e120cb65", "metadata": {"papermill": {"duration": 0.04597, "end_time": "2021-09-16T12:38:26.561985", "exception": false, "start_time": "2021-09-16T12:38:26.516015", "status": "completed"}, "tags": []}, "source": ["<center width=\"100%\"><img src=\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/04-inception-resnet-densenet/tensorboard_screenshot_DenseNet.png\" width=\"1000px\"></center>\n", "\n", "The overall course of the validation accuracy and training loss resemble the training of GoogleNet, which is also related to training the network with Adam.\n", "Feel free to explore the training metrics yourself."]}, {"cell_type": "markdown", "id": "f257e8ce", "metadata": {"papermill": {"duration": 0.046097, "end_time": "2021-09-16T12:38:26.654051", "exception": false, "start_time": "2021-09-16T12:38:26.607954", "status": "completed"}, "tags": []}, "source": ["## Conclusion and Comparison\n", "\n", "After discussing each model separately, and training all of them, we can finally compare them.\n", "First, let's organize the results of all models in a table:"]}, {"cell_type": "code", "execution_count": 37, "id": "3697d93c", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:26.750263Z", "iopub.status.busy": "2021-09-16T12:38:26.749790Z", "iopub.status.idle": "2021-09-16T12:38:26.752565Z", "shell.execute_reply": "2021-09-16T12:38:26.752163Z"}, "papermill": {"duration": 0.052746, "end_time": "2021-09-16T12:38:26.752665", "exception": false, "start_time": "2021-09-16T12:38:26.699919", "status": "completed"}, "tags": []}, "outputs": [{"data": {"text/html": ["<!-- Some HTML code to increase font size in the following table -->\n", "<style>\n", "th {font-size: 120%;}\n", "td {font-size: 120%;}\n", "</style>\n"], "text/plain": ["<IPython.core.display.HTML object>"]}, "metadata": {}, "output_type": "display_data"}], "source": ["%%html\n", "<!-- Some HTML code to increase font size in the following table -->\n", "<style>\n", "th {font-size: 120%;}\n", "td {font-size: 120%;}\n", "</style>"]}, {"cell_type": "code", "execution_count": 38, "id": "da5c4666", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:38:26.858803Z", "iopub.status.busy": "2021-09-16T12:38:26.858324Z", "iopub.status.idle": "2021-09-16T12:38:26.861272Z", "shell.execute_reply": "2021-09-16T12:38:26.861653Z"}, "papermill": {"duration": 0.062486, "end_time": "2021-09-16T12:38:26.861785", "exception": false, "start_time": "2021-09-16T12:38:26.799299", "status": "completed"}, "tags": []}, "outputs": [{"data": {"text/html": ["<table>\n", "<thead>\n", "<tr><th>Model       </th><th>Val Accuracy  </th><th>Test Accuracy  </th><th>Num Parameters  </th></tr>\n", "</thead>\n", "<tbody>\n", "<tr><td>GoogleNet   </td><td>90.40%        </td><td>89.70%         </td><td>260,650         </td></tr>\n", "<tr><td>ResNet      </td><td>91.84%        </td><td>91.06%         </td><td>272,378         </td></tr>\n", "<tr><td>ResNetPreAct</td><td>91.80%        </td><td>91.07%         </td><td>272,250         </td></tr>\n", "<tr><td>DenseNet    </td><td>90.72%        </td><td>90.23%         </td><td>239,146         </td></tr>\n", "</tbody>\n", "</table>"], "text/plain": ["<IPython.core.display.HTML object>"]}, "metadata": {}, "output_type": "display_data"}], "source": ["all_models = [\n", "    (\"GoogleNet\", googlenet_results, googlenet_model),\n", "    (\"ResNet\", resnet_results, resnet_model),\n", "    (\"ResNetPreAct\", resnetpreact_results, resnetpreact_model),\n", "    (\"DenseNet\", densenet_results, densenet_model),\n", "]\n", "table = [\n", "    [\n", "        model_name,\n", "        f\"{100.0*model_results['val']:4.2f}%\",\n", "        f\"{100.0*model_results['test']:4.2f}%\",\n", "        f\"{sum(np.prod(p.shape) for p in model.parameters()):,}\",\n", "    ]\n", "    for model_name, model_results, model in all_models\n", "]\n", "display(\n", "    HTML(\n", "        tabulate.tabulate(table, tablefmt=\"html\", headers=[\"Model\", \"Val Accuracy\", \"Test Accuracy\", \"Num Parameters\"])\n", "    )\n", ")"]}, {"cell_type": "markdown", "id": "8f90a284", "metadata": {"papermill": {"duration": 0.047038, "end_time": "2021-09-16T12:38:26.955980", "exception": false, "start_time": "2021-09-16T12:38:26.908942", "status": "completed"}, "tags": []}, "source": ["First of all, we see that all models are performing reasonably well.\n", "Simple models as you have implemented them in the practical achieve considerably lower performance, which is beside the lower number of parameters also attributed to the architecture design choice.\n", "GoogleNet is the model to obtain the lowest performance on the validation and test set, although it is very close to DenseNet.\n", "A proper hyperparameter search over all the channel sizes in GoogleNet would likely improve the accuracy of the model to a similar level, but this is also expensive given a large number of hyperparameters.\n", "ResNet outperforms both DenseNet and GoogleNet by more than 1% on the validation set, while there is a minor difference between both versions, original and pre-activation.\n", "We can conclude that for shallow networks, the place of the activation function does not seem to be crucial, although papers have reported the contrary for very deep networks (e.g. [He et al. ](https://arxiv.org/abs/1603.05027)).\n", "\n", "In general, we can conclude that ResNet is a simple, but powerful architecture.\n", "If we would apply the models on more complex tasks with larger images and more layers inside the networks, we would likely see a bigger gap between GoogleNet and skip-connection architectures like ResNet and DenseNet.\n", "A comparison with deeper models on CIFAR10 can be for example found [here](https://github.com/kuangliu/pytorch-cifar).\n", "Interestingly, DenseNet outperforms the original ResNet on their setup but comes closely behind the Pre-Activation ResNet.\n", "The best model, a Dual Path Network ([Chen et.\n", "al](https://arxiv.org/abs/1707.01629)), is actually a combination of\n", "ResNet and DenseNet showing that both offer different advantages."]}, {"cell_type": "markdown", "id": "ba072451", "metadata": {"papermill": {"duration": 0.046843, "end_time": "2021-09-16T12:38:27.050355", "exception": false, "start_time": "2021-09-16T12:38:27.003512", "status": "completed"}, "tags": []}, "source": ["### Which model should I choose for my task?\n", "\n", "We have reviewed four different models.\n", "So, which one should we choose if have given a new task?\n", "Usually, starting with a ResNet is a good idea given the superior performance of the CIFAR dataset and its simple implementation.\n", "Besides, for the parameter number we have chosen here, ResNet is the fastest as DenseNet and GoogleNet have many more layers that are applied in sequence in our primitive implementation.\n", "However, if you have a really difficult task, such as semantic\n", "segmentation on HD images, more complex variants of ResNet and DenseNet\n", "are recommended."]}, {"cell_type": "markdown", "id": "871b776b", "metadata": {"papermill": {"duration": 0.047182, "end_time": "2021-09-16T12:38:27.148031", "exception": false, "start_time": "2021-09-16T12:38:27.100849", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "![Pytorch Lightning](data:image/png;base64,H4sIAAAAAAACA9ycyZKjyNqm93UV0bmlSeaprPJYCyQ0IAkECIQ2xxicScyj4MZ631fWKCKzKior8++qJvQvTphFhAbnlfujz7/BAf/ll8fPb/9jKQu6paxewiZN/vXLb49/L4mdBV8+gezTi2c3NuzmSV7Bae6BL5/stsm/vpxEQdjATQjS6fXXJ1/f8Ozq9u31x+NP//rl5eW3ENje48H0MAWN/eKGdlWD5suntvFh9q1NEmW3lwok03FZDRcV8EHjhp9ewunRl09h0xT1rwgSRE3YOp/f/tn1JFJ/dvP0n0jY3dTRqv6q0dagcvOsAVnzT4XeBGA3yVvvc018tlN7zDO7/8c9evQBjlI7AD/pFfK92qQ0vZ0Bt/m7gF7cKq/rvIqCKPvbYv8vVNPPN6V38pOlZHk2pHlbf3pJgRfZ0ytJ8uklmo4NqqgZvnyqQ5vCcNiXx8NWJdSTH906wtnTZd0KCI4F29JGk8vGjWQNkitdkNex0KZMhvSieSwLYWgWWMmZLbRsSkjZ3zVA46q+XSBpdzx9+fLpbWR1MySgDgH4u5iQt4eIX9kp6PPqVsOMC1zXxTHM8zAS81mbdHyGdh1g2w4OUOezW0/DRL6a9wwWMUGcj97O3TW3nU9owJeyrawf2up2zA/4YnfQeNs7G2YjhgpBu/uDn9BSs6Lp5RpSBSjz2Ei/6xy7rTdR1soXY8MevH4+CweEdhflVQ2zPuN5JEF5OOXitMehHOMzJAcY1+ZcluD+hOK7P3PIBJHHMAsTv0c34QRWGdJsq7TeC8nWxb1LpXgLhowcvVzjhwCctqeVme2tzVG7onVFnE5kcr0dUdwduCSPwkAfh2N3uX8Ama8OgMVxhnGAi1IO7dI4h+IkBjCc8YGH4w5n/47lMVlqt4qK5qcgvMlDVJO3ePz7EYoj8DOLlRlhzRy7QD/J5yWn007vHO8BNx4tXTQx/RZltLppz/q16xiNNCoO0xeNEfKWO/hifR7posmKw11VmoVOss0rimYoHj6+KJLItZsoz5B4mv5v3f30Ulfu38YCsi6q8iyd/ARMUIABDE18jutP//oNeZP7Oldmolg6SVlQq/rg7YyD194JbGfv617Y14BIr9XF1bt1z09tTbDDufWSQBrF84f7itMXldcBVb2G2UndQNxVdXyeWt4QjT59LAo3bLMb/M6boJ7D0bbNPIGHrZ16NPUGIk63NX4vlyil7RPnDLVXM2zoneS3F1xJJZe8mGlP7NpBI52twSuF44pLtBCXS5fL5aqk4w5V8bO7btxN8AweHci8KaugOZwkUA//C4sPmCc4F8ThGu3jkJST7rjyd+VtWbirTOnkLC+I43q3Gpv2ZLNHbx+YeOKdnJRaczWISrpQgtBIvcWBuJADvz6wq71u3Cvtg2H84Vg9x2NxgiV/RGI+C/68dKrL1ZG4U16dKx+V8pVzXexCPRshvjykGOqUV0Gie2rTD4xly+qJFMnlJdzfFobe2bvwGN7v4BKkwW6voDrP6PLf8BmvCeGUOrYJgCPvy6fPX79+2+vszAXeY7Rvjf65CX3TgFGKnOaU7T1hQlGNKWkVk4aLUQtJ7U5ZqhhIfuOt71K/kTJ5bfXUMrnHm/B2Ms6qJZ4J/kZtD/L6ILWLaKcnAiFfrANxH9fuEI/tSjz9DQfzU27ZlCM2AM4zeGqaJ8ksgN+LwYD2qCm7sZ9AskYVZXJDFxkTzCW6VBQDVTjCdiplYSWIKOvbzk9DSJCRi5ArxMpTOG7ICHHfHRtyf9gyu9Tk/XVkcztRuRZVfmYX1QwLdKd6Jo2qKq/mIPxDBXYIknB9/xnsnBCMLc0PA8Nsh9Aq8hNKM0TAXvOy4nmUIBzI3g3cyEQ4CV0Y7YDcepKEIPm01lRzGEvkdrrR7A3KVG5TnV31wp70v+HJfs7uW0VYz4P3uwxMA4xxCMd5Aj0G21xbTKd2ZqkeRa6vUvx4uS5Ky+jG+5VddnlFnakE2bcr3lkL/IVitGygUTWGyvP1uuhW/FpMjvWIMdxyleEbsQ+MxRx62VQCNtE8dG8aMPApzOOewk3BuTOaHNPwHBMUYVSCWdDu6IXH4tCIDHo3JJ9cT2gMScjLdXMv0HroFwcWIWkoEtCbJ0Zqd8CIFd8ts5LOB1Pto1kzNmuqyGkfB9RwXUTeVDIHlV2EM0n+RBUmfNqnyB8kJ/PZdkWD616Do40sqU2lmo1s2GqE93y+sP3kuD6r9L09BWcEtXr/3jn2WKICHoZS30YC69ALFiOCyzqFWD9Gj8Iqrx1rRlxx82KYR7EYYAdwDkOTz4jDduUnB9E1RmNh2QqeMSkT3BJFCM3aqIVloyp2hwsO7ZR7sPPZInDy2NwRqxpIl+1e2Jb+PUE62q54r71y2j4tDqvrDFv0QGIPU+qR5LYXZQEMEvCodeYg/IkkTNsOAygCfQJVOkZyTYRujgkNoDkq3ZZmSWNctFwxtgFtxHgIES2294RWDVUyztpoc/XBSUpqIru7OqIRGyjfNozYRMEKsUEeQ8GMuOJVdjD9yYtZHL+JwMAmfMCS7BPIGZrmQdYY4dFBg4rIsehwiZjeUNwXKy93KBFXD9K9ldzFRsNXhWjtFKBfcuPeoMdLgBKNH+PezcbTVbPRoC4uLM1ezrBH4EUNHOb5Da6BW4HmIwzyZ5owReEk5/ncE7iuqvIqJpNFjVtnn183Xe+X3DXOqTGHLGxBYmgGlOW6kyDT0A0FQ6Oj7uelJ8va9agNWNDspa253G4j2SDQpKLH806YEasfDGYzxHDH5jiMfoZfjMrDdr2+rnN2cTrEbr5NJeWg11fDUBM6bATNYvhCTYu1skYWImSWkILIMeI1oXmStzfOFY4HZWXXjIyPC5XOywxt5vBK8ziCi8i9TVH0I2zwB3qTR8Rt1sWf4RHLU7lOOAOTk7B0q4uoiqRzb04dxVR+6KTKQuba/eLSWYudZp2IqGNXTrDWK7zsJ69oEqlGWsgaNR2CyMotr6yzLXmeEZd9O0qcfBbCrxKwzaE2S3PPWHaqB1Tix4TYVC3oobzaYciKdIf2HN0bA201lNC0TWGpY7gAhQniNhiUmlj0m+IErVJqYx98DzsoI3IgofvKFp2TSwUzrNAHdtNWAHbtJMnbD3GGP5Gcqj7cQ230GVRDdq+We9/rPAlHbpoUsztroOOB9xJ96O97Mil7g5YN2SM0QVGD845EKkLaiKer0lo9umYWphnuLGSbR4217xamCKg5thglzeMkVVa081C+04FZxsU9zH5GjI74273ak5VPSpkEZC61jXsX3kRO93pFOi9RrNTw7YiE2g2J9zjXI41zrHhkLWJUYYbLprGvnjey9AKxw2t143eMvpxhlcEUQitQ5B9hjt9rwSyLcQRmu0/gyIuYf3XDpj0JK0zPi3Spaq686xRX2hyYmPe3w96uR0kO3dWOF2nfE+zbDU2j0ZExahAO0eUGkcZ2qR/GYJ1ytIzshBl2GGX142xzDZctqGZVLX9WglGS8iifpp5xoiLiyMEV66PT9LRFjXjtbijkUm2QlBV2ZUst1MU1lqsTHflMq+3D+5E9nPl86y2RqtZD7M7wPdL640nId4V19cbVjp3FsIua1+azVnDeycAowH2GdZ+RFXKccYw2K2tvIt41cc3t9WofFxdraY7a3T22Q7UgTrVfF8iFCLc3NhfEIYgSos28usZodc2XkJaqEpax1C6wqTTmr9iMmRy3aQE3+RxyXyVgn/Eo0mOfUqM0nt0fEFmrW8ijO5y6WvTZPdnIer1faI220uK69Fb9HQIDy66P1vp2wskc2oliTiYWxPT7g+oztlDt77cFpZUtnx9n2Fxit5kbvi6ZfoQL/IEcTNEeQ9nuMzJDVaXON0hacQdEqMu9mwYqEEKNuS7jlA3urpwtLdO8HTyg11VhIvdlstucVsHZb1cCvdld/EE5uiaU8/qmPGStO/HW59CMsikbCe1qHsTfVWCSwgBFcs84TVsYtb/wTmEa3ZBlMV7rUsdzRQrprbxsOfVmub3e+inEblFDZ0ZcNZo4XlfQ+TiW2MI8bPTqpJ1ZbMkOtKx7uj+ihTJj/qagqSJ3KmsT4D4O+ghz/JkmbFOUg/ko9oyV7fCKElgWRCJB9+vad/NhEEy97K17fIFKoO8bwWCEHbSVoX68jUKKqblKmKPpjr6Z9vtjWUGblrid5ODG2+uMzBpzhk1meRP5Xw+YTKtuYD9321kR5ieSMAAYzZCe/wSq66yJ177cHy718VGI3NNRJ1DaE2hiL6GsXFKbympHHlRbE5cRHhnZcuPEy20qnnnyOrbsrZMM7H7m+PTMCPZ+D45gxqpYUYE0alO4arMMVLNoficFYzbj0AT2jJjdX5IOOR1Iub5Yw7nVfDG+E5dSSzt/XzZxx9/pXkzWx3QzevTyllbY5qZfa6wsBEsX4mjosTGlZNUmZTIwF5ILuZIyY4VsGnoNMhfA3y5om4fxT1qwizEc7fvPWPlObVSWByKLURdzS/6O8jc33Yr1vnS1K3TYtrfTfViJU5w39fFarKHTIZBzU1lsriNJHGh/iBfnNRZvaBeh90oUSYowzOKYTwUcgIspef4Iv/kjvSnhsz0UEM84qxUGuSbt9WTPbomtWSH6Xj4KZ+hI3jyh8aqNUFWtRx6lMFCTfaX36OWshCVvL73rORK1cXlSeyQTT9itxw2gp2jAQcmMWPR1/B+BkKVRG8PxZ+SSquwUVd+JKh72ayVfQtZKWPDn1Cx753yXiMO5d1RLbp2Oijjs6sZrG4yrUlyyyvVK7/B7appXh0MiSLTMgSC8yoeCGZGmaJN6mn5tEz6unXk7XfcRxvhzWZgkAQ04+xkR5yiMuqrp3V2IWhe9VyDARgznq6QhfI2DzCaz1mfXWy2XxcWJETSqDDek9YpW48kh+IclsHdxT9zO/qrOtM1JO+WXcUbEqYDtpWAqT9yPgPpXNZggcdcjyKdU2uPoqaGeD1qr3jc3K9G3JunvLwQyWLSRGLcAAhU+SEN21pxoX+pqHEFNgnNnMfJdUTOcm3lsyjOkjDvKOS1Iww4Os1j6X5PCeVeavNeBUZ9wPY98xhkF3yIsixVp3TQ3mGllVFuL1NlkN/LIN1bskuesWVGLSORW/Ha7Ellvx4B0rB256WQ26RSBiqsjI1mcgB5jeSHeDHsxI95MGUveTDVKEhVOblfe7FPVPxSEGY/1PPYpF6lumWjLofgSW9A126bCEg8dkczRUDKxpCqOvrrDN0zb0753MFSE3XlNhnEOsUaWy61OJsVFPuv59aKnjjUFnoU15aT2LIusizyrow7AbeaB6lEDZnY3j+kPJWGccFiW9Z+xOq5NYYTWcHZXQ4g11p0VpAW0XXkCWjYHl/fp2gov+sqNMVtUhL0i63ImS+UoRsdhg+dHY+lBLXWKWFBa26O2bXF2L86ISG8p9RQz8raYlVz+SQgmOdp1GPCMDF0zPAmVaE7JTqQohF6R0BQqq1BXJJZ5KT3vfryt1lJ48unmBsKyk27iKqD2J+0aCIbU4I3vckJm7pJUEnpH16NiFc+wyxpUEajhxnbmpUPvdSZ+FENOyeQzViUXpX48rdBaOHJIW+Do2Pmee2x2tGgZbblWdDoufINVO0pTbe1SMZt7tRbQy+riXlG7SwS/tXZHlI8vEnPVOdQy0N0sfh14dPBxjsptE/sRKD4gfP8XsrBPojZHEM+I46GvgzE6rsIIq7YlQfKurEku5ldouyrPoeQYUrhDRIfsDt3ZI8OgVXTWPuqutO4cNK36tO54pjAMkO6OrLpV3Tw0ZsShaQxvFvXtku1ZTL8Xg1nKJz3ggieQbEDkVxsoofJtVmcx12bbcap/Lv2aH8+XeIFVtXA1kvNuu2sM3GnHKzk22kjYKh2up/juNAdlHO8ANKpDhl6b7v0rNKPiqRvbvQEPtqfEcP4K5l/VYIcCLOc/5VwY2maM5Wvk8sAzVy5VefR+2+/jo3hJDqaZefGIjP3V2J33dmny3R7BRstU9ukqk0k5xynpmq0J3+Uvd2TJqAsu2Y3dckbUeR19/fVc6uOi0CiAuwj0s4n+QBP2cMD5Dv2Mlcxdcdyt2eJ+Pm9wYkzjWMlXF6jzdisnqrQM3V8lJ7+emtsJNAFtiUTAa4YULQX84jQub2PMSbph10OJImvG2cQr3HfbOb70jcFUWQdTblN/FNM/6cE4zREkhz1jlSOTbOLsLXCZMQC5y3AysHkcHc7+rtOYPsk89sQKUGQu2QhvLvvAlkC6NG97AO7g5FU1ZGccKwoJ5cvRgO6gaMygZoadNmEFwOe3N+ZwfK8Dc65NOwz1DHtky0aXEyPi013RJsWOEBCIWTgEoTEHhWoWq62/ue6ZHZlCRBQ0xnqxIitmCAas4tUYcd01u22xQpC2Urfw15YxakUyh18065rI6XDYx22HIohn3J9gSqxx6YnEP58QQzXRMQkkJXQP53KDtnrGI9IIlLPiXUMhRTbF3b9cNpK+7mJKQGVwZ6DR2R7wlTCe906jCzrZBsQwY/Y2uV3PCiqvAjDF2j5GUc9Y8TloqsZqVpCJ3UnbSrFSskRTk+OpbDTGkq9TRrgkLz6qgQtPbpRmBDVqgeVCPAHhVtxbtoAEV8bPzUBWi0GxrfEkrOYQ66fwXude1KazuP0hAxMYTlK4/4zZ6XeuLFt2IdyV21I9mSNunsaq3NOVGl5YdtiYIsv4DnfJBlwsyaSKUPloG+KeuSw1PdKHlpT1AmLNfPAPRhGsz7ZtzKDXZh2IZt2K9aYAM8BnUEA8o9pr/GXetX5cFsuDcknLc93a9vEoy4h3iVhbFfST1gq7Vidxi79DZnvrAJlANo3yIdNzVnZgz+NZrBbKHrEgKI4Pi+WciPC64cEUFpu2huvWSeddZ/tXtSkL9FHAOs/IqF2pRaALE+iQaWzI+iI1aECsGcbxT74b9G2lys4B8hr2MiRKyoGVuYUQ25RCw3Zoo7barmu3FK9XkXQJL/6un6pDbwbLHjhBAvd2NStI/KECM6jtOP4PrvD+gNtyN1vqKrAUdL8BEB6DRq6ZPe0q1EYkXCyClFUodXRTHblDahNOes9psUgGylGCbBky6WINePsuCy1vHQ66xzNx50V/J4X+J0Ael9bV0VTpPhYSMJcFFId+n6bNLiXANQVcOmp8jq7qgTxuQc9qUm0BRhukxmR3bR+NWxJQcbPQ+kL3T0hm1ZsrD6G6rB01x4gZs11qneUuoWUMdtap+eDbk73I92vYwwBN08z3AH75tpVLZj+2enmktMVU1356+bpByJdPfeQ14RcPdJEL4Ncnn74ZURM1CfhXMUyM3fBtL5ksygIkyYP8c5EFL3bzkk5BGFQv/+d/vyiD/mi3/73ZXw58tFpHzaZ1fkPetN/tNfPWQQ+89XwC8q6PegheXlV68Pj77aNe+moCOH26n1cv4fQOPD2ZHqePW45fFtuXxylge2r4+UVz7QS8DHlbvTzuZUzq//mS5c1LMwk7eZSAqkjsBnx+gV8+brif3u818raDxWtvfvTl5wXI3t59RwC6p8m3rS7etfj8+vIrwC+f3nh+v++Mn7SRB0fug+IPt8p42M9ro0ebx+B+KPj6xUz10IT1MSl859epz/+OvHdfDUaiGMmyjyqJoCmcfXfg2zf6GOXkCZs2m6bp9OTdsdOzh4vESIYhGJpDqT/2Z3kn0PRRMzH/9XXjnV+nmfFO4duofvcFw3+9Pw/GsDROMziKMBzmcSSLPm78ADCGAQdmKZuBOZZhGQfFMAb1Ht35UU+mTwLvOvG/3j7rZ61du3oPrG7T1K6Gfyd2FYB/v/b2Z0e+fiPvDv1A0/zJJ/5HTb8/G9MfZpwHv37F/t9uR3/txK920vxHoP7BAB/z5N8PA3s3wK/u5ccHPPziu7a5E79udvXjtv8Nk+NPH9hWyQ8s5p0//f9W/o+bda/7jU05xJ+D0lu+8nf3bnrLQqbf/wsAAP//AGQCm/08bWV0YSBuYW1lPSJyZXF1ZXN0LWlkIiBjb250ZW50PSIwODA2OjA4QzM6NDdCOTA1OjhFNEQ0MDo2MTQzMzk1NyIgZGF0YS1wamF4LXRyYW5zaWVudD0idHJ1ZSIvPjxtZXRhIG5hbWU9Imh0bWwtc2FmZS1ub25jZSIgY29udGVudD0iNzM5ZDg5MDVkYTFhZDVjYWM2MGFlYTMyODhkY2NkNmQ4MmE4YjFlYjMyNDUyZTM5MmMyOTFlYjk1ODIyNDBhYyIgZGF0YS1wamF4LXRyYW5zaWVudD0idHJ1ZSIvPjxtZXRhIG5hbWU9InZpc2l0b3ItcGF5bG9hZCIgY29udGVudD0iZXlKeVpXWmxjbkpsY2lJNklpSXNJbkpsY1hWbGMzUmZhV1FpT2lJd09EQTJPakE0UXpNNk5EZENPVEExT2poRk5FUTBNRG8yTVRRek16azFOeUlzSW5acGMybDBiM0pmYVdRaU9pSTBOVE0zTVRZek1USTNOamM0TWpVek16azVJaXdpY21WbmFXOXVYMlZrWjJVaU9pSnBZV1FpTENKeVpXZHBiMjVmY21WdVpHVnlJam9pYVdGa0luMD0iIGRhdGEtcGpheC10cmFuc2llbnQ9InRydWUiLz48bWV0YSBuYW1lPSJ2aXNpdG9yLWhtYWMiIGNvbnRlbnQ9IjBlNjhjOTkyYzI0NGFjMjk2MmFiZGU1NzMyOTBiMWM3MGEyNjE2NzU0MjJhY2QwNWVlZjU5NDhjNjI4ZDBmZTQiIGRhdGEtcGpheC10cmFuc2llbnQ9InRydWUiLz7sPWl32ziS3/MruOrt3t3XocT7SGLPcyfO0XGOiZ1Od8/M44IkJDGmCDYP28q8/u9bBZAiRcmmZMk5Zmc6Y5IggCoU6iYA3bsnwf8ezWhBpITM6MFgyi5oFpAslPPS/0iDQi7IZCAFLCloUhwMMpqyPCpYNn+g2o6lWbamDKSQFEROP5IruchIkkdQ9fDevXvLXU+iYlr68jmd+4wDmLKsCMoiX9v9/ZyVWUDlgIV0LYCDQZGV8Gp0iIC6wHIaA/I0lOMoOR9IFyQuqejeEx1fh3SXIBPGJjGVASkqA2micRSQImJJC+lAPS+fyC+15y9+eWrpZjxPg/zDW/bEOI9iZsofj7zLn57Oz4zy1W+Dwy5R+rt/eWZOcmeqXF4QMjl5efTLhz8c36evL5Pff3+pZuqvf53nv+pXZXby/jbd//5p+sv8+Omlb1/qVJHLN2dxMXN+zoNz7al5WvwSTZ8o9OpSG+dHt+n+2a+5+ZK9f3/++jEjR78n9uXb13KhqG/nl6n7Ss+PPyaFp3u/f3gbQPf32r2zoGDxvIiCXJ6yvGhTnMU4vSwbCqYiaToM2AyZ4ZoOoIYcha0uRMMbWtALqCeXWdxqNC2KNH8wGl0Df1SxOL0qaJaQeORn7DKnmcf7ugHWbaDUr2sJaHdNALjoOWYrMzL6IS4eloCWjJV/mBQPeQnKRlPix8wfgYRebiR7XfgsLaJZ9InGcxlbj6OYthD45w9/lKx4CAyTA2bi4YEkroa43K8eMxgjAxVRV/rbP+o3xTyl4VEZAjoBXfOeJCyZIw4v3tYvEef6dZox1G4vwmXwqmXrtm0pqq0sI3JBsoj48TpIY0qKMqNPYzJZ85ZepSAYMxh487IiQF5Aw3wZgXdlkkTJZBk4qYb5IlwDgGOGU7zS/w04R51xa4qhO5YO2ktbBg3aerkmziLMinj48750K1iO4Zq27to9sIoMSIukq6Dd1KXtapptaEZPlxOQyGLqTaY08FjCDRGQ2xOzhDRebh+TOc26TALQLNsxHNM1l2uDfIxBBR7FtdR1ZwSGEhXzNf2tJwhNwjfjdySZ0LqBqihKQ/Rru1s/l6vdmfvtztqsu00H6+y3O6Sd0vDQmIEXEP6yIj3//LOB+VVJqKk4mq6Zqv4ZJNTUdRAmxTF7YG0hoaaqmbrquOpmEpqWcexl9I+S5oVHAk4+D7T2LC02ElFTcx1N05QO095aRK+hyG1l6prJ/BfkWkM3bU03+jhpL1yrKo5t2eYe7YqpOopua2Yf+hXXRnle0tuxq2ZqhtaVjtuz63pS3Jpd18/iLuz6jy4XrvAZsEQYLTUT17/9XVxZ9vcanX9WdzNSBNO/19WrG3oFE7KoWt2go9qt+J/gt3phOZvNPVIUWeSXBe22Q7+z2y4o84LNrm/Do88bgLXf//mPZRJ3WbKNY0W45QbchV9q8qbxxp/RhGYE4mKpdp0lmCLpJxKcX4IrlEuPgWNhqvwoBj7oigcweZmukcQ8PH/ZFYgPZ4FFLpNn5W9Pwmeu8/i3dz5JutxzEWUsQUFcaQ6iE5ZBw/ELZqlpvMItXUKpFuh/wwbp7ZNdjIW8unkjCysd2qqhQ2igmX1OZp4SmJ/e/hzVNi1L0RW3pz+WTUgSfeIi5KVxTcebunZUXVFUsxvGrHQd5V7MJhMaelF/r65i6w5oRK0P4QllfZ1pEGOZhqXpZp9DUDkBNHxcZhmw7Apb+qx4GsUQci8M3ANpTOJ8EeyRIGBlsnmwx8P1FQZrIrm19nKVXVwVplg1jL4BTudhxrwgjoLzYUjyKY9KhgUlwRQYs2As9tmVFxSkS9NtMXJUzVBh1EqfC5mX/iwqhm2+yzH1kMa0oF4eTRKvTHfHxjAdV9OdPnOdMG9GQeyDIZjD4Bx4FVMCUUg9NvaaVMPu+NiObZqm5fTgI2YKiNOOIGGugvM4yoshCUMPkym74+MopqlrvfxTzVaTO/WiWcqyAqYMHR26B0QsFb2gzQgzpXE6jCnJEm/GMvCGfJgur4Udx2qh2XfCyzF1RTX6lFyBMQQX6mHIvIQVHqh8UHhe18rsgouruhAlG322QdCIgl8495CFeEoaZHvIk2tjEPiGUntAyrRUzXY2m7g1SAlG2idKmqq4hmm4fSakR8gi8B1AFc3ozKdZvgesVMUCUbs1oYSk7ZdQ4HEDh+9IqDINETHgdMzC7o6Vhj6Q3ZsY6MEqjPIZxEq7o6Mbjq5pjrW9QeNR2gW/XZi23fExNR1N2uYGtsUy+1LXmglmTDX1PrvaQkLIkcelqiLJvnBxIbTVrD4VvW6CKp6tWHhnVEzLAGfb0vpY9yKil8MgJnmeMYjqxJdEr43ZHlABiwoecB/bCilq4YKeF9gtXr47FjbMDkQzfXPTIUgTYO4E29AUCCY20yMN8ErP7nc2HEzvGeq2uNwJZziuYelAm9vRZVGwOyKuqms6ODS3Q2QcZeBz7RUdC3Sr1ucLd9GZgFeF+eOA7sHagOFTddXYVoNs5++Km5UvWjeg5eiqBaFCr9EpU6BGiOlJHg6vzSRsSxNXsQz4Z24oPKjAdo8cXcUG66ZbvZGjAIrTsRi0H8Ux+iIpmeysxQAPWzdNpzdCq2J74fR4gghL2eGdcHCBK119M9cV0+zg+Hg5WeTUd4ANNgyCHqU3y1TDnqUkW+a/PeBgqpar6OpmvNCVgT1kVVyQUdXV1d6sWOUTpzRZgM8vowKzPDvjYII11RWnVzXVEgEeZzMP+4kNXPAkXBvCqA1VAR96MxG4OuRqd3GwdUV1Hac3iVM5nFM2o6gKvD1pJnAlHF2xtM24cd/QYeCmrSrKhlZSmGronmZpFuXUK7KIrHyC2xILTQEn29Qdzd3QdVj16LyCkl1dBkADDJOpaxtmGqIkLYsVWnhjlu0BEwukU1F6066Nio6SknogINM9kcICt9KGkPDWMzLO6K4KAtBwLEu3LHMzU1WZyZocQ/xMs4fYA9AAX9Jw3A2zYl00KupUemtnZFSQFt22tvKfGmTojES7C6yqqhiPbcgeXRxS8HMvWbby/Wx7NFBmDUCmBw3uXoOQBnvy4wAymE8LM5ObGY19swBIBDCl27f0roJeuzApmSOQ/ago1UYa6Op2FNhj6KspmCYyTMvezGxkFNxJ6D2kIXckMS0iOBP97KAEP3vOysxLKA13dTIRN9swwKxtmFLja6Hn4FhFM5LNPf6BeT+iqukGRqG9+bReIhWAV2Pt9oAXfs22nZ0nby/2TtNdx7A1tXcxax82C1u8F/unmaBibIgVetDqIrQzYFzBCf65vtns1Ko1IHFQxgQXoKtVhnh3TEzLcl2rN1dxPSbmnj76iFW8gIm+mbe8Dhdlf7jYuJfG0DfjWOGbhmSeexnqlWZp376zSxwz3bStDTUO6l7cQuRVq+L3QBhXc2x3w5BmzSRVi7P2NlGOqiq2siED36BaSJrGc69e5eHThI6jYh/46Yrj6r3OxLX02lK+bsFQjqE5urvhl/LW58OC5Lt+9BDLxV1Lczbjp975y4syRCdsu/m7DdEsW9VtbbMvRh2ieT64a2GQlTP/zvDDPKTa7z8uZ+EKtrUPchvUcKciaLDNVCuJzxGv8u6m0jVUw1A2TF7XSePbZWlug51rQ1WzdyKrT+ahR+KYc9mdEcxUwOOFQLlXYSxQ6vD/3WEGHG8plr2pn0lCL2TBdth09k3ciA04ecDofdjUy9NpgivrN4zbb0Mcx3UUt3dtxjKfC9fGJ0myqbfZgtdeqn8TZrppgPO34cfEdWv6F+hu8xHnFoiatqLgKubbI5rToky9S5adj2N2eWes55rgFGkbpr2X9vbcBsEbMAE32lIUXd3Mu8jBS809nvvdHbKh6aqiK5tl8kKKORTQ5gKFpNxLeGVZmusqSv9umyrnfT0OCyAZxa9F3X26i3Vpf/JdwJL06D9kWTpjUprxhZYS5uWkcUzyKWjh+1IxpVKzVFf6+VTiiRmpYJJPpZiBNxBKUYL1xBb8KSjLQ6kgE3g/ZhnlPTx580rK0BvLckmWOdg8yKK0kIKM5TnLokmU4PZn3PvLynwghXRMs4MBvwwAQEEnWVTMDwb5lAB7yzYxsl989uPo5On79ER7Nc0/Pvvt0/vzX1j44VzV/8jJ89+OkqcfP72JX7OXz85+zt+w55QVx6Pgr1f+rz+RJ5f5h1+jy9J2Ez26SH4/vWLp5cHBQMKNIoAKeInVpvjRR3JBBLoDKc+CZqO32NBd7/LOQSL4auuRuB21dlPTkFDNN83hx3xw+GgkejsUG7DF2QXd4xzyAu9WNr7z7fKHK6cd8G3hMQMqtloMxFkO3brInPyIhT4gq4cqVJZHroLE9gkQP707ev34uff23Zuz48dnL9689t69Pzn2Phz/9PzNm5f3Xx29e3l89vbk6PGx9/b49ZMXr595L16fnh2dnBxh7dNBazM80lcGNRNdHAyuxBb2atN5C6DiQDTi+4GvanhjqYEb6qaqmGEYjlXXdyHgUwNNH49NzSCq4YahaxPbIgoEpo6htg5FWAMvyNM1MG1KLDClYagSh8AlMFxqjxWTgm84JiHofV8lZmhYjjGmTqhRywdrhSs9wrGtm34fzHwNTG1smJrt24Zt2sR1QzM0LUMlJHQU8K1MxQwoOFqBQQwH7btt+MQc65YPow+d0HVuhvlxHUjf11RLd2zbGAcaDcaur/lAQYhiAssIA6pr9pjothM4Ci51Jw5VxyoxTAJuFVBHDFNwz8rBE7JYmryW50Zv52csC6Yn0WRaYG5klM4LLJDjukSC2lJH/jZriWI6WDlxoXWYQ4gKGykhVxuLWjiaju5YrqnecBZEp3lXGLsIbtRTe2tAG53WES7b9ZJcshtwWiXatv2npQ9qswWiPm9iq6Hm+G3kvNUL3xW09Vhpgd6JlzFW7It87S5vQ8t7tVLF423AJMYHgwCtHpiaeCBNMzpesS6bcbc4+GNGcog2RxCr5CNxXM7Iw13FUTCKZmDY8xGwJRumiMxGZ/9UB6HI2Ml1x52QNBq2cMWvJxekoPUhKiPedlXy6q5plrHstn2Lxtd33jLB4DpB17uBa7rzRHfLCDSTChNxLkcBatW1k7riMqQRBE6hjAwIXscwv+CHN8UMfKDvFP4/occbECTmh9UUVBJw+LrFgwEodH6sz5hciHLhz/DpH/GJ3wyhqn1e33Cm6aCwOWAYz49Xs+tYvB840qNzyhD4lTM8Zgpo1JrJ71SqgXmCyu26vJacB9ikVZlLDwgC6ppRxfztOUyiMcXziwTSo7oAnDg+bnRd31SuK+h8GZxyzClCHJx3ka0dp3rhkJC5mPuXckySSYnS2fHdAJsRd6f5rc/CeU1rsfVTZmUh0eRCbrZDcfcdNFaeAvWiCyqeUTeA61rMY0AFP/nLlxlJH0iYXjyXseBh5VNW2imMLmpQXPcJNYjoQpcw0YgVSBf2AoGOVDWGhqSi1Xcg9Fkhs7FcjWjBJumVrEnpXDYEe8v+RI6SMbgFCfohtCrFJWXy5RT4ScLDjGSAP2ZBmSPw/DxK5YItej48hQIMSKqCRyOywCdPSSI0HS4+kDHTxJKWYaqRytgEiJYLfcijmgxBtR5ln2TS26qetKDKOLqCAOgyCoupPIbwuPbNOeCK4j4JznELdhIKdn0gfWe7vjMeP+TtHkjK9w8HG2BZQ5eBLDNpHc6IJI8vADpIgLgKUlQUWUQEi5Lq4ZGY0xrWczHDLA6larJbPAeUCYG3ozjnk0CiBBo+ESXSKsOMDZxwbSBlTJADc0RCl7QYbdGTfBVLoRxP5HFMryT8wwcMsHguVUpb/F2Tu9VP2Gr3scyLaDyXfbDblCarnS04l/Nu1cMsk40bjPFAwmMRQGp91BLPK4muJnBCZB6jHwz++4QTTAKC/Y8kyHlfmjBk1FoL3Oeq+wHaZC6FM9REh615eQR6D+iPaupgoGsV5GkUgp6pmQNZ5id2BQGRpEiqBf+gTLj04OkMwWXlXCba9zMZWCBESqquMuIki9GvyGZN/ZQUU2kcxbGclTjDmElgIXhc4cHglSMpj/WhCRf4r7pxAn5rSNpQcyVraOqSOTRsyR6a7tAYKnA15aEqLjq0gAdXHiqqPHQ0flWHhitrcDfUbbiauiwKLKjmGlAFrhoWOuIZmgEtoECDAhV6teBqir6g9dDS8VZSh4qDKKrQGNoMbQ1vsdyx4R/gq0NdCzDE1rwzeGEiPlCkDm0o0GSogiBdcaMPXRNHABV1rAhDBGw0joXC6w91CzvQ8BHKNU6qoQUwADRAQUywRxUR06G+BpjgXxgFTjq0hwFLmoQvVCQG9MaJiwOvr0PDgJfwH/DI0EVgEgID7IemhfVU7JHXV00+V3xIMGp9aJs4FlPCscCUDTUTZm9o8zmEqRBYIFE4DV1BTICL49AAJMwuzLIJc3nkwCtdEn9hnAqgA+wgG0MDCWc6Mv/vE6ox5Cuuxi4mLTlF9d6WkSXZB7UB7jSYEVCMwNdJgA5S+0EGwzShRa2yIZpMQTUdLivDdvcj6L8F8DpVc41W6djFkfgI+Rd4wCXsB6fw+GOZ/kBm6UMsi1lwIPTtj0Lf/gjqY/EWdcbB99rT7/XHi3MAv9ePRcniHEBRgjYfLmg9eXsRDlSd85hm0EFSaoYVJeAGcb8hOJcaoo7NVfOcMLnExB42kHxQYmhDhGXnD1A3A5com0vcAoIFAfNf+QCqBKpWxz/5TDZX8eHaip+yUCvVOr+KqUoPHcwqu1l9ICnBKQQHLKi2Qja77e6LS0rmaCirRlVn9RE9fGP5YlVi1SW6tJUFXOppKTKvWiRg/+83mKzid/ri2Wvv/duljkTuExBIJh4EJUv1P0skuITN0pEqfEB//jlYmQd5OiMBRCCEarZGNd8nNtXUUDOoGRg2pb5r2cFYoVbojDXFpoFvmdQhvqlS23RMkxiWqQRhd8K7UiNJKB4/JH6ePizTrkh11YBfFgXYrbZtPgMRioFFIYiYVKdp8tf0CvwiYMVFYkGEKqKHTYxkywGq1IlfJPzI2pa0zApZJBnvdXwtbtXXWPLa0GvGFlYdK29v1QtQRxS9xfx2Rh2Vu20ewf/hH6pxVcUbbYr2hCyKVRXtgvlc7VbmzT/NFGm1D3vjPmzs4xVvpGqtBgpv0O1GUeS6m1XjwvOVj0aCAw47ur9rBtomQDh2r2hSSs2t3HaWO6FCwVJZkTKcabj6DADO4EbMvbBHdQNgpIUP3XGIex1bUQJqiDXB2arlaqn2dq80CStVLgsEm1gtp3i6F6rzVNbb3vPXIX5r5G1radPAczL2Km1XFf34w4KCGwmaiS4o/iFLcqJYknICjqKqgHcbg0umyfinLTW8Ev45UXV+q/F6Uree3KoIHWLdWF5TUxE1ZVET4UKXHDVw4G2nXVHhlTaTshX/CthlEYihiICzoOMf4FRFmvmyiX/wYTkOewZWj8TL0VMZLzmGnOXxGAuZB+ac8VddtThqGrUcoEb2uMhg6mPxuCR4y4JTPUGlhaiDu8P/QJku0XBCeXtUD+tiZwTRAOvK/XUeqLQyLpT/KkJfUV21NEm1VOFK0JjMF88QcDcuc5Pl6PSflzNcNr+m//pNq4gbS5hWBR1BvaaTgv4kH+4S8dv+6FrYkvRhOpeeRcXz0v/L2vdcDaCEp1cDaV5dl62sITkD6WoWP8hTAq7yIMVhZxcg1yidBwPOL/XouCwHU3oBl5BdQqB8XkyaCSR+zuISfMd12vyaIdS6gBvY+2psDbX7KL331UaS1rZaDpCWXgjCr33ZNgcZS8Ug0JAtTEftsot5EtLoywaofhiMsYZb24NdUCCmY2zJLyg7xjXDb4Kk1uflKj83h4AhnmK2CTR4Dpa1jIuMLJjkBJhJlqttG8shydSUfipnKec3WeY/k7BNquZpjcphfVfl9irMmr7z+cxnMRCPkULm9l3iTlXCshmJ227WIiJKM7ShP2Qkyx5WybpW8rJDnUaZdVQYhmUwLXrrc//aDhrFtqR0gLcWhJ8xXxz5vg3ZG5dgifDjNYQ/fMUB3D0JH43i6Doh25QYNReOqpVZd0iWIwHhG6ML/sQHV5V3SZrHCyDfGHUA5/Pqo85d0eZtBeIbowwMscTFXHdImdMKxDdGGZQoGRfu0cvRHcuUJMB8YwTiS2Xzu6TNCw7hWyMLXx9J7tpMvWiB+VwUWu/SPBqV8eE1r653lCBwVFoRWcFSKeU+qX69R77BNPCPkbgKZFPKL1H67nzX0xqvQxEYLQq+LHdvQlKx+Y8vPGJZtIUR/UykfVzhJ9X4HXZLviCJhXCsK1+kMzvFIsxfSYWsgdCavK0SHevyEYvZxr3jixnu5giWpo3nCq5PDWwxg2cI8xD/rgm5PtvIm91hn3v8xw3kw+b+Wlr8O0v2L5ElO75KQcHQf2fImhd1hmwfKTLwJjTMxXyGZNmNXs5u/gwVTPIVZN8qdgUNJW6qDO+X9F9u8jynRrOYbRVUGxV+P2MJQ0cbJky7zi09wfP4JZKEfHWj+AmVR6Op8fm5AtCKgjuMLrax3QKVQ3H90rHaRv6s+M3Pu43PtvFfW/gcth6+/sigwH2EfDfFV0DGsxqZw/ruqydgvaYoJv5waVHrV0BOruyQiIBcpfrw6YR8UY2/FVnxlFix3mo4KaPwDr+lbEHXN4CUJLCSOFYg9Ktl/4+s6mOWJKDxJDypU2LFlGb5bY2qsotRXbPADw+bmN3lJ7izKZXeAZBXx7idAX9M+ZsRrhaZxE99fRXSdSxQORTXb5CYszK5068xW/klFTb484blDCV1qeCboS4Ny2oVcmtL/ddA4SqCOq7xWySHFyXfDI1xj1l+LX15GP5liXyKCIpNY5hsXCpNRemXN7rrym+TJ/4c6VLcjESLNCbBZ8+XvmqBPmw9/Dtj+q+dMX0rTlr8djOm30jK1KhSpne8tLA6OLMxFf6Xym1WjDU4fIs/p/IZVhV+9oxhRevvqmUKsvgdmShvHRrwhX1N/Fkbif+cDfc0m8dvxgdqPiC2sznVrwV9JWTmuEin+MtFPPRuHr+89/N1LCHZPGro82p3Wv+wTdDbxA9fReBwV2selvt9NEpIs1llZefxdQcWtHew8HFWL/g9lKL9Wt65srbnWZTIwlfiG2H0xUaY7u7le/faHVRbfnNKsmDaMsP1gS3Vi2t8xJzGYxCAjBZ141koCnknNR5Q2Hig8KBLecBSGta9C0hLRQD/I/JIwQb3uodBrCBTb5bC0wr/a/C/eIogv380QiKC3kbmuJqlh/wNP6iHIc/gj2wcPsK/rc1jrWHL+Ko+mUIULe8rOsUtkUIwOPqy2Kz2brH/d+llFC6dLtW8qUeNBw+BadxgG2+NDO8DvCicwjU9sWySr3a31LhM1jVcDJZ/+dkOJ/xdRlB7wRQiWVocDN6fPZXBz53RYsqAAhPasPMjTsea+EhtfmYIEFzix3fK+Uxa4tDFqTK4TxImK5hmbEZjfu4Jb9AcObK2Xc1Uaxj61keDcMDVNkXkt/bO5S1GJrqp8AMTQOOwJQVNwTJ/1sfetAt5zQhQxd9rx6OF2hiJ7dqsOKfzg/z+qPVGHEb0x0pl/KF1WfwsKp44BUpuCeV2fR70Tlkc4vGcp4Ifut0tGG5N5So3stLm+hbtqqhzApJGBYmjT3hg23jcfi3EGIymz3x2tdSQbwAlecrSMhVmfV2N7tbU7nsOvzquWvSyUqXiAdQ01bxCyAdGOl+pWWuY9USsW+cl2OCcf52VMVwDScVfR02nf8SjZ7Q4Fa9p+HqxzfYJVk+q1cCtbvOUxjH/Le91w1sa2RJdG0H4PwAAAP//ALcASP88aW5wdXQgdHlwZT0iaGlkZGVuIiBkYXRhLWNzcmY9InRydWUiIGNsYXNzPSJqcy1kYXRhLWp1bXAtdG8tc3VnZ2VzdGlvbnMtcGF0aC1jc3JmIiB2YWx1ZT0iZE43N2pYSCtlZWQ5RkRQRFBCRlRXdmozeW1zdFV3RFRHaUh1bVdSWGRTNG5nU1JIRVN2VWFkeXkvQlVuVVJvL1VPVDREc2JOVXZwMWszN21SQXZhVVE9PSIgLz7sfWt33LaS4Hf9CmzfczMz54YU3w9Z0hzHiePMyE7OtePZnS9akESrGZHNviS7JTlnvu4/2j+0v2SrAJAEHy11y3JsZ9rHYpN4FoBCoapQKBwR+e80Xa7WNanvVuxstkiThC1nJM5oVZ3Nfqu0Kq2ZVjFaxgsN02jzlGXJjCxpDukxZEbOj4jy77TaXJHbPFtCAYu6Xp0cH9/c3Og3tl6UV8eWYRjHkGJGbtKkXpzNLGtGFiy9WtTwbswILVOqCTig/HLNWmjyUjMhKU1Y2UB0ze60CmIXs/PTFa0XZJ5m2dlsWSwhW1WXxTUA+ZfQD5+HL2akWNE4re/OZrozI8nZ7LWtu7q7MK3Y1H1iEJuYug1Pe2PasQEfvsYDNPhbaJBM4+k0G4PhaW80TMhDeQgm/QCgHCMsPYhaGLBa09QD4l0ExHR1c6Hp4YVpYMjCVDJjJ50fHSlde5qkm6YvvituyaqAwUmLpUajqsjWNSPFhpXzrLiR/UcSDXuC/LbOV1pdaNX66opVmKUiMLQTwVpcLGuaLlk56w/q0ek6a+puSp0uoWb5KqOANP2ijo5Os7QrYZ6xW4IPKAMyzmEYa1rWIghQLgdQ2LJmJVlpBpm7gG+b9Iry5mI01j4RNAZoRsoiAyQoVvwLQTmlpKZRukzY7dlMM1v8WhbaGgLLDIAmKoh0XRcTkE21ng/5JCBtlAJ2sYJBWmk4A0o2P5sBetCa8lo0MSGVhogBUZGgqaSI6xR6W623CeJQV4syXV5DP8IMskjNbuumCWIo27HmU7eZjKbXzlF8lXU2BctfrWSrYlDLGAyRSlZG6rTGEfk74whclHdy0mc0Ylk/fJOyG0D1s5kB09P0CAICOF5BfwBUutmMbppfqRRAK9cYyjZsWSSJmHQWsXT3OfzhL5RmmA7+LgLddyn8wX8eKl43JqRSgzXxvtB64aaJsx9ICGbRrIUWUJOYmMGADKaDNEQtxNQN34JAAyFxQoQFnxgHwVDSRgs/5EANXM18H0Jxsabbrgc0R/dCBzI7QAJ1ywjeIxiyKhPKXQQfXrsEgLbcjQ0PCn/wn0OiO7qVmbrjYvWBr0QhdTMuAt1DSuT382i6tdGGRUGgi38LiBlFwP8R/Xo0Xq3K4jcW1w+jVpNwgF2/iOA+arWBT4hXJh/c5/zHlANt8MVDINELAzvXMj3dDxxeF08JxB7xsZ8NY2Gw3PcYMoxzeF+/woAPUKuI6g2mHBuBur1wRIthsByyV8NimjF+L2viENjKXDDkXNj4vSliIEzGRhuEYmL8+5BrAdbSn20ijzuYa7IkCP7wOpiu256sG0KfK6GQ9+kwUrAcDyOkTDfAx7c8tI+OTdhTYiN2hE8dICshEU+kUtD1YRgAmvYjoP95ehgbPbAg2HeoRzyeBWiFhwTDy2zdcAg+VLrHowiP1zCOPyY7+/QYlqxzwcicQhOa/qUbWOtKsSZt7VRl/ZTpm36lWY0Lptqf7xjNge0rY4xo2MtAYS+DWQPHxCqqVIXM7RYoeFTHFQw5Lr64ZmxeE+Ambkq6InEFPFG5XsbAE/U+NGjNFatnE72kQBcVJbAkgADImiRA/ftdFRdZUWrRFbBdZZ3S8o6sbiGRCObAtBF5hrk9MmbfIppcNUy+wgus6FIRBKYSixoSNqfrrKPA6pCk0CWLtCJlt6h3jOVPo8im7mOsfH9IrrIiotk2QGiWkWJOfkzrV+tIAeM5hIvAe6rfLpYkWrpEllGLsiK+Ft280WiWXi21HDJkwF/9v//zf9VCR4O9vfRPNv6JxrF5o82LeF2N0QG/2k76N/ggdTE1II9vPr7Q86PT4yyFvjg9Xmf4c5+cAXOqZBXg2lYx4wEhQ7K9Y14eJIz7xBrOpEvg1cYr/VwxgCShiOBvCiKzsoRLCdB1RALe9YNot9LsNOmIkUzdUP4sreqouG2xAoWiXFBKRZxou2JSNGl6Tkk1LcGNEn4OAS4G2SgZrqVftyjHm3TZI7MHae4gzR2kuYM0d5DmDtLcQZo7SHMHae4gzf0ppLk/Wl4obpa4K/jnkxp4wy4PssNBdjjIDgfZ4SA7HGSHg+xwkB2+YtmhKK/oMv1AFZsWVXpQow/yw0F++IPkB4ErfyrJQTTpIDMcZIaDzHCQGQ4yw0FmOMgMB5nhq5QZDvsNB3nhfnmhsaMi7b82mfjgA4nJ50WZn7fRal/28rUh6nRqT9yULAM2fMOQ/tgkjzQHH9kV9Gm/Rb1jNCA9CGb9OCuu0uW/lqxel8vLujjDc1LVX+3nf7Vewv+rtF6sIz0ucvj45e5dASh6gURomS6vIGh1V2OQlilhUFsEPzmtYMjgJSniCn6qYl3GDF4uQVyCFQbe0pwCOYIXAKLQV8urWe+kj2zqK37O6jVbrqGa5fUAcVR5p5+dSyGLu6QstDhL4+uz2e/f/GNd1M9woakvUTQR3yfiB+jgAiLSmIs1Os8jYr4VPyt6lxU0kZlkYdC1PP1lurxcQXN6ReKRNXlOjOTQgF5xHbW4TJtCl+ss+7YDZwzk259+fHP56y+9gooyhTEEKJZXl+sy66Xnw3lyfNwN5PFwGI9Hg3iMQ3gsBvAYh+9YDN6xHLpjMXDHzbD1oFlXrOw36L/+azYaDG2RU1jeYjuYO5Yf48OkocFYaEfePGLUg6dtJI4fxm4UUj/yWRT4iRUkSRKGvm+HXhx4EyN+RZvh/ueLApa7hBTr+l+IwKJvCY+DwLcwr0m6/JavdCcVzvJ02ZsjpEnTm5K0SzKeserEwiLXK5hZ88u4pmdY1t/Wq29ovnqGYYA3ZwIz/pZxMP8GYLaxiElnOAXtF9iffLX+q/2DCEHEUULkfKsWxQ3PL8ZKFs4lViX4Er/PdpnJ/a7dcTYOiGhPGSFXAEnp+UdH1OW6gFTfIqs7zRxWf5jN/91m8yPGfEmzO2hRpfMEDw83sq9XMGT9QcFZDzNVzU1jxIpeMg4RmlNXIj2B5ZzQOAZE7tfM1/te1maGnxx/k4ku5tP5m6v6GQ9pJzgP4cOHs/uZpCUnEsKGjpxInMxacvdnQSjDs0LDsWnMgsBivhfEpm/5lmvQKLTjeG6ZlmfPE4PFhhEbsTN3IsYoDYzAZIYbJH2EOh+gl+zGPv3uCLzCtimMnuTVRJc3rG+7EHAWDa3nuUpYK+bcdB1wseWDcSCRY+XcKoqZIiv/x8vCEnh3rH6jt0i+M2i+sMivtDmeMO+Zw/NamyMIinAhUjYRE0pXnoCIZHMYBkJ+/13EvUHREEbl/KifgxPn5oz9abSu62LZKw3Grqi4RKZ8zuSBfpG+L8x8n1Z5WlVCespZVQHe9JW4EwKFqu14WOOg6EN4r2IO6MB8BbLDsh4IKUNVyW3Tb/fqKmzdtwg++gohw0OdJPH00Mls3bJQCWmpqodW83AR4lvAU5Fhqk5BAYWFUk8xSid0skLRcYFVkoCDhLqQQE1m8CSdguOop+EQgyS+mmFABAXUkMMDiKGge090we8G4Tp5RpR1xN07xNk6Ydq8pFewftYKti6LOp3LtRq4CZbNtWFqOX4RrUBWRtXIBO1Si6mOI1bTY14YtnZY3nkz57CBRwqHQ1errAElh2lGBBXhtUMleVprC1SXxLRMKo0taZSxpEuSpBXMbMTAe5MB2q+ZRpeJtiqnEyqMJqcquGuDxlf8TTrJEE4tqnjBcsqdWrwt5vUNLdlbTpFfFEmH3A0u8zZJesJXGk5oOpqiUJ/ugE4DzFGHGi3ROupRvo4R6rJrgly2oCxu+fong0HsJ6sapFeY54zTIq2KS8aWKMm6szE1bHlulTrJvS7IYiOh4o88AXEYfkEcdmdbJOlOB5anS42TC2Bkxe88rblcPZALThdmSxPVLTauJxtvsd1EWlQyeq3dANNL5rZQri1B+qdZe5Trs9M6vrUlmHOp/pWnyLhmcydCeNia+kO3pjrS3VNkoahRyJOFFVLSqi5Zjfp5xMpVWayaNM2g0iYrsIdkjo5KcHGWaSRv1lCoxhhUtZ0YJoFiQACGJNWYm2xTNdvFoySz82GIZMqUA4tqe/PbVpuotJZMH4g8Vgupy2J5pfQKstyqjx9rohPbLmup0tnsL9so6ZYWjlnq2fkoqGs0B3PU7AtkoQh/aspk7assCXTmOoLlrGk2cK6m6stnQuvYHXtFGo0kWhPiTzUS9qUCNm/EfvR0dIdkYUWTBNpwQqzVLTGeCdp7ivpRWUmLcnVRZHW6WoHg0r1qFYnqJf5pVd5nGf9XsSb5uqpJxLjkBflQNw+ka0GXwJ6oLACpWI1iTyUxelnMiywrbsbyxmfQJvQBXUew5KXcvIPrFsgNBWz7eA3Dxc8/Xv705s8iDyYRM12XJj4DqS9yYyNymeuYZuSwxLfjwArtxDUdm9m2MTeiJPYTh/rm3HdN16KePdum9d5Ru/3lrNYwF7LemozrbwC1UFiE5U5yGOBy5Luxbpi+pps2PIzQlwuM3a0yMUYGhK8wlm6aNo+Gj+dNYVjylDuxKS4A0gIXAMUSW3ABBnIB7sYCqcSPDaLbjgcVGpbuBVgTLLqhnyGkho2Mg+tRADjAP7nuGYYNX5mOjTIMD8swILv4c+Qfz+Xjn1hdIQRjMJePORew5Iee+MYiTUzv4V+bXhNF9UMdDPV45fjm96vhwGkInWgA8i6uQ3smDyZvpAatfN/vF95TH17bxKXQQYLPgEfXU4YLJcNfCL3hwMjYYdZ1ExpWuCYvQ/QUDJzuBMDk2K8sDKPDeADR8hBE28sUaJ8rTIZNfKz8/SSbQcgbVcA6apcoaSinEPg+ke+IOf+5AeqicUUZIVvo/45EH9UshKq7sl8gpedACsn6QYJu+oFneb5l/Dei6gFN5mwe+14QGtSI7MhIWOyzmFrwF8dzCHQTz7NtM4oMN45DN7RjO5pbfsJsw4meiKp/EXSd40rDw3HmNSoAcXIUwsydhLCAcIFCkbY8H+iCGcCUDwLUGwWmS0BegQizp0GCRKhosvxQmMJ4QGZCkK1MNLQxw5G2KQC5LjTReSRQFMfHTL6HJYTBOCnIamGm+5aGRTkXAASqxmzfGUKARCrIED4QyEzzwtdty9c9L1AMokzeyA+5gcKd4154kBIo6FCcdD0HSuSNgXY4KApaDjzMoF8rGrV5sDRADgvVZkaAqXwv1FCk8/qdieZHGcSio02IhDy2xrMMi9R4mVilhlXyVCGEOmMoIRD70UIJdIMrzTT1bSSA7WjVkd+3gEhHkutvldodRa6KOKWZJMLoXFX53kNuOUZ8vaIfYAI0unaVbpsurPUEiUHFSXWJVH6LGQ2sc/a1VENSqV2cWld2WVU+blGZF+X1F7+oCEtkBpJmVgmIDyvM9o1Jk0UeDT3DtcMkShLqWswzk9i0DMYC37JMK06c0E0YgyRGDKIEdW1qeZ4ZRwadP9kK84WsMlxZgUjDkp3WFGRZLXewK4EvQwtYQZJNy6KoJiOWYCuFBg8YWz3wg+dqjGHw/AHX76GIYFoBz2oS8cTMPO97mYb2s+OLxlVyGha+vdpJRaPTM7OFJcRrS/qQ2xgMq4+a956G21pfzWkavHrFitfg7P4Wyo7/XsKIHKk7kHQPSrxk9Q3kP85ZHiE5nqL0yn5oj0wHbiCptECKe2wdYXW/bjdIFfa/Z/XW1yopWwMVoGeVbpiWM76bMrHn0Or4extI+O/odEk3O+ndth3DeHimNCViTdznOExQulF2MrAqwlO1bWlNXDDT0JD21ybyDURu35dArdvD0KmFAWOY3BF0x6ZxpdvgBEyW7lBea66Dir3mWBGOVFwkaNcb7aPCFPVBJr77cTZLja4UHlWxjMU1LMRoOFSJnaJLQfI5rl0mxc0Sl0GBepdis01+wArMaMXkV02v5FtU0mW8aMJXNL7GNUOWB6hU3OFmX0V2b8GiqK/Z3dnsisSzoUlZh0zfkjftEShhVPYtwe020rVXIinHJ2kGMI2p8Ro4IxwcXOV3wdIeHvADZIvbSxUNRWBjur0dXdvTZ+3YDE+mNRGz3qaw+u8LWM8Q0XozrT031uqoq7zRUe+y4DliS98ab+nzDXar2a8fbfz39ut1x8elzR7vv6N842oOF2I8HXef1IVDKcK0+V4+l61GdRnd2YW2PPUwiaxL6yq7Z/FRRAuJrGczvn98js++KbmaQ54LeVPUhG6AG8Xt613G8AUuSGjC0hQt8AsXFbGiPAkh49vs1Z6k7FjkGlM0Uy1uO00TqQQN4quQfM/TjFXAo7OdyFEfio4qpXtSpZ8ENPvRpS+JBn3RpEfYceAZWGCbHkOChM4mxL1mfhau4R1t0n0j7zh5nce0AsigAQlEMdDc5+ID1c2GOHAXUKA5xNOlGZINP10AKmwgZF9SIbBsdi5+HyQXlu/sRSQg/SelEyu0OinZPwD2em9ygZknqIU1Ueh2osELIcDKxNe7kQe11o46rPakDr+g6V8D44FIfBoicZVC5ynI8KS8iq+bvo9aS9++ADLi27rn+8qeDu4p6YHrbICpQSWuekLWsXjui6YM+FYTGGga43x4zUXhvlWOkGYHzAYejrXk4dieJG7aQi2wwaK9oYQvher3LgjbVl8/wKVrVD7gAV4EYaGZ752F2RnmEBMK9baqDXr2SIYo40MOuYyBRmMk1wugZONN6+G270kxezMP7TSUzwfpZ+juRT5D95NSz87mcW/aqWQdU1B7VPB2+qkk3Yl8jivuiOjVnkT0e6XuAwn9RCJejoK8Yl77pETU5ERi2g9BMOmGwJ30QiCt9tRz/i7KkRe45w80xnHehwMprXEsAJRtyn2BdB8gLApG7hkQZgTlhQEsZOucwZDOGYxX1gZa5tgU96rEjpUv9aZOgN+GDbTf8JAeLmzdDMKhtwaLBA0QwyikoMRcaAGHD408nH4PKraMUxaduhvrocdP6fO4wCFtDZtRfaZwHGFar0xnulGmprQqRBt5NNYM+kS+cQZgYTw+Nrg7Fw7MU/l4mD1rTXU09qT0CnmYnSsfX6tQLc0B9yX1MtuYzDu9AreT+MYMcf/KOtJO9yTtz2WdB7L+Scj6KqN3T07Id5JvH5CT0TYhJDD9LZWXNjwIchzkY02zb80NyYEAO6iJ8xxuWBWq8QYy28GFUuy+VERi4uxcvnyt1EM6Otpfypb5xvTD7Rd5j4wt05Elu7lUQ0jv4xHQdAQm2lcAb2DqYD8Qjh0Ihxyrp6UdBxdYX4ULrB0F7GaKnjdvD5LM/aTqTytUVyxel2l9ty+dbPKN6aTXL3KSTjZJ+EY2Ak5oxkogT6sC6NUdqYtrtrysYrrEygjuvHVfjwCvI5zVnoTzbQPpgTX7NKaoC7xf/YlVlY7n66ZtDwzkUSIDOTDjhMvUvWAo+qEO0Ane+/wudNfzNN3G3VAzsPACdMMGydODdz0McNvECQNkwAIU6QPTBuJj8OSBi5ssKCD6DXGygRWMUW508IZ1w0H7VMMJ0Ube9UVGjWd8YWKVIPOaaGGP27po6kr89wjYqDmWicV5ns+bhK/Bh1xHMRu4wLAvIpsuwJBpbdsHsT4sFnbTdNsFwst1sR63tDVc1NoGckFCy1quZ0Utp+1Ap9iQytatEFlVXRzlgPocaDPapQIxD11M4ELfQT5DZsEziWHwwkTFhUtCURcqMAKIFW+y3UNQ8eCIL9oiGv06xD5rzlGaIKLjj9TdWmg3xdeo/lrSGm7ZU+vCvstAQydm583b5DIwOp3OD5nvQ9GOkWTSrLWipXHMVjjL2G193BT7t0WdZ1vOo3+KVSQFsgSAPmZPq2LjJcTvl7ed1b4C0rhoDXmWdZlGa6ijIglbMSApy/hOpJEBNCrqy/UKCmSw1mDlZMWKVcYIav7WSyT0+8K98w65bNOBBd9vV4sP31ML72afrWxIAbK8eCTKAebZtj2RgGvpHH1k9IKcsGR6c2S20bTetQelNt4yTIOfeYIHEGKQ7v1ROmJccJcbQd/HR2d+c+ETXygVHR+PJPijVGJt07Yd4L6PfjXYOTtv3j6D5C/MO3lFaM0pj/lu0iqN0gzm5onAwmd7T59Gp3ffLGlsKknr9ZBGVZGta6AR6PUQHsKisuQWlaTEXtKM2bnsmsZMfgcyKlLyGlEz1XwDZKwmI6eLgtBW6zwXPswyxc/NPYc0lFFTHFGMqUpeaoZ0m4atGYz3Z5//1yyChWFRlOkHZK+nzqOGDxq18Dn/cLIcdXa7W8iMZ9bg9HxVasUyA57gdVFOGLe1jm3k2LaukHYcVYk2/CS3QAt83aWTk7JYoR2syNv70qobcf9xr1XNvOwFNjwDzzY2xxUoM87HuYYOXr4CKZbRDxqJdvCKrFutW798U+A9XDdMdCPhtsCcfo56+Lh/SvaeEeuZGz71mA2G6g+zZtzbCnO6f4Wd2cf28JSJ1mfr6MdagO1rqDbdoT37k4/t17Hhxmfr1Y+yC3mcHct0Bysbvx/bvf0N08/WtY/aj91/v3i6O+Uu2Md25WDv6PNN/k+xNfWIzbUtxEFGf2xvDzTQn6O3/0AF9yN09tPd3yiuPrb7B9qbz4bsn1k5tK8WbAsHIvtyx0GRByRBslCEg/POg6YI7QKgcxWPs3ivgCIrxsABl/P0FqmF6nGyTnPGnW13BxZvs23HDgdOG7fumMg6B44dUUWKzmCFpkDxC8l/8P+p9AraeLalnbKyubljxdAJIroTq0COrON1PdgW2msW8fPkUeC4jul4TuIZUUxtZgSJE7tmMg+oHxlzz4182wvYLifOZ+e/NAA2TgmgHf9D0whWRSTqEoD0hIf0kPlkY1knIXXtKGFsbsydJJ7PmWkHfhJEpmMnkROa3twxWWI7CKnvOyY1LI/5AYuSKI7RXFnTJi5TUZ1OCleTyhV0rX+2FbrCHDimzLHvu8/mBruI1TfocLNJ07vZrsX/o3tvoJAYJXUtA4WKUKAM1StcwZFH8JhxNBQSnKQZfIK0HrekjkVx/aDeQTN184PEoJt+XHON0U2KTgJbkbEoCQqSX5IrGDw0IODbTXdqPpWR/nuv58PTNHC365XXud1Em3tz6qh+z5B/ULhU4+qB7XHHnBYRT6zCI/7CacuHUjYamvQPjgKE7VEAaQn6cEtf2/rYd8E2S/97fa40hHfi4iNlnZX+k4VPi3tu/2lXypiWeHVS/1aZVuk0Ne/e8snxupkb01FaXiSdQk76hx+nkm55J0FUkvE5MzsfzJljnDCDNvadgitFcFfg30nVaN8huNhiKq6uMoY+KaapQDcf5bH+F9zVuNCq9ebpnPI1vrtp7OAw/FM4DO/7Ch86xE+XqzVeqJWvb0UvClkOeo9GijMF4bPzREn9lwqWTvYTBpCptEkvsWAJeeoOiXvZplbMuMjWOb9ltEFlDVEZeeKq9WCKzqAF4pwIB6YtHzc94WCsa2Uqdb3QbKEiqcDt2BZ6XfAew0sv4J/wwLAUF5xyzWeKTvp5DfDDsmQi0xgkUT3eMsX7A2bERDY+d4AScQZvrnF/Ek23TCRvt6agNFzV51K+SHbLjn62+c0H01E4zTJWo3/h+XwbsJIAvOTd0adGE1m4M5FFkQFy7pxH0CfsfjVSQYF7LuZSySbHJ0GI4B37ZrYdG5oh6l1TsZ2gcgGuT0jbmiRFbESvZoPiO9ns/tT9qIrO3/E1YFjeZAe1uWDBYNmMTOOM6CCJ6w8P3j3ze+JyQ/yaKfcm96crAiP6rCj7PKNoveDFxpymGM1qMLlR3J1I3BDCAdaJrNxfPNA4FZK/8KgfRMwgF5K6ZpT7mSDmrYwY5OHTj3d6PwcPf5mWVX0Bca+5F6Fe1j7I/1gzEARBNl8VKS6Lu8hnUN+gS3ofMYVx1TjTvjFOTM82/dC2bE+3HC90rMHlcGJXSdJHJJAwQv/x8kPy4/u7QbeLax0lim1Lhb6/hQs5IIaIbr++yn6N7H/7jf5wu/pPa2FEebb+z7vwhv3Pv2/i/E1x8R9K+M3ZYERX0Fb0F45XveTFumKIgz1i0lcOtDe49OiDOkA6L+8lyE8sede/1kUpZws3KK9WEaVL5P/9d8Lf+FUvL4p1lqDzbIK7bu0k611dKKtQ7vgY1r5DK5bFa/Rs/W77xTQTcJ+/KeoF6t/Q5euiuGmVJAoofSBRRn/38/c/n8i7ZeitXM1JiirWGAsGcRLEPobapHrBZJu5RyVyUyz/qSZVDDQr0wm5Wdz9K5fER329vaFYzotOiaLuAo8byivteI8W2hNi28bqdnfz1h0RAgcZ+5Nfu7ECAbzvMwvTYEfJZBMKMM6PS3Cj4lar0g/c33sDFoQ9Ey74T8iGlv+sacrVEqsyRfnmX5613LZtdWw6vk+x6chNoxPG5U4WLXSZ5lpZ1A2OncZpGaPCEAoNINUd/4F57mO3l8U1NETSlBcIaROqFSsapzUkxyOAbWhzKy9KDzjcGpvP4YXDp1UxqoyuNJF2Ro45AK1ZhIknlNAntUXEkwvVvvYQJF2dMgA1fTFdAeLh7Xm7gjJhGkHI5EQfX2+42xxHlVRvgreO7D7CTySQq3WZ/bBEhXzydzaXt1TtpMabYo65irqv5C4B24uRs09hxIC7sshKIdWsXojhIa3LySmSetTOk898chpBJ2rTMXbwrYlku2maQE4M0INwT5w05CXaPupm/IEP4saMjp+6DXuHcg1TkUPRWBikXw8dBXuTAvKWm6tUzQVn/Mx7Lqxu7cn4YAq+EvDsBWbnQwfhZYthPd0Mr6cZSp4dlqbv5cXRiA73386h3ifCNauSOVFuJRFXCW9b2CDBvCjqSW2OiID+2cdtZMt4n78Xe2GZsvij9Z8otW8cpc728+2XiWxl+7m4rVhETbP7y4RQ1Mp+LJM/sc/1EK+Ptf55efe9pJanYfT/PLz9F865C6Xs7lz743n2J4B9mq3fCe4DJ3LgRA6cyGfkRJ5a/D1Ivwfp98uTfgfpnpb3FhYHLd8tlu7deO5JMGGyqntfzdacwuApFkW9W3yl8dHR0cBz+8ISdhmw7Gk4gi2C4YWmSVyu80hhwTuaIm8+VW9UVS49JfmtZjVXrKLZsXjNE6UYcerFbC1SxDe3G8px2sIDQtVTMT3K2/hxL6EnibxHJ0twH7mfCBsFIIsTn+0hSmgvLFTtva0U71obXPl4r8XUHocHOEBTtz+2N7e0r+Knd4qFrWhJa7zJ83gq/vM38LguGVPv+GhajO8TjfzKWyfZv6aR4utP28yGy22aKz//9O2VXH3TbPH1mFaLC2gbZjFd0kxQ2fNGXlBvfxXEfGG17/ssNVB4Ils0kLW7TufdRVf1umTSdgDN7Pg9uOpVr3wYJhQKmH+7GV6tUOofC37NEsjDbbNot+Y0FoTtwpMXJTIi4sSDjN3pKN0fcZhTsUicOJw3tFrC04ZENuVz2Sh9Bcc2sR8fffiyu794h7OUSr4s7fSMgtQkwlx2OaHxoKM6hAX8yDA3ohUTVyOL6bD7pWGdvfWuF4YJQaxXyMuf3nx/+fKnix8uv/v13buf3+x9PdhXfOtX6DLDcwIrTCzDcGiCVwf70XzuM5P6CfPNKPANFlvMcMPYCVw7iv14HiaJ7fssMJrJNe1FAgkqJ2HfkuaethN+J1tzrGQ2pH/qQiVVL48i31Pi6FDnwHXg5x2dnfIYMMop7krPURSZvDx9Qnt0/q67WLx/CGLiAERzIqVn67mVwI+mnLRQGhXx2zpf1QUXB5vMWpJSwJ9JS9PdTl/z2Yy3CqLtf0PY7+++e+b+5BGbewcNs20btCcYtoutCNH5f0AKLA27Hh7YYf8BuU6TTvEjiOhk1tE4w2RbRQUtExil1d0+KNM3EMbMQmbd0GwNQOykn90XR+J1WUGf882RdnKOurUFBju138CHO/fT9lBzwKXtpo+g6k969uYTDcWnmLP9nnxwbu2FAuLc2OicmIhqzoaNjKiAfyTbtkvbA0J5pNmd3fTjHWCpp572WNnbYRT7fFwt21vfBg2SxxUmmCyhbupbc6qq32uWMVw76IbyC8P5j4bMxLBHMm1posiDz5o/I3i2umnOeJ9Yzur2mVRR8/fZ+UhdOFl/86JxGo3kWnOxTmt2/s0yqlbPBsXcZ2iMHcKvANzaHeQx8JhbQBmeg+lWmgTt8kuy8IgYQI2VJYq5L+gSdyVLBrgBLC5RsYTQWhjm4YHF4XxpcX3kIm2A9ALxO9wf7EbQJGdAF2NdKgRm6uTgSsSRDMjL6s67iXL51xgNV3cgGK+EdNyNQItKjeKyLG4mxqahTWJXRR0c3qd5sQTGzVN0q9zRkaIZtQaaUfNpJnLC+Bb5XtN4uzM7K9QD8u/fHamYPHGOEXrSxAcqdIfq5Gk9sGx9lWtX0L9NruFpRuB5GN4OzX/zZDZxkvK7evljWaxXXe/to1Up6c1e9I6bvNMbDQSp2WM0DPKqa9JA3fmIg//fS5c5knVrrvxWSMjpcEarV2m3vtumL89e3ggPXXnBvXNlrKrQXqfG4yUDgxgh58i4ConmEhbp6lsQygYp+1vxCauu62Kld4zHILnKvPy8ggHmJATlG7y5+8e0frWOyPeilCmYpiU5vLuNoHUIkQCMRY/Pv3WesE0ao3yjQriDN2u8MWmLE2l/Px/S0hv0yEc1v/oDj2O+mvB0LS8o8YXvbLPznW1J39nWAg+d+bGmo29Uw4HqXIwLfPTsavFTor7rq56gHel02+v7/dNdD32OWo4NhfmQXQ8CDDBDG56+6YnyNF6eqHXCZbfRtHTSZbf54XWoG2aAt5B4ehhY1MUjc/whL5b2RJ/b6B92KhYd097rAVBRhfJvtGT/p9n/RvNz/o478rewyDHUM9/mq3Mecyw4fb53WObnp/hscKnxDIozlTQ2bDv5tEHuYAdFuELhhM9TPKxXVrgK//rupRbMSM7qRZHwQ+P17Pz/AwAA//8AowBc/zxpbnB1dCB0eXBlPSJoaWRkZW4iIGRhdGEtY3NyZj0idHJ1ZSIgbmFtZT0iYXV0aGVudGljaXR5X3Rva2VuIiB2YWx1ZT0iREZxWEJ3TVJlUWgyYlB5bldDMEd1WjM5b0UyUm1MbGdLZWZ5TVVidUFmS3AwNmtOeTZIbmorVWxsckRGOFFobHVSa0pjRCtKcnMyU0dBNGZCTG53RlE9PSIgLz7sG8uS3Lbxrq+AeXAuAocA39Luumy5YjklV1SOo6qcFBAEh/TyFZKzu6OUq/Ib+b18SboBcobkzEor27G95axGQ7DRABqNRj+Anidk9neR7IahqYksRd9fWslQ00YOhQTQrExTUW9VR4amKYeibVU6K9L61iLDvlWXVr9LqmKwnpDFn+gKQUuRqPLS+luzI9WuH0iiSF9sa+ipqKEzUolrRZqOtF3TNr0iMsche4ukYhA0LXqRlIreFkN+ter+or/ZmjHyIk1VfWkN3U5ZJFfFNh8uLRZY5KZQt180d5eWQxzCAqJhquuLBtCZzSxyW6RDbpD1iNiCyqZqm1rVw9TnyKaJRRN7hk70uWUIu2jFkJOsKEva7UpgirpRdZOm0O+l9U1g+4TZoS9s7sMHqHEYPCn8z7m9BsPnjZtT9w02eVd50NgBAAe4AAh8NCZMyfZfct3tDEoR6uu2L3xih5FHfP3tkEAj4YgvmGNz4Aj0wQwOPPVw33i2FweAGsz71d16MbeZX9qBrv4c0XUbHNfxbc+BVz/3bRZLO8ZGduBzageR7tpjSFkYYHsK7WdEj51Tm3lxScf+50wBRnmxzYFbFLs/qaJYp1tiz++sq4sNrsfVk4sNiMlSdC42RvixMmu66soA0+IG8caCWVMoTmuf0qxUdwRkTdFya553JUEgbbpUdZSbl23X3FLHmsa8SNUgirI/9AOCnja3NRnhtFO9Gg5vDYhnKfYkpUVdFrWiSdnIa+s4g4t+V1Wi25/Zuta4HUTfNu2unYR3vg9hj/UF7CgioElT99acNb+BHXWtEpHQvOmKd009iHKxuXAjRSQWTG8mI5UOdcnx3XEc4r77Bl8+jFYxlzgfxLpXlODVrMQoLBq2K9frTCtV78jijfagOfthj1pCc+sZYaHf3lkrOS2Ltc4DoDgZoBhUda+KJt+jhFUgVnRXl6rvaVuKAcV+ra71n16xCaNH8moYo39aCXkWP+9Udmnlw9D2zzabVPXXQ9PaW1DXu8SGRb9/kK2gsizk9aX1rQKpLIam2z8lTatqgtqejH1Zpywg5M8HrC8N1imbNmK96ZfcPGHuPYy1xiluXu+/azqZv8KtUBf1dtPuBwTQ8gDpxO2mEv2guk3ayH7TN7tOqs3bfhAg3ZuiEmDZNmWzbey23p7M7EsYs2xEuqJ7MREzjZkM7sqDotmMOmTcMkaPTepsRHoyKjWcGppc2I7qbjhsR9jZNGnSPWmpQ0DzJPS2EyBJnV4zbe0pNiBkW/QDoGrFh8ruoO2OGhMRqYQ9D81b6q5UTSuO3ofpBoeC3VZUW9J38mEsRxo/guefwRJdjlqxBF10XAvYzUDRjJdnrMGRlU+enFHs79fny3fwrLprixSg0r7fVe3QUK3sJ6y0EEDapPwmja/3Td4M12oPpC8V+5+gF/SnsBs9m0k3mVVZ9DtbbYIrPjM62sZpMyabclfVRNRFRTMBxg78tQwYrUdAmt83/sTGTyglf7D+Tii90uWLDQqF6JQAEu+q9krXbJoWjdHVaI4v8HuiEdQXDkYnDum6g5gagq3Rml1aWJKqBbHLRQfLcGn99bs/0sgilRryBpi9VcPclhZ1uxumobBrmHY9dM1o18UOPdQOrMI0aWpanFBVqDKdvGGzo0CJSpU3JUj2kjuf5qosi/b5e/hHcOCskbt+RuvosWshkCX4yuNq4qwXbvgDzC44DZoN8Pmq0fJ8n1O0EJxxR0xq5slRuzw5UTimfSWKermP5vWjTporjaxpUF/gMkBT0ArgY2kbSbNdWYIege0FU+qLG5gNLBRMG3FhikWdNXpS8+60ZQHRgGZg1KDRQsbRU+sUui/KAMotwkwZ1dEBWjfH1+8hjimy/aTaJpQJnKjhVoF9AikMSJuAW1hhKQtgUmXTaf1JewVUp7ipR+UHRmysHwFHjEkNHH2LEpWvdiCAsHo5J00n9EQZ1w+gzP8ouitNcoJVMy+2LKbB9X6AL1P/qWza/XPCHc7IV8Xwcpc8JV/X0l7a23ubg9VduRCgxGf+w0bVG/O2gYVUtG3AZ9iPIGAlOCm0yYBZ3U0h1Sj6+R5M+ORd/PPTf+ya4TlGY8Nb3Cfm/Zl5iFqUezAUva0RDPCpebRij+Z4xB/7kWJQW3BVFr38UQvtorHRSAusbYMbXNO8QNUaYIGJIvLsFBG84m1RgxjX27e7btlk4t+MdT+/+VxQswOmvy0m9tSwO3/44XQBaA6u46UV+hHnMpVxlAZRlLHIZYEUSZqphIWZ5/MgE5KrIM7SiGVByOIs4UkmlfKjLJEgKd8hO9Ab+kUkq+2KGyH3FPmgwGcfHoNojUQ/RLjOoT5i8QpE6HMn4UHoe6GT+Z7L3UgCyHH8KIzcIAyUHyaODLNU+dLPHClllClPummS8ci6em0Y8nEC9tsXCbAiu64YHiQTZ3EfsVDEXsADlTI3EIFkTDlCKBmmcZR5wlPSSaM4CWTsuW4oMj+QSRx7MU98cHa9kCWZtVIgs2lOrLKu/jKWfopmur29HRUTsmDX6yEeg8Ix5D5Itk4xH7FkeahhROy7InKdNHQzL1Fh6quUMxFyKdPAT5gfg5VzQ8djMgjSVMVeBOLmcy4yEBvNj7NCc1AsxxMSsyZPieE6xA/tU6LZ+iXMci2mKztnXSHSciRzbGCKYhGGvGwq1QKXIKQoBnSujU9nLd3zw11BOvmfINn6oJJU2qf07t87R5fyA6eN3PuI00ZE/vjTRoiQr6kh7UGn+BFxXri2H+HhIBkLkdRFj3CbxySwfZf4theS0PZj27MdePrUZubhQgt4iamN59YR1088+qYcSrYbwtN3qQEEgBZ7gIKn2wiMzDs0g6kDgAOA+Xi+Tm3f9AWt7cDFImG2EyGJDBpDGzvkWER4FMIH6HUBNwAKsbXuDCp8pAdAzA4BwCmg4JCxKbh27OMMANHVp/kxUsM1FY7Gt90AO+D4CnCuWWUHMAYMDaMgJdgjQ8JcwOdACX7DLPSNgQsUxYQTrGDIDOhNMxcnPj1tz4NK+AciYcc4GMHBgHrbDxCPYY8an/l6rfSUYNauHfo4F5/gXGDJ8CbB9+xQryEshaECmaJ5GBtmwrg4Dw5DwurCKvuwlp9HUOUS843nxUAOiAP1bA8Z50dU/zs9Qj4c6v2qMd3HmKh+17ZNN8z0ymeD2PaXaTNAmRq18BhMlj5YkMNDbNY51EdstIIgDrgQTuYEPI4hAONcsdRnUmZSuH6aRMzPvFSqzPGd0HVCHrHAi+IwcyPQzyAVLwxDxjj/p3g8MxZBJCKBD49Bdl4bUh8iO+dQH7HspGmg4ozFnoycWGVJmHiR5yR+FvOIhzJkIsgymUVZ6Pt+nLmJh36QC063ZICtdHyFDFkLzY8O3x+DvIi2eIisrNEesZwwHjMQC+GgcnG4DFjiJ3EgXS8LMnSXvczzIhAdEJnYS1yehlJ6mQpiziPlc+vq89df/3gZGY8BH5ucDJ0o6gcqlrO4j1hiMjAxaZqxzPFk6jLHcQLXF8JzAsESV6YKQifuxFEqGIuZDKXnqBC0ClgzplwmrKvvRo78DPYowUu3RyAwSOdDhOUE7xELShQmTpCg8Ykdnvks86QbZannuhBlMwZSw7IwSJXwOITjwnViyRzupNyTAQ95bF19AUT8uIhbJM1uGENuXX7P0ZCpv/ocH+ei7vnl8ml60Tmnvk1ocLgKnl2bL2+UAHMoRLefX2WvcwAOt/8QzIrvxR1VXQftK9X3OuYfOz6tArpEn5tvU2MRE7VPdP3qmUOiBAY8LIq3Oddpa94sjQyCWogpnVcMwm4ITV0IsPm8GtpAaBrlEGSCV7yqwag3enXs912F8TPG4aETSgg2Y4pROIaxHg/HMsSkUUCcEuJdDFgx+o+WKXWMeabOfwmBKIvEshbjb8/FE4PQfxXYno8Bs+OF776JoTsB0avO7qMYuZoXB+NxIA4zD/1lMqHOGiTOjU5GPEINEMc4nwg13UqbC2jzcjws0uKi76rxxnwmVmnRV0XfL6/AvzRAYuTrocdEv4Bo3T1IrFw8YcGveTImLAis8auIBHbslS7KCn7N8ioNCn69irEUaSyyxqJHtIggYknP4DkGT3+9wiFJpEkinh1G6yRR5548yWMqAL5h5q4U9X/+9e+BtKrT+RdDLoYx54JAaciLngxFpezlTf9MvcHyg0EoFe1BoeB9/EyhjGrlVnT1WExEXauDitFk/F/P/L71zNzwnRcmavLKaVEfbBPK7jHbXCcMiroZckxpFwmmnZv8RpscPELr6luFHhuabjT9AOwUyOge3BoyjmfPk8U+gjLjOJyQBmAk72em7OBlQBGT/UpwOrXh17fdZxLNfno+Gylz6DgTu3KYeyZtV+j0tfzubddDQIY5nqvMNpPIM5mPuVF4oa3HROb/Mq/NkDd1Nb5VTXpIQj751YROQ5v9cqLCpMnZ+yH5SCQ9jAr871BNAdbQtBCHrKzmvdM+zfmap4P/6hrxd2Ye1wZyaeiAKQ3Ed7RvC7RhpNrTAJ2fpaRO1TjAcZu+P9sOcyfNNj7z64TXTYvCjiPlWJCiS+mYGHcqh4fsc/DDoMf9M4JXDs9RE2H+4fhqoSoCLaRQck4y68YRDxHC6p3SpAEeVbRU2XBaWYoO2uBGNYqizwVoOwNe5ca7gdPePbfOxDNndFuNee8DRnFt0hgetHs67rD1FN6prpljjimHH9y343ItB1luXwAc9suLCe8Loz1AReA6rZpXYD9bPcQxe78BsYHlmRQATiVTKk0EhqgwRqHST8bKsRmITqfGZNjb35IP3eIRwWraUIIZjaqTP0iL6B3KFv6EKd+40iG2B56M6wYGBJ/cW7oZePmHyNRdd4H/cwro7yoEu2/AFcndN15OXXBkuB2FHnhuQbhuh3exPF45VI7WdP5NbPtIVRwE5hdWBxz8yiPbXzVjHnqEHOiDlksvTF+6eug8MX/+oywkgWgS5j8hYzywOQuRgNVPpbCc0+gUTg/jrjrCm9vwXlX460uWzJW8XkkWgoxgzXNud1KC/pmSFh4qcrAiYQQWYmlI9CUxWJkQWRWufphHR7vG0QTFNo/WNm+0QLCAaPdAOjgNzprG+yK0pfJY6MaZlRjzsjFVHmvyocK0j/8CAAD//wEAAP//6Ps4Ajf5AQA=){height=\"60px\" width=\"240px\"}"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", "   :header: Tutorial 4: Inception, ResNet and DenseNet\n", "   :card_description: In this tutorial, we will implement and discuss variants of modern CNN architectures. There have been many different architectures been proposed over the past few years. Some...\n", "   :tags: Image,GPU/TPU,UvA-DL-Course\n", "   :image: _static/images/course_UvA-DL/04-inception-resnet-densenet.jpg"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab_type,colab,id,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7"}, "papermill": {"default_parameters": {}, "duration": 34.215563, "end_time": "2021-09-16T12:38:28.107448", "environment_variables": {}, "exception": null, "input_path": "course_UvA-DL/04-inception-resnet-densenet/Inception_ResNet_DenseNet.ipynb", "output_path": ".notebooks/course_UvA-DL/04-inception-resnet-densenet.ipynb", "parameters": {}, "start_time": "2021-09-16T12:37:53.891885", "version": "2.3.3"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"00d2536fa8f84cb682fda61b782facff": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_0ca470bfecac44febea68756f24bf7ed", "IPY_MODEL_784ea8b6d85b4f89b0044c162f457e6e", "IPY_MODEL_b3c631fdb1d847359d8921cb27e83022"], "layout": "IPY_MODEL_a09ed7bbf77f4feca0e6d376ac171ac5"}}, "0203afd7101649028c6435f93170aee1": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "02b72077e1c247e8951b28cbe101709e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_c2e35d29d4e44b50b062f1df467eed57", "placeholder": "\u200b", "style": "IPY_MODEL_961fb7544009416fb4bc2b0fa7527231", "value": "Testing: 100%"}}, "042a54a083af49a0b554cb82a7789f96": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "057c559633fc4af3bb30492a2bd70742": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_1323f69330da40fea47fac9a724ddf72", "placeholder": "\u200b", "style": "IPY_MODEL_f462f71441e141b9985ac017c18f01d7", "value": " 40/40 [00:00&lt;00:00, 91.92it/s]"}}, "0ca470bfecac44febea68756f24bf7ed": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_7c4795c7452e477c8af22f2a3952fe8e", "placeholder": "\u200b", "style": "IPY_MODEL_0203afd7101649028c6435f93170aee1", "value": "Testing: 100%"}}, "0ccaa7e4ce5b4da3a8506e93d8ad374e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_851af2c5099445b7bd1edb5db52176d3", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_fc9939dfb2924319a7e20350adf1d2cf", "value": 1.0}}, "0dad7007ea8a484fa3da413cffca2ceb": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "0f2221509f3548c0bd9e405861a50221": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "0f3fee89ad31415ca7b5c0dc7bbdb22d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b7553f5ca2d541f4831ec58be3b18ef4", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_378d21754d3245e9b088ff80b5b17df7", "value": 1.0}}, "102f8308e96d458fb9d3e1852382c669": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "1323f69330da40fea47fac9a724ddf72": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "15861526722d44debb3b4c59cf080d83": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_f27665e646704f9ca9ec4cf92ac4776c", "placeholder": "\u200b", "style": "IPY_MODEL_0f2221509f3548c0bd9e405861a50221", "value": "Testing: 100%"}}, "1a23deac54d64e7aacfcc3dfe2dfbccd": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "1f10c6d6685344bab0397dc7b9f3538b": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "208111d43b2547a599fdcd62ee551ad4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_02b72077e1c247e8951b28cbe101709e", "IPY_MODEL_dc1b8a933fd9447fb4b678468c913fcd", "IPY_MODEL_f92a835c5a4b4ecf90c50a07fa6e7f2b"], "layout": "IPY_MODEL_f790196a03e442dbbbd8c44f5d298e2d"}}, "21830d7003f44e2ca6f267a5673cef44": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "28079eec78bd4933971ada11232f56e7": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "280b8c5f3e374e48ae6b5ec8c3b5fc2a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_50c41d246e074c7792dd0d30efb6f69e", "placeholder": "\u200b", "style": "IPY_MODEL_63478e6a162f4bd6a4c2dc5948c774b2", "value": " 40/40 [00:00&lt;00:00, 59.05it/s]"}}, "2a9550b5cf1540f691caa1cbd3509fab": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "2f4fba3dcfc04d2ead27c6c6277f0d0e": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "30634bfeaf3541f881e1eab7a041c9f0": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_37cf4442d9d44893ac38e75407c666de", "placeholder": "\u200b", "style": "IPY_MODEL_042a54a083af49a0b554cb82a7789f96", "value": "Testing: 100%"}}, "33eb164d6f8446399823b259486ff7ba": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_15861526722d44debb3b4c59cf080d83", "IPY_MODEL_0ccaa7e4ce5b4da3a8506e93d8ad374e", "IPY_MODEL_97ad9a436d8b4639845537362e8b6ddf"], "layout": "IPY_MODEL_3d1589c360e9428b9396765b0408550d"}}, "378d21754d3245e9b088ff80b5b17df7": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "37cf4442d9d44893ac38e75407c666de": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "38863e0c240347e99551ab1c627e9ef3": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "3d1589c360e9428b9396765b0408550d": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "433e4ec37f6c461faff8900075cecb75": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "435866984af24cbea2bee95b9574e50f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_6fe3642eec8e431fb490b44a09e16037", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_7845c2ebaa7d4331b00ad896b6e9e465", "value": 1.0}}, "43b4aa5e9b654d498238b15da523a0db": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_bf3f4380798f45828a92a41f76280163", "placeholder": "\u200b", "style": "IPY_MODEL_c3e1371765df4379b1d4d180eab942a5", "value": "Testing: 100%"}}, "470e5f9826e0410abafaec637b28da46": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "4a39e78e773e4ef894a35bb3cfc22a7e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "50c41d246e074c7792dd0d30efb6f69e": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "583731e705914b0eaddda1019d5d5b9e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "62f007cb989544e1888350cd397dd348": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_30634bfeaf3541f881e1eab7a041c9f0", "IPY_MODEL_ce17aa3e5eb4459f948d03cd53393e01", "IPY_MODEL_b144da2341c747509c583f8d85731f7c"], "layout": "IPY_MODEL_7195282dfb5b40dc85d32df1d3a30245"}}, "63478e6a162f4bd6a4c2dc5948c774b2": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "6fe3642eec8e431fb490b44a09e16037": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "7195282dfb5b40dc85d32df1d3a30245": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "73bd8e95d78d49b3b4407ba0b31f203f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_102f8308e96d458fb9d3e1852382c669", "placeholder": "\u200b", "style": "IPY_MODEL_db6f2a05cab24e9c85ed0e77330ced31", "value": "Testing: 100%"}}, "782447cfdeb548a2a5aa0aa64586e890": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "7845c2ebaa7d4331b00ad896b6e9e465": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "784ea8b6d85b4f89b0044c162f457e6e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_470e5f9826e0410abafaec637b28da46", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_583731e705914b0eaddda1019d5d5b9e", "value": 1.0}}, "7c4795c7452e477c8af22f2a3952fe8e": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "7d2c120d53a44c438d14a356bfc592d2": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "7ece9c5c517e4dfe9131a5a78db598a5": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_73bd8e95d78d49b3b4407ba0b31f203f", "IPY_MODEL_435866984af24cbea2bee95b9574e50f", "IPY_MODEL_280b8c5f3e374e48ae6b5ec8c3b5fc2a"], "layout": "IPY_MODEL_df8c653340514e778f0ed63e5f936cd7"}}, "839f3c6b5dae468382177e2c3585a3fa": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "851af2c5099445b7bd1edb5db52176d3": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8c1aba25160f46f2a9b1890e6922313a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_0dad7007ea8a484fa3da413cffca2ceb", "placeholder": "\u200b", "style": "IPY_MODEL_4a39e78e773e4ef894a35bb3cfc22a7e", "value": " 79/79 [00:01&lt;00:00, 65.01it/s]"}}, "8eb1ac45df8148bcb27983b629002641": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8ee50e576d954731bc435879e155e74f": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8ef99bb7da534d18b9ee15902fa411a6": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_a1674b1a32174d99943e7c45aad95247", "placeholder": "\u200b", "style": "IPY_MODEL_e55ac5380eb44acfa3af5e84ce8e56c1", "value": " 79/79 [00:00&lt;00:00, 130.13it/s]"}}, "927c69f50d9d491c95e92f4fbc5f84b5": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "961fb7544009416fb4bc2b0fa7527231": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "96ad9a476be744bc9723a9fbf21c47a8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_c02cde8683cc46be84ecef98fbb0488f", "IPY_MODEL_f3070c8e69f545d4818b6d5b165e3bab", "IPY_MODEL_8ef99bb7da534d18b9ee15902fa411a6"], "layout": "IPY_MODEL_21830d7003f44e2ca6f267a5673cef44"}}, "97ad9a436d8b4639845537362e8b6ddf": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_a2b9d6f935e243099d1f4190b332fb6d", "placeholder": "\u200b", "style": "IPY_MODEL_782447cfdeb548a2a5aa0aa64586e890", "value": " 79/79 [00:00&lt;00:00, 125.85it/s]"}}, "9f70aa34468440bbb446044e610ae62e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8eb1ac45df8148bcb27983b629002641", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_c63f81881af541779d23ee5d91dd4b49", "value": 1.0}}, "a03e388e9a8542fdb879274e69b9bbe4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_fea5d8f25909491f9d02434f466df50b", "IPY_MODEL_9f70aa34468440bbb446044e610ae62e", "IPY_MODEL_8c1aba25160f46f2a9b1890e6922313a"], "layout": "IPY_MODEL_a153accab5e94a819eec55e23fca9389"}}, "a09ed7bbf77f4feca0e6d376ac171ac5": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "a153accab5e94a819eec55e23fca9389": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "a1674b1a32174d99943e7c45aad95247": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "a2b9d6f935e243099d1f4190b332fb6d": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "b144da2341c747509c583f8d85731f7c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_f70fef6aa3d64e3fa63bb6046fd9edac", "placeholder": "\u200b", "style": "IPY_MODEL_bb5f2159834f40cfb5cb7d7fad4f3e21", "value": " 79/79 [00:01&lt;00:00, 70.25it/s]"}}, "b3c631fdb1d847359d8921cb27e83022": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_2a9550b5cf1540f691caa1cbd3509fab", "placeholder": "\u200b", "style": "IPY_MODEL_927c69f50d9d491c95e92f4fbc5f84b5", "value": " 40/40 [00:00&lt;00:00, 61.55it/s]"}}, "b7553f5ca2d541f4831ec58be3b18ef4": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "bb5f2159834f40cfb5cb7d7fad4f3e21": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "bd7944d0ee914b3b9cd70fc912ad5346": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "bf3f4380798f45828a92a41f76280163": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "c02cde8683cc46be84ecef98fbb0488f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8ee50e576d954731bc435879e155e74f", "placeholder": "\u200b", "style": "IPY_MODEL_839f3c6b5dae468382177e2c3585a3fa", "value": "Testing: 100%"}}, "c08f2cbc7f98400baf0aa65975955f62": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "c2e35d29d4e44b50b062f1df467eed57": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "c3e1371765df4379b1d4d180eab942a5": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "c63f81881af541779d23ee5d91dd4b49": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "c9b73c4d4a1d4fcdad7106c9a6b32ea2": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "ce17aa3e5eb4459f948d03cd53393e01": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_c08f2cbc7f98400baf0aa65975955f62", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_c9b73c4d4a1d4fcdad7106c9a6b32ea2", "value": 1.0}}, "db6f2a05cab24e9c85ed0e77330ced31": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "dc1b8a933fd9447fb4b678468c913fcd": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_2f4fba3dcfc04d2ead27c6c6277f0d0e", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_433e4ec37f6c461faff8900075cecb75", "value": 1.0}}, "de6156dcb2674899bbaa068149530b80": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_43b4aa5e9b654d498238b15da523a0db", "IPY_MODEL_0f3fee89ad31415ca7b5c0dc7bbdb22d", "IPY_MODEL_057c559633fc4af3bb30492a2bd70742"], "layout": "IPY_MODEL_7d2c120d53a44c438d14a356bfc592d2"}}, "df8c653340514e778f0ed63e5f936cd7": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "e55ac5380eb44acfa3af5e84ce8e56c1": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "e7cc9d9ed37e4afabc5ceea1aa244870": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "f27665e646704f9ca9ec4cf92ac4776c": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "f3070c8e69f545d4818b6d5b165e3bab": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_1a23deac54d64e7aacfcc3dfe2dfbccd", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_e7cc9d9ed37e4afabc5ceea1aa244870", "value": 1.0}}, "f462f71441e141b9985ac017c18f01d7": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "f70fef6aa3d64e3fa63bb6046fd9edac": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "f790196a03e442dbbbd8c44f5d298e2d": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%"}}, "f92a835c5a4b4ecf90c50a07fa6e7f2b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_38863e0c240347e99551ab1c627e9ef3", "placeholder": "\u200b", "style": "IPY_MODEL_28079eec78bd4933971ada11232f56e7", "value": " 40/40 [00:00&lt;00:00, 101.99it/s]"}}, "fc9939dfb2924319a7e20350adf1d2cf": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "fea5d8f25909491f9d02434f466df50b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_1f10c6d6685344bab0397dc7b9f3538b", "placeholder": "\u200b", "style": "IPY_MODEL_bd7944d0ee914b3b9cd70fc912ad5346", "value": "Testing: 100%"}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}