{"cells": [{"cell_type": "markdown", "id": "7a37951c", "metadata": {"papermill": {"duration": 0.023386, "end_time": "2021-09-16T12:40:39.258673", "exception": false, "start_time": "2021-09-16T12:40:39.235287", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 7: Deep Energy-Based Generative Models\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2021-09-16T14:32:29.871712\n", "\n", "In this tutorial, we will look at energy-based deep learning models, and focus on their application as generative models.\n", "Energy models have been a popular tool before the huge deep learning hype around 2012 hit.\n", "However, in recent years, energy-based models have gained increasing attention because of improved training methods and tricks being proposed.\n", "Although they are still in a research stage, they have shown to outperform strong Generative Adversarial Networks\n", "in certain cases which have been the state of the art of generating images\n", "([blog post](https://ajolicoeur.wordpress.com/the-new-contender-to-gans-score-matching-with-langevin-sampling/)about strong energy-based models,\n", "[blog post](https://medium.com/syncedreview/nvidia-open-sources-hyper-realistic-face-generator-stylegan-f346e1a73826) about the power of GANs).\n", "Hence, it is important to be aware of energy-based models, and as the theory can be abstract sometimes,\n", "we will show the idea of energy-based models with a lot of examples.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [![Open In Colab](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHUAAAAUCAYAAACzrHJDAAAIuUlEQVRoQ+1ZaVRURxb+qhdolmbTUVSURpZgmLhHbQVFZIlGQBEXcMvJhKiTEzfigjQg7oNEJ9GMGidnjnNMBs2czIzajksEFRE1xklCTKJiQLRFsUGkoUWw+82pamn79etGYoKek1B/4NW99/tu3e/dquJBAGD27NkHALxKf39WY39gyrOi+i3xqGtUoePJrFmznrmgtModorbTu8YRNZk5cybXTvCtwh7o6NR2KzuZMWNGh6jtVt7nA0ymT5/eJlF9POrh7PAQl6s8bGYa3PUum//htmebVtLRqW0q01M5keTk5FZFzU0oRle3+zxwg5Hgtb+PZiL/ZVohxCI+hL5JgjmfjPxZ26+33BG3dA+ealHPM4gQAo5rU59gsI8bRvl54t3Ca62mvHyUAhtOlLd5WSQpKcluBjumnoCLs1EARkVd9E8l3p9y2i7RbQ1B6pFwu/YDgW8KbHJHMTQrwnjz2oZm9M4pavOCfo5jWrgCaaMVcMs6/pNhDr0+AMN93XlxV7R6DNpyzi7W/OE+yIrsjU6rTrbKV5cd/pNyItOmTbMp6sbBB+EqaYJY4cWE3VUciNt1TpgfcRFv71Fi54xT5kSoyLvOBEJMOMxWXkFlBeBSX4u6Zkcs+3KszYRtiapbNRqF31UgetVuc8z9vBXIv1qD+F1f83B6uDlCUyfsZGepGPpmg01OB7EITQbhS9ribKy+DmP1DUiClLz4bnIHVOqa7BY+Z1wg5g3zgUvyehiNpnJKxSLc/ts76LKm0BzX3c0RNy1yXjDcB5lWoro4iNHQxM+f1kWeWQARAWQS++trISJTp061Kep25X/MycwtjuctSC5rxo7ppi7VNUox5+PhPHtrsS2O1qJ6yx1QujQUzm9sh6hbkBlvvGcN8hYnwjUjH6kjfZEd5c/jitz5Jc5U3ENnFynKl4eB7nyEgP2UZ+Yz3/rVEbyYr27qELrtC4FIC0J7sc7xWnmccdHfRRTs0VB+cA4lt+oFcRR/wUeH8FG5w2Mbx8FQ8TXEvv1xYf4wBP3O2WyL3/UVjpXWgIqaFeUPr+wTmDvUB7njH6/bOv+HRg4SqioAg5GDe1aB3ZeMTJkyRSBqkLsWqSEm0fZVBEN94zEZnYvrdx1JL5cxe+a+AbhSJecRRHW/ikTFRTa38dtQlNZ5CRKwFvUtZU/kvBoEF9Uxni/XqIM+dwKbTw3rhcxIf7gmr2M+H6SMwx8iBzJbw5oxeG3Lv5FX9B3AGaHPS8e8z77H7v9VMpvPG5ug1enh7eGK8h0LBTwUb+GInqzInlRUK65DmTPQu4c3+uQKjwKK77zwUxBX4Tq7yR1RuiwUsqlrABCM6esHdXoy47fk4+prYKy8ZF574x4V5BnHQBuf4g9Z9ld8U36L2aktZNNplNfw7zotwWTy5MkCUft4aLEopJj5/OPHl1BQqeAVOnHgNSQOqmBzq9V9cfEm/yx5ubMGKS9cYPZ3vx2OS/c6PVHUuUO7Y1Pci3BO/1zgq18byebfGemLtNF+6JRtOvMk926ibussZqM+1mNz4TWkH7rCbM5phwGRGDAaoF8fY5OHFnlldAA8sgoEXKnDukA1NgSeNjqkJT9brbN4pC9WRweYXyLugR73c+MYvyWfu0yC6+mjzN1Isfw3FKJS98CU/zI1IHFkFPR52cHL2FJk0sB6kMTERIGo9GzcPkLNfA0cwdwi/hfEYO86ZMd9w+y1egfM2T2Eh/vesMNwljSzuZRT420SW3eqy8N6aHMmwmnFUZ7/PGVPbIoNZvNU1BURdHs0bT2+HjL8sDSM2e6vi4Lj5NW8WOLVA6RTT2azxLV+bglaFNqLieqemS/gWkw7NyoAHo+2dEsiivengjKsPFoqWOvbSh/kxPaxyW/JRzH2Fl3EzD9/xjAefJqB3usKUFn/0Gb+S/d/jy3FN2yLOmnSJJtn6oehByEiHPSeXnDxFGPRnoFoaBJjcdQlbDwcjL1zTNuQpoxD7R0OG0uUTMi0fkVwdzBdYIwcwZunxrVJVLplNm54BZp7jfDfYLoNyqQi1K6KxIdHzmN+QQ2WjFIwUT2zTGdlRXo4NFXVUO4sgX5dFC7f0aP/ZlNeUjFBuL8Xjl6uRuP6aMjSjpjzsH62FDU7JhBuGccEXIvDfJFFBc/gHw80dklfCVYnRaDfpiJcutPA4F7qJsfJeUPQI+1fqMlNhFx1FM0GDqkjFVg7NojlQ0Vt4aM5ReSqcbpaCg8nCW5lRsBvbT4T1TLfFptsfh7gItzuKTdJSEiwKSrt1vcmnEXXrsLbYnWDA1bu+z2WKy9Arq+1KRqdfKsoBo0GcdtEpS/B1bO4v0cFiUhkjskvKcMrWwtAPHuwQq8Z+4LZ1vTQANfXt4J0DwZX9gWa9qh4XDM/voC9JXfwYEMMHJcfNtusn82ihvliVUwg5KrPGVf6GH94ZJpEZBen6EC4qYTHA1dXhW0JIex8txzv//c8lhzXIi/BFxOH9jGbQhZsRalTIBZZ8KkGyZAxeRQvXkFF1TWz/Hm46jNYUnjPbt3JxIkT7f6dSj8qfJJyVvBxgaIlblOyjtysNHWN9fjjqWi7glJfW3/S0Hlj2XnA8PhKT9w6g3Qx3XiXhvuxQsuT1proxBKI/AaZqY1Xz5muvY8G8XkRRCaHsfQsRAFDH/tZPbcYuHotOG0FRIqB4HR3wNVoIPLtz8ycTguu+jpEigE218vd1YCr5m+HpHMvEI9u4LTXwNWaLjl0iPwGAmIpeHx1VeCqTJdPs1/vweweQPO3HC24NhOhnTphwoQnfv6QSY2ICbkNmdSA4h87oaLaiYfn5diIEd4att2erOwJXbPUHp953p6orQVSUVWRAXBT8c/dJ5L9xhzaJGp71GR/wFP8P5V2z10NSC9T93QM2xUg8fHxT+zU9ijeU4naHon8CjFJXFzc8/kn+dN06q9QgF98SYSo2Xen2NjYZy5sR6f+4nLSK5Iam2PH/x87a1YN/t5sBgAAAABJRU5ErkJggg==){height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/07-deep-energy-based-generative-models.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/latest/)\n", "| Join us [on Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)"]}, {"cell_type": "markdown", "id": "7f00f90d", "metadata": {"papermill": {"duration": 0.021442, "end_time": "2021-09-16T12:40:39.301749", "exception": false, "start_time": "2021-09-16T12:40:39.280307", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "33e8ad5c", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2021-09-16T12:40:39.347844Z", "iopub.status.busy": "2021-09-16T12:40:39.347375Z", "iopub.status.idle": "2021-09-16T12:40:39.349918Z", "shell.execute_reply": "2021-09-16T12:40:39.349436Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 0.026972, "end_time": "2021-09-16T12:40:39.350031", "exception": false, "start_time": "2021-09-16T12:40:39.323059", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# ! pip install --quiet \"torchvision\" \"torch>=1.6, <1.9\" \"tensorboard\" \"matplotlib\" \"pytorch-lightning>=1.3\" \"torchmetrics>=0.3\""]}, {"cell_type": "markdown", "id": "9eea4d0b", "metadata": {"papermill": {"duration": 0.022362, "end_time": "2021-09-16T12:40:39.394135", "exception": false, "start_time": "2021-09-16T12:40:39.371773", "status": "completed"}, "tags": []}, "source": ["
\n", "First, let's import our standard libraries below."]}, {"cell_type": "code", "execution_count": 2, "id": "8882a3d0", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:39.460040Z", "iopub.status.busy": "2021-09-16T12:40:39.459550Z", "iopub.status.idle": "2021-09-16T12:40:40.727064Z", "shell.execute_reply": "2021-09-16T12:40:40.726625Z"}, "papermill": {"duration": 1.298062, "end_time": "2021-09-16T12:40:40.727182", "exception": false, "start_time": "2021-09-16T12:40:39.429120", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/tmp/ipykernel_1940/3480345581.py:30: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n", " set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "Global seed set to 42\n"]}], "source": ["# Standard libraries\n", "import os\n", "import random\n", "import urllib.request\n", "from urllib.error import HTTPError\n", "\n", "# Plotting\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "# PyTorch Lightning\n", "import pytorch_lightning as pl\n", "\n", "# PyTorch\n", "import torch\n", "import torch.nn as nn\n", "import torch.optim as optim\n", "import torch.utils.data as data\n", "\n", "# Torchvision\n", "import torchvision\n", "\n", "# %matplotlib inline\n", "from IPython.display import set_matplotlib_formats\n", "from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint\n", "from torchvision import transforms\n", "from torchvision.datasets import MNIST\n", "\n", "set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "matplotlib.rcParams[\"lines.linewidth\"] = 2.0\n", "\n", "# Path to the folder where the datasets are/should be downloaded (e.g. CIFAR10)\n", "DATASET_PATH = os.environ.get(\"PATH_DATASETS\", \"data\")\n", "# Path to the folder where the pretrained models are saved\n", "CHECKPOINT_PATH = os.environ.get(\"PATH_CHECKPOINT\", \"saved_models/tutorial8\")\n", "\n", "# Setting the seed\n", "pl.seed_everything(42)\n", "\n", "# Ensure that all operations are deterministic on GPU (if used) for reproducibility\n", "torch.backends.cudnn.determinstic = True\n", "torch.backends.cudnn.benchmark = False\n", "\n", "device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device(\"cpu\")"]}, {"cell_type": "markdown", "id": "8fa2ad82", "metadata": {"papermill": {"duration": 0.022316, "end_time": "2021-09-16T12:40:40.772238", "exception": false, "start_time": "2021-09-16T12:40:40.749922", "status": "completed"}, "tags": []}, "source": ["We also have pre-trained models that we download below."]}, {"cell_type": "code", "execution_count": 3, "id": "8ddb6202", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:40.823433Z", "iopub.status.busy": "2021-09-16T12:40:40.820802Z", "iopub.status.idle": "2021-09-16T12:40:41.030878Z", "shell.execute_reply": "2021-09-16T12:40:41.030402Z"}, "papermill": {"duration": 0.236743, "end_time": "2021-09-16T12:40:41.030985", "exception": false, "start_time": "2021-09-16T12:40:40.794242", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial8/MNIST.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial8/tensorboards/events.out.tfevents.MNIST...\n"]}], "source": ["# Github URL where saved models are stored for this tutorial\n", "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial8/\"\n", "# Files to download\n", "pretrained_files = [\"MNIST.ckpt\", \"tensorboards/events.out.tfevents.MNIST\"]\n", "\n", "# Create checkpoint path if it doesn't exist yet\n", "os.makedirs(CHECKPOINT_PATH, exist_ok=True)\n", "\n", "# For each file, check whether it already exists. If not, try downloading it.\n", "for file_name in pretrained_files:\n", " file_path = os.path.join(CHECKPOINT_PATH, file_name)\n", " if \"/\" in file_name:\n", " os.makedirs(file_path.rsplit(\"/\", 1)[0], exist_ok=True)\n", " if not os.path.isfile(file_path):\n", " file_url = base_url + file_name\n", " print(\"Downloading %s...\" % file_url)\n", " try:\n", " urllib.request.urlretrieve(file_url, file_path)\n", " except HTTPError as e:\n", " print(\n", " \"Something went wrong. Please try to download the files manually,\"\n", " \" or contact the author with the full output including the following error:\\n\",\n", " e,\n", " )"]}, {"cell_type": "markdown", "id": "54b9725b", "metadata": {"papermill": {"duration": 0.02216, "end_time": "2021-09-16T12:40:41.075989", "exception": false, "start_time": "2021-09-16T12:40:41.053829", "status": "completed"}, "tags": []}, "source": ["## Energy Models\n", "\n", "In the first part of this tutorial, we will review the theory of the energy-based models\n", "(the same theory has been discussed in Lecture 8).\n", "While most of the previous models had the goal of classification or regression,\n", "energy-based models are motivated from a different perspective: density estimation.\n", "Given a dataset with a lot of elements, we want to estimate the probability distribution over the whole data space.\n", "As an example, if we model images from CIFAR10, our goal would be to have a probability distribution\n", "over all possible images of size $32\\times32\\times3$ where those images have a high likelihood\n", "that look realistic and are one of the 10 CIFAR classes.\n", "Simple methods like interpolation between images don't work because images are extremely high-dimensional\n", "(especially for large HD images).\n", "Hence, we turn to deep learning methods that have performed well on complex data.\n", "\n", "However, how do we predict a probability distribution $p(\\mathbf{x})$ over so many dimensions using a simple neural network?\n", "The problem is that we cannot just predict a score between 0 and 1,\n", "because a probability distribution over data needs to fulfill two properties:\n", "\n", "1.\n", "The probability distribution needs to assign any possible value of\n", "$\\mathbf{x}$ a non-negative value: $p(\\mathbf{x}) \\geq 0$.\n", "2.\n", "The probability density must sum/integrate to 1 over **all** possible inputs:\n", "$\\int_{\\mathbf{x}} p(\\mathbf{x}) d\\mathbf{x} = 1$.\n", "\n", "Luckily, there are actually many approaches for this, and one of them are energy-based models.\n", "The fundamental idea of energy-based models is that you can turn any function\n", "that predicts values larger than zero into a probability distribution by dviding by its volume.\n", "Imagine we have a neural network, which has as output a single neuron, like in regression.\n", "We can call this network $E_{\\theta}(\\mathbf{x})$, where $\\theta$ are our parameters of the network,\n", "and $\\mathbf{x}$ the input data (e.g. an image).\n", "The output of $E_{\\theta}$ is a scalar value between $-\\infty$ and $\\infty$.\n", "Now, we can use basic probability theory to *normalize* the scores of all possible inputs:\n", "\n", "$$\n", "q_{\\theta}(\\mathbf{x}) = \\frac{\\exp\\left(-E_{\\theta}(\\mathbf{x})\\right)}{Z_{\\theta}} \\hspace{5mm}\\text{where}\\hspace{5mm}\n", "Z_{\\theta} = \\begin{cases}\n", " \\int_{\\mathbf{x}}\\exp\\left(-E_{\\theta}(\\mathbf{x})\\right) d\\mathbf{x} & \\text{if }x\\text{ is continuous}\\\\\n", " \\sum_{\\mathbf{x}}\\exp\\left(-E_{\\theta}(\\mathbf{x})\\right) & \\text{if }x\\text{ is discrete}\n", "\\end{cases}\n", "$$\n", "\n", "The $\\exp$-function ensures that we assign a probability greater than zero to any possible input.\n", "We use a negative sign in front of $E$ because we call $E_{\\theta}$ to be the energy function:\n", "data points with high likelihood have a low energy, while data points with low likelihood have a high energy.\n", "$Z_{\\theta}$ is our normalization terms that ensures that the density integrates/sums to 1.\n", "We can show this by integrating over $q_{\\theta}(\\mathbf{x})$:\n", "\n", "$$\n", "\\int_{\\mathbf{x}}q_{\\theta}(\\mathbf{x})d\\mathbf{x} =\n", "\\int_{\\mathbf{x}}\\frac{\\exp\\left(-E_{\\theta}(\\mathbf{x})\\right)}{\\int_{\\mathbf{\\tilde{x}}}\\exp\\left(-E_{\\theta}(\\mathbf{\\tilde{x}})\\right) d\\mathbf{\\tilde{x}}}d\\mathbf{x} =\n", "\\frac{\\int_{\\mathbf{x}}\\exp\\left(-E_{\\theta}(\\mathbf{x})\\right)d\\mathbf{x}}{\\int_{\\mathbf{\\tilde{x}}}\\exp\\left(-E_{\\theta}(\\mathbf{\\tilde{x}})\\right) d\\mathbf{\\tilde{x}}} = 1\n", "$$\n", "\n", "Note that we call the probability distribution $q_{\\theta}(\\mathbf{x})$ because this is the learned distribution by the model,\n", "and is trained to be as close as possible to the *true*, unknown distribution $p(\\mathbf{x})$.\n", "\n", "The main benefit of this formulation of the probability distribution is its great flexibility as we can choose\n", "$E_{\\theta}$ in whatever way we like, without any constraints.\n", "Nevertheless, when looking at the equation above, we can see a fundamental issue: How do we calculate $Z_{\\theta}$?\n", "There is no chance that we can calculate $Z_{\\theta}$ analytically for high-dimensional input\n", "and/or larger neural networks, but the task requires us to know $Z_{\\theta}$.\n", "Although we can't determine the exact likelihood of a point, there exist methods with which we can train energy-based models.\n", "Thus, we will look next at \"Contrastive Divergence\" for training the model."]}, {"cell_type": "markdown", "id": "5f67ccab", "metadata": {"papermill": {"duration": 0.022199, "end_time": "2021-09-16T12:40:41.120470", "exception": false, "start_time": "2021-09-16T12:40:41.098271", "status": "completed"}, "tags": []}, "source": ["### Contrastive Divergence\n", "\n", "When we train a model on generative modeling, it is usually done by maximum likelihood estimation.\n", "In other words, we try to maximize the likelihood of the examples in the training set.\n", "As the exact likelihood of a point cannot be determined due to the unknown normalization constant $Z_{\\theta}$,\n", "we need to train energy-based models slightly different.\n", "We cannot just maximize the un-normalized probability $\\exp(-E_{\\theta}(\\mathbf{x}_{\\text{train}}))$\n", "because there is no guarantee that $Z_{\\theta}$ stays constant, or that $\\mathbf{x}_{\\text{train}}$\n", "is becoming more likely than the others.\n", "However, if we base our training on comparing the likelihood of points, we can create a stable objective.\n", "Namely, we can re-write our maximum likelihood objective where we maximize the probability\n", "of $\\mathbf{x}_{\\text{train}}$ compared to a randomly sampled data point of our model:\n", "\n", "$$\n", "\\begin{split}\n", " \\nabla_{\\theta}\\mathcal{L}_{\\text{MLE}}(\\mathbf{\\theta};p) & = -\\mathbb{E}_{p(\\mathbf{x})}\\left[\\nabla_{\\theta}\\log q_{\\theta}(\\mathbf{x})\\right]\\\\[5pt]\n", " & = \\mathbb{E}_{p(\\mathbf{x})}\\left[\\nabla_{\\theta}E_{\\theta}(\\mathbf{x})\\right] - \\mathbb{E}_{q_{\\theta}(\\mathbf{x})}\\left[\\nabla_{\\theta}E_{\\theta}(\\mathbf{x})\\right]\n", "\\end{split}\n", "$$\n", "\n", "Note that the loss is still an objective we want to minimize.\n", "Thus, we try to minimize the energy for data points from the dataset, while maximizing the energy for randomly\n", "sampled data points from our model (how we sample will be explained below).\n", "Although this objective sounds intuitive, how is it actually derived from our original distribution $q_{\\theta}(\\mathbf{x})$?\n", "The trick is that we approximate $Z_{\\theta}$ by a single Monte-Carlo sample.\n", "This gives us the exact same objective as written above.\n", "\n", "Visually, we can look at the objective as follows (figure credit\n", "- [Stefano Ermon and Aditya Grover](https://deepgenerativemodels.github.io/assets/slides/cs236_lecture11.pdf)):\n", "\n", "
\n", "\n", "$f_{\\theta}$ represents $\\exp(-E_{\\theta}(\\mathbf{x}))$ in our case.\n", "The point on the right, called \"correct answer\", represents a data point from the dataset\n", "(i.e. $x_{\\text{train}}$), and the left point, \"wrong answer\", a sample from our model (i.e. $x_{\\text{sample}}$).\n", "Thus, we try to \"pull up\" the probability of the data points in the dataset,\n", "while \"pushing down\" randomly sampled points.\n", "The two forces for pulling and pushing are in balance iff $q_{\\theta}(\\mathbf{x})=p(\\mathbf{x})$."]}, {"cell_type": "markdown", "id": "2b9dc4a8", "metadata": {"papermill": {"duration": 0.022022, "end_time": "2021-09-16T12:40:41.164700", "exception": false, "start_time": "2021-09-16T12:40:41.142678", "status": "completed"}, "tags": []}, "source": ["### Sampling from Energy-Based Models\n", "\n", "For sampling from an energy-based model, we can apply a Markov Chain Monte Carlo using Langevin Dynamics.\n", "The idea of the algorithm is to start from a random point, and slowly move towards the direction\n", "of higher probability using the gradients of $E_{\\theta}$.\n", "Nevertheless, this is not enough to fully capture the probability distribution.\n", "We need to add noise $\\omega$ at each gradient step to the current sample.\n", "Under certain conditions such as that we perform the gradient steps an infinite amount of times,\n", "we would be able to create an exact sample from our modeled distribution.\n", "However, as this is not practically possible, we usually limit the chain to $K$ steps\n", "($K$ a hyperparameter that needs to be finetuned).\n", "Overall, the sampling procedure can be summarized in the following algorithm:\n", "\n", "
"]}, {"cell_type": "markdown", "id": "c20bf67b", "metadata": {"papermill": {"duration": 0.02196, "end_time": "2021-09-16T12:40:41.209186", "exception": false, "start_time": "2021-09-16T12:40:41.187226", "status": "completed"}, "tags": []}, "source": ["### Applications of Energy-based models beyond generation\n", "\n", "Modeling the probability distribution for sampling new data is not the only application of energy-based models.\n", "Any application which requires us to compare two elements is much simpler to learn\n", "because we just need to go for the higher energy.\n", "A couple of examples are shown below (figure credit\n", "- [Stefano Ermon and Aditya Grover](https://deepgenerativemodels.github.io/assets/slides/cs236_lecture11.pdf)).\n", "A classification setup like object recognition or sequence labeling can be considered as an energy-based\n", "task as we just need to find the $Y$ input that minimizes the output $E(X, Y)$ (hence maximizes probability).\n", "Similarly, a popular application of energy-based models is denoising of images.\n", "Given an image $X$ with a lot of noise, we try to minimize the energy by finding the true input image $Y$.\n", "\n", "
\n", "\n", "Nonetheless, we will focus on generative modeling here as in the next couple of lectures,\n", "we will discuss more generative deep learning approaches."]}, {"cell_type": "markdown", "id": "89370a6d", "metadata": {"papermill": {"duration": 0.022103, "end_time": "2021-09-16T12:40:41.253274", "exception": false, "start_time": "2021-09-16T12:40:41.231171", "status": "completed"}, "tags": []}, "source": ["## Image generation\n", "\n", "
\n", "\n", "As an example for energy-based models, we will train a model on image generation.\n", "Specifically, we will look at how we can generate MNIST digits with a very simple CNN model.\n", "However, it should be noted that energy models are not easy to train and often diverge\n", "if the hyperparameters are not well tuned.\n", "We will rely on training tricks proposed in the paper\n", "[Implicit Generation and Generalization in Energy-Based Models](https://arxiv.org/abs/1903.08689)\n", "by Yilun Du and Igor Mordatch ([blog](https://openai.com/blog/energy-based-models/)).\n", "The important part of this notebook is however to see how the theory above can actually be used in a model.\n", "\n", "### Dataset\n", "\n", "First, we can load the MNIST dataset below.\n", "Note that we need to normalize the images between -1 and 1 instead of mean 0 and std 1 because during sampling,\n", "we have to limit the input space.\n", "Scaling between -1 and 1 makes it easier to implement it."]}, {"cell_type": "code", "execution_count": 4, "id": "d20babd4", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:41.303522Z", "iopub.status.busy": "2021-09-16T12:40:41.302963Z", "iopub.status.idle": "2021-09-16T12:40:41.332018Z", "shell.execute_reply": "2021-09-16T12:40:41.331579Z"}, "papermill": {"duration": 0.056251, "end_time": "2021-09-16T12:40:41.332136", "exception": false, "start_time": "2021-09-16T12:40:41.275885", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Transformations applied on each image => make them a tensor and normalize between -1 and 1\n", "transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])\n", "\n", "# Loading the training dataset. We need to split it into a training and validation part\n", "train_set = MNIST(root=DATASET_PATH, train=True, transform=transform, download=True)\n", "\n", "# Loading the test set\n", "test_set = MNIST(root=DATASET_PATH, train=False, transform=transform, download=True)\n", "\n", "# We define a set of data loaders that we can use for various purposes later.\n", "# Note that for actually training a model, we will use different data loaders\n", "# with a lower batch size.\n", "train_loader = data.DataLoader(train_set, batch_size=128, shuffle=True, drop_last=True, num_workers=4, pin_memory=True)\n", "test_loader = data.DataLoader(test_set, batch_size=256, shuffle=False, drop_last=False, num_workers=4)"]}, {"cell_type": "markdown", "id": "3e1b26e0", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.02242, "end_time": "2021-09-16T12:40:41.378644", "exception": false, "start_time": "2021-09-16T12:40:41.356224", "status": "completed"}, "tags": []}, "source": ["### CNN Model\n", "\n", "First, we implement our CNN model.\n", "The MNIST images are of size 28x28, hence we only need a small model.\n", "As an example, we will apply several convolutions with stride 2 that downscale the images.\n", "If you are interested, you can also use a deeper model such as a small ResNet, but for simplicity,\n", "we will stick with the tiny network.\n", "\n", "It is a good practice to use a smooth activation function like Swish instead of ReLU in the energy model.\n", "This is because we will rely on the gradients we get back with respect to the input image, which should not be sparse."]}, {"cell_type": "code", "execution_count": 5, "id": "38169d71", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:41.429707Z", "iopub.status.busy": "2021-09-16T12:40:41.429207Z", "iopub.status.idle": "2021-09-16T12:40:41.431319Z", "shell.execute_reply": "2021-09-16T12:40:41.430920Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.030493, "end_time": "2021-09-16T12:40:41.431417", "exception": false, "start_time": "2021-09-16T12:40:41.400924", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class CNNModel(nn.Module):\n", " def __init__(self, hidden_features=32, out_dim=1, **kwargs):\n", " super().__init__()\n", " # We increase the hidden dimension over layers. Here pre-calculated for simplicity.\n", " c_hid1 = hidden_features // 2\n", " c_hid2 = hidden_features\n", " c_hid3 = hidden_features * 2\n", "\n", " # Series of convolutions and Swish activation functions\n", " self.cnn_layers = nn.Sequential(\n", " nn.Conv2d(1, c_hid1, kernel_size=5, stride=2, padding=4), # [16x16] - Larger padding to get 32x32 image\n", " nn.SiLU(),\n", " nn.Conv2d(c_hid1, c_hid2, kernel_size=3, stride=2, padding=1), # [8x8]\n", " nn.SiLU(),\n", " nn.Conv2d(c_hid2, c_hid3, kernel_size=3, stride=2, padding=1), # [4x4]\n", " nn.SiLU(),\n", " nn.Conv2d(c_hid3, c_hid3, kernel_size=3, stride=2, padding=1), # [2x2]\n", " nn.SiLU(),\n", " nn.Flatten(),\n", " nn.Linear(c_hid3 * 4, c_hid3),\n", " nn.SiLU(),\n", " nn.Linear(c_hid3, out_dim),\n", " )\n", "\n", " def forward(self, x):\n", " x = self.cnn_layers(x).squeeze(dim=-1)\n", " return x"]}, {"cell_type": "markdown", "id": "86a8d447", "metadata": {"papermill": {"duration": 0.022437, "end_time": "2021-09-16T12:40:41.476079", "exception": false, "start_time": "2021-09-16T12:40:41.453642", "status": "completed"}, "tags": []}, "source": ["In the rest of the notebook, the output of the model will actually not represent\n", "$E_{\\theta}(\\mathbf{x})$, but $-E_{\\theta}(\\mathbf{x})$.\n", "This is a standard implementation practice for energy-based models, as some people also write the energy probability\n", "density as $q_{\\theta}(\\mathbf{x}) = \\frac{\\exp\\left(f_{\\theta}(\\mathbf{x})\\right)}{Z_{\\theta}}$.\n", "In that case, the model would actually represent $f_{\\theta}(\\mathbf{x})$.\n", "In the training loss etc., we need to be careful to not switch up the signs."]}, {"cell_type": "markdown", "id": "5321df9e", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.022306, "end_time": "2021-09-16T12:40:41.520797", "exception": false, "start_time": "2021-09-16T12:40:41.498491", "status": "completed"}, "tags": []}, "source": ["### Sampling buffer\n", "\n", "In the next part, we look at the training with sampled elements.\n", "To use the contrastive divergence objective, we need to generate samples during training.\n", "Previous work has shown that due to the high dimensionality of images, we need a lot of iterations\n", "inside the MCMC sampling to obtain reasonable samples.\n", "However, there is a training trick that significantly reduces the sampling cost: using a sampling buffer.\n", "The idea is that we store the samples of the last couple of batches in a buffer,\n", "and re-use those as the starting point of the MCMC algorithm for the next batches.\n", "This reduces the sampling cost because the model requires a significantly\n", "lower number of steps to converge to reasonable samples.\n", "However, to not solely rely on previous samples and allow novel samples as well,\n", "we re-initialize 5% of our samples from scratch (random noise between -1 and 1).\n", "\n", "Below, we implement the sampling buffer.\n", "The function `sample_new_exmps` returns a new batch of \"fake\" images.\n", "We refer to those as fake images because they have been generated, but are not actually part of the dataset.\n", "As mentioned before, we use initialize 5% randomly, and 95% are randomly picked from our buffer.\n", "On this initial batch, we perform MCMC for 60 iterations to improve the image quality\n", "and come closer to samples from $q_{\\theta}(\\mathbf{x})$.\n", "In the function `generate_samples`, we implemented the MCMC for images.\n", "Note that the hyperparameters of `step_size`, `steps`, the noise standard deviation\n", "$\\sigma$ are specifically set for MNIST, and need to be finetuned for a different dataset if you want to use such."]}, {"cell_type": "code", "execution_count": 6, "id": "84649cbd", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:41.577495Z", "iopub.status.busy": "2021-09-16T12:40:41.576997Z", "iopub.status.idle": "2021-09-16T12:40:41.579100Z", "shell.execute_reply": "2021-09-16T12:40:41.578638Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.035877, "end_time": "2021-09-16T12:40:41.579195", "exception": false, "start_time": "2021-09-16T12:40:41.543318", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class Sampler:\n", " def __init__(self, model, img_shape, sample_size, max_len=8192):\n", " \"\"\"\n", " Args:\n", " model: Neural network to use for modeling E_theta\n", " img_shape: Shape of the images to model\n", " sample_size: Batch size of the samples\n", " max_len: Maximum number of data points to keep in the buffer\n", " \"\"\"\n", " super().__init__()\n", " self.model = model\n", " self.img_shape = img_shape\n", " self.sample_size = sample_size\n", " self.max_len = max_len\n", " self.examples = [(torch.rand((1,) + img_shape) * 2 - 1) for _ in range(self.sample_size)]\n", "\n", " def sample_new_exmps(self, steps=60, step_size=10):\n", " \"\"\"Function for getting a new batch of \"fake\" images.\n", "\n", " Args:\n", " steps: Number of iterations in the MCMC algorithm\n", " step_size: Learning rate nu in the algorithm above\n", " \"\"\"\n", " # Choose 95% of the batch from the buffer, 5% generate from scratch\n", " n_new = np.random.binomial(self.sample_size, 0.05)\n", " rand_imgs = torch.rand((n_new,) + self.img_shape) * 2 - 1\n", " old_imgs = torch.cat(random.choices(self.examples, k=self.sample_size - n_new), dim=0)\n", " inp_imgs = torch.cat([rand_imgs, old_imgs], dim=0).detach().to(device)\n", "\n", " # Perform MCMC sampling\n", " inp_imgs = Sampler.generate_samples(self.model, inp_imgs, steps=steps, step_size=step_size)\n", "\n", " # Add new images to the buffer and remove old ones if needed\n", " self.examples = list(inp_imgs.to(torch.device(\"cpu\")).chunk(self.sample_size, dim=0)) + self.examples\n", " self.examples = self.examples[: self.max_len]\n", " return inp_imgs\n", "\n", " @staticmethod\n", " def generate_samples(model, inp_imgs, steps=60, step_size=10, return_img_per_step=False):\n", " \"\"\"Function for sampling images for a given model.\n", "\n", " Args:\n", " model: Neural network to use for modeling E_theta\n", " inp_imgs: Images to start from for sampling. If you want to generate new images, enter noise between -1 and 1.\n", " steps: Number of iterations in the MCMC algorithm.\n", " step_size: Learning rate nu in the algorithm above\n", " return_img_per_step: If True, we return the sample at every iteration of the MCMC\n", " \"\"\"\n", " # Before MCMC: set model parameters to \"required_grad=False\"\n", " # because we are only interested in the gradients of the input.\n", " is_training = model.training\n", " model.eval()\n", " for p in model.parameters():\n", " p.requires_grad = False\n", " inp_imgs.requires_grad = True\n", "\n", " # Enable gradient calculation if not already the case\n", " had_gradients_enabled = torch.is_grad_enabled()\n", " torch.set_grad_enabled(True)\n", "\n", " # We use a buffer tensor in which we generate noise each loop iteration.\n", " # More efficient than creating a new tensor every iteration.\n", " noise = torch.randn(inp_imgs.shape, device=inp_imgs.device)\n", "\n", " # List for storing generations at each step (for later analysis)\n", " imgs_per_step = []\n", "\n", " # Loop over K (steps)\n", " for _ in range(steps):\n", " # Part 1: Add noise to the input.\n", " noise.normal_(0, 0.005)\n", " inp_imgs.data.add_(noise.data)\n", " inp_imgs.data.clamp_(min=-1.0, max=1.0)\n", "\n", " # Part 2: calculate gradients for the current input.\n", " out_imgs = -model(inp_imgs)\n", " out_imgs.sum().backward()\n", " inp_imgs.grad.data.clamp_(-0.03, 0.03) # For stabilizing and preventing too high gradients\n", "\n", " # Apply gradients to our current samples\n", " inp_imgs.data.add_(-step_size * inp_imgs.grad.data)\n", " inp_imgs.grad.detach_()\n", " inp_imgs.grad.zero_()\n", " inp_imgs.data.clamp_(min=-1.0, max=1.0)\n", "\n", " if return_img_per_step:\n", " imgs_per_step.append(inp_imgs.clone().detach())\n", "\n", " # Reactivate gradients for parameters for training\n", " for p in model.parameters():\n", " p.requires_grad = True\n", " model.train(is_training)\n", "\n", " # Reset gradient calculation to setting before this function\n", " torch.set_grad_enabled(had_gradients_enabled)\n", "\n", " if return_img_per_step:\n", " return torch.stack(imgs_per_step, dim=0)\n", " else:\n", " return inp_imgs"]}, {"cell_type": "markdown", "id": "da0da692", "metadata": {"papermill": {"duration": 0.022214, "end_time": "2021-09-16T12:40:41.623643", "exception": false, "start_time": "2021-09-16T12:40:41.601429", "status": "completed"}, "tags": []}, "source": ["The idea of the buffer becomes a bit clearer in the following algorithm."]}, {"cell_type": "markdown", "id": "a39ea7ee", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.022698, "end_time": "2021-09-16T12:40:41.668518", "exception": false, "start_time": "2021-09-16T12:40:41.645820", "status": "completed"}, "tags": []}, "source": ["### Training algorithm\n", "\n", "With the sampling buffer being ready, we can complete our training algorithm.\n", "Below is shown a summary of the full training algorithm of an energy model on image modeling:\n", "\n", "
\n", "\n", "The first few statements in each training iteration concern the sampling of the real and fake data,\n", "as we have seen above with the sample buffer.\n", "Next, we calculate the contrastive divergence objective using our energy model $E_{\\theta}$.\n", "However, one additional training trick we need is to add a regularization loss on the output of $E_{\\theta}$.\n", "As the output of the network is not constrained and adding a large bias or not to the output\n", "doesn't change the contrastive divergence loss, we need to ensure somehow else that the output values are in a reasonable range.\n", "Without the regularization loss, the output values will fluctuate in a very large range.\n", "With this, we ensure that the values for the real data are around 0, and the fake data likely slightly lower\n", "(for noise or outliers the score can be still significantly lower).\n", "As the regularization loss is less important than the Contrastive Divergence, we have a weight factor\n", "$\\alpha$ which is usually quite some smaller than 1.\n", "Finally, we perform an update step with an optimizer on the combined loss and add the new samples to the buffer.\n", "\n", "Below, we put this training dynamic into a PyTorch Lightning module:"]}, {"cell_type": "code", "execution_count": 7, "id": "5733f772", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:41.724998Z", "iopub.status.busy": "2021-09-16T12:40:41.717399Z", "iopub.status.idle": "2021-09-16T12:40:41.727028Z", "shell.execute_reply": "2021-09-16T12:40:41.726628Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.034742, "end_time": "2021-09-16T12:40:41.727126", "exception": false, "start_time": "2021-09-16T12:40:41.692384", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class DeepEnergyModel(pl.LightningModule):\n", " def __init__(self, img_shape, batch_size, alpha=0.1, lr=1e-4, beta1=0.0, **CNN_args):\n", " super().__init__()\n", " self.save_hyperparameters()\n", "\n", " self.cnn = CNNModel(**CNN_args)\n", " self.sampler = Sampler(self.cnn, img_shape=img_shape, sample_size=batch_size)\n", " self.example_input_array = torch.zeros(1, *img_shape)\n", "\n", " def forward(self, x):\n", " z = self.cnn(x)\n", " return z\n", "\n", " def configure_optimizers(self):\n", " # Energy models can have issues with momentum as the loss surfaces changes with its parameters.\n", " # Hence, we set it to 0 by default.\n", " optimizer = optim.Adam(self.parameters(), lr=self.hparams.lr, betas=(self.hparams.beta1, 0.999))\n", " scheduler = optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.97) # Exponential decay over epochs\n", " return [optimizer], [scheduler]\n", "\n", " def training_step(self, batch, batch_idx):\n", " # We add minimal noise to the original images to prevent the model from focusing on purely \"clean\" inputs\n", " real_imgs, _ = batch\n", " small_noise = torch.randn_like(real_imgs) * 0.005\n", " real_imgs.add_(small_noise).clamp_(min=-1.0, max=1.0)\n", "\n", " # Obtain samples\n", " fake_imgs = self.sampler.sample_new_exmps(steps=60, step_size=10)\n", "\n", " # Predict energy score for all images\n", " inp_imgs = torch.cat([real_imgs, fake_imgs], dim=0)\n", " real_out, fake_out = self.cnn(inp_imgs).chunk(2, dim=0)\n", "\n", " # Calculate losses\n", " reg_loss = self.hparams.alpha * (real_out ** 2 + fake_out ** 2).mean()\n", " cdiv_loss = fake_out.mean() - real_out.mean()\n", " loss = reg_loss + cdiv_loss\n", "\n", " # Logging\n", " self.log(\"loss\", loss)\n", " self.log(\"loss_regularization\", reg_loss)\n", " self.log(\"loss_contrastive_divergence\", cdiv_loss)\n", " self.log(\"metrics_avg_real\", real_out.mean())\n", " self.log(\"metrics_avg_fake\", fake_out.mean())\n", " return loss\n", "\n", " def validation_step(self, batch, batch_idx):\n", " # For validating, we calculate the contrastive divergence between purely random images and unseen examples\n", " # Note that the validation/test step of energy-based models depends on what we are interested in the model\n", " real_imgs, _ = batch\n", " fake_imgs = torch.rand_like(real_imgs) * 2 - 1\n", "\n", " inp_imgs = torch.cat([real_imgs, fake_imgs], dim=0)\n", " real_out, fake_out = self.cnn(inp_imgs).chunk(2, dim=0)\n", "\n", " cdiv = fake_out.mean() - real_out.mean()\n", " self.log(\"val_contrastive_divergence\", cdiv)\n", " self.log(\"val_fake_out\", fake_out.mean())\n", " self.log(\"val_real_out\", real_out.mean())"]}, {"cell_type": "markdown", "id": "1ae0ae51", "metadata": {"papermill": {"duration": 0.022675, "end_time": "2021-09-16T12:40:41.772185", "exception": false, "start_time": "2021-09-16T12:40:41.749510", "status": "completed"}, "tags": []}, "source": ["We do not implement a test step because energy-based, generative models are usually not evaluated on a test set.\n", "The validation step however is used to get an idea of the difference between ennergy/likelihood\n", "of random images to unseen examples of the dataset."]}, {"cell_type": "markdown", "id": "6ae9a058", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.022224, "end_time": "2021-09-16T12:40:41.816602", "exception": false, "start_time": "2021-09-16T12:40:41.794378", "status": "completed"}, "tags": []}, "source": ["### Callbacks\n", "\n", "To track the performance of our model during training, we will make extensive use of PyTorch Lightning's callback framework.\n", "Remember that callbacks can be used for running small functions at any point of the training,\n", "for instance after finishing an epoch.\n", "Here, we will use three different callbacks we define ourselves.\n", "\n", "The first callback, called `GenerateCallback`, is used for adding image generations to the model during training.\n", "After every $N$ epochs (usually $N=5$ to reduce output to TensorBoard), we take a small batch\n", "of random images and perform many MCMC iterations until the model's generation converges.\n", "Compared to the training that used 60 iterations, we use 256 here because\n", "(1) we only have to do it once compared to the training that has to do it every iteration, and\n", "(2) we do not start from a buffer here, but from scratch.\n", "It is implemented as follows:"]}, {"cell_type": "code", "execution_count": 8, "id": "553fb562", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:41.868808Z", "iopub.status.busy": "2021-09-16T12:40:41.868332Z", "iopub.status.idle": "2021-09-16T12:40:41.870405Z", "shell.execute_reply": "2021-09-16T12:40:41.870008Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.031581, "end_time": "2021-09-16T12:40:41.870500", "exception": false, "start_time": "2021-09-16T12:40:41.838919", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class GenerateCallback(pl.Callback):\n", " def __init__(self, batch_size=8, vis_steps=8, num_steps=256, every_n_epochs=5):\n", " super().__init__()\n", " self.batch_size = batch_size # Number of images to generate\n", " self.vis_steps = vis_steps # Number of steps within generation to visualize\n", " self.num_steps = num_steps # Number of steps to take during generation\n", " # Only save those images every N epochs (otherwise tensorboard gets quite large)\n", " self.every_n_epochs = every_n_epochs\n", "\n", " def on_epoch_end(self, trainer, pl_module):\n", " # Skip for all other epochs\n", " if trainer.current_epoch % self.every_n_epochs == 0:\n", " # Generate images\n", " imgs_per_step = self.generate_imgs(pl_module)\n", " # Plot and add to tensorboard\n", " for i in range(imgs_per_step.shape[1]):\n", " step_size = self.num_steps // self.vis_steps\n", " imgs_to_plot = imgs_per_step[step_size - 1 :: step_size, i]\n", " grid = torchvision.utils.make_grid(\n", " imgs_to_plot, nrow=imgs_to_plot.shape[0], normalize=True, range=(-1, 1)\n", " )\n", " trainer.logger.experiment.add_image(\"generation_%i\" % i, grid, global_step=trainer.current_epoch)\n", "\n", " def generate_imgs(self, pl_module):\n", " pl_module.eval()\n", " start_imgs = torch.rand((self.batch_size,) + pl_module.hparams[\"img_shape\"]).to(pl_module.device)\n", " start_imgs = start_imgs * 2 - 1\n", " imgs_per_step = Sampler.generate_samples(\n", " pl_module.cnn, start_imgs, steps=self.num_steps, step_size=10, return_img_per_step=True\n", " )\n", " pl_module.train()\n", " return imgs_per_step"]}, {"cell_type": "markdown", "id": "10fb6c28", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.022491, "end_time": "2021-09-16T12:40:41.915420", "exception": false, "start_time": "2021-09-16T12:40:41.892929", "status": "completed"}, "tags": []}, "source": ["The second callback is called `SamplerCallback`, and simply adds a randomly picked subset of images\n", "in the sampling buffer to the TensorBoard.\n", "This helps to understand what images are currently shown to the model as \"fake\"."]}, {"cell_type": "code", "execution_count": 9, "id": "d973c3d0", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:41.965194Z", "iopub.status.busy": "2021-09-16T12:40:41.964725Z", "iopub.status.idle": "2021-09-16T12:40:41.966940Z", "shell.execute_reply": "2021-09-16T12:40:41.966479Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.029353, "end_time": "2021-09-16T12:40:41.967038", "exception": false, "start_time": "2021-09-16T12:40:41.937685", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class SamplerCallback(pl.Callback):\n", " def __init__(self, num_imgs=32, every_n_epochs=5):\n", " super().__init__()\n", " self.num_imgs = num_imgs # Number of images to plot\n", " # Only save those images every N epochs (otherwise tensorboard gets quite large)\n", " self.every_n_epochs = every_n_epochs\n", "\n", " def on_epoch_end(self, trainer, pl_module):\n", " if trainer.current_epoch % self.every_n_epochs == 0:\n", " exmp_imgs = torch.cat(random.choices(pl_module.sampler.examples, k=self.num_imgs), dim=0)\n", " grid = torchvision.utils.make_grid(exmp_imgs, nrow=4, normalize=True, range=(-1, 1))\n", " trainer.logger.experiment.add_image(\"sampler\", grid, global_step=trainer.current_epoch)"]}, {"cell_type": "markdown", "id": "adaa80d7", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.022387, "end_time": "2021-09-16T12:40:42.011824", "exception": false, "start_time": "2021-09-16T12:40:41.989437", "status": "completed"}, "tags": []}, "source": ["Finally, our last callback is `OutlierCallback`.\n", "This callback evaluates the model by recording the (negative) energy assigned to random noise.\n", "While our training loss is almost constant across iterations,\n", "this score is likely showing the progress of the model to detect \"outliers\"."]}, {"cell_type": "code", "execution_count": 10, "id": "9ac8745d", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:42.064684Z", "iopub.status.busy": "2021-09-16T12:40:42.064208Z", "iopub.status.idle": "2021-09-16T12:40:42.066806Z", "shell.execute_reply": "2021-09-16T12:40:42.066321Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.03245, "end_time": "2021-09-16T12:40:42.066912", "exception": false, "start_time": "2021-09-16T12:40:42.034462", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class OutlierCallback(pl.Callback):\n", " def __init__(self, batch_size=1024):\n", " super().__init__()\n", " self.batch_size = batch_size\n", "\n", " def on_epoch_end(self, trainer, pl_module):\n", " with torch.no_grad():\n", " pl_module.eval()\n", " rand_imgs = torch.rand((self.batch_size,) + pl_module.hparams[\"img_shape\"]).to(pl_module.device)\n", " rand_imgs = rand_imgs * 2 - 1.0\n", " rand_out = pl_module.cnn(rand_imgs).mean()\n", " pl_module.train()\n", "\n", " trainer.logger.experiment.add_scalar(\"rand_out\", rand_out, global_step=trainer.current_epoch)"]}, {"cell_type": "markdown", "id": "70834e47", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.022843, "end_time": "2021-09-16T12:40:42.112671", "exception": false, "start_time": "2021-09-16T12:40:42.089828", "status": "completed"}, "tags": []}, "source": ["### Running the model\n", "\n", "Finally, we can add everything together to create our final training function.\n", "The function is very similar to any other PyTorch Lightning training function we have seen so far.\n", "However, there is the small difference of that we do not test the model on a test set\n", "because we will analyse the model afterward by checking its prediction and ability to perform outlier detection."]}, {"cell_type": "code", "execution_count": 11, "id": "5561a42f", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:42.164214Z", "iopub.status.busy": "2021-09-16T12:40:42.163735Z", "iopub.status.idle": "2021-09-16T12:40:42.165765Z", "shell.execute_reply": "2021-09-16T12:40:42.165284Z"}, "papermill": {"duration": 0.030454, "end_time": "2021-09-16T12:40:42.165867", "exception": false, "start_time": "2021-09-16T12:40:42.135413", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def train_model(**kwargs):\n", " # Create a PyTorch Lightning trainer with the generation callback\n", " trainer = pl.Trainer(\n", " default_root_dir=os.path.join(CHECKPOINT_PATH, \"MNIST\"),\n", " gpus=1 if str(device).startswith(\"cuda\") else 0,\n", " max_epochs=60,\n", " gradient_clip_val=0.1,\n", " callbacks=[\n", " ModelCheckpoint(save_weights_only=True, mode=\"min\", monitor=\"val_contrastive_divergence\"),\n", " GenerateCallback(every_n_epochs=5),\n", " SamplerCallback(every_n_epochs=5),\n", " OutlierCallback(),\n", " LearningRateMonitor(\"epoch\"),\n", " ],\n", " progress_bar_refresh_rate=1,\n", " )\n", " # Check whether pretrained model exists. If yes, load it and skip training\n", " pretrained_filename = os.path.join(CHECKPOINT_PATH, \"MNIST.ckpt\")\n", " if os.path.isfile(pretrained_filename):\n", " print(\"Found pretrained model, loading...\")\n", " model = DeepEnergyModel.load_from_checkpoint(pretrained_filename)\n", " else:\n", " pl.seed_everything(42)\n", " model = DeepEnergyModel(**kwargs)\n", " trainer.fit(model, train_loader, test_loader)\n", " model = DeepEnergyModel.load_from_checkpoint(trainer.checkpoint_callback.best_model_path)\n", " # No testing as we are more interested in other properties\n", " return model"]}, {"cell_type": "code", "execution_count": 12, "id": "0d734155", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:42.214793Z", "iopub.status.busy": "2021-09-16T12:40:42.214331Z", "iopub.status.idle": "2021-09-16T12:40:42.228657Z", "shell.execute_reply": "2021-09-16T12:40:42.229038Z"}, "papermill": {"duration": 0.040314, "end_time": "2021-09-16T12:40:42.229150", "exception": false, "start_time": "2021-09-16T12:40:42.188836", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model, loading...\n"]}], "source": ["model = train_model(img_shape=(1, 28, 28), batch_size=train_loader.batch_size, lr=1e-4, beta1=0.0)"]}, {"cell_type": "markdown", "id": "3de610c0", "metadata": {"papermill": {"duration": 0.023634, "end_time": "2021-09-16T12:40:42.276465", "exception": false, "start_time": "2021-09-16T12:40:42.252831", "status": "completed"}, "tags": []}, "source": ["## Analysis\n", "\n", "In the last part of the notebook, we will try to take the trained energy-based generative model,\n", "and analyse its properties."]}, {"cell_type": "markdown", "id": "879c2039", "metadata": {"papermill": {"duration": 0.023472, "end_time": "2021-09-16T12:40:42.323822", "exception": false, "start_time": "2021-09-16T12:40:42.300350", "status": "completed"}, "tags": []}, "source": ["### TensorBoard\n", "\n", "The first thing we can look at is the TensorBoard generate during training.\n", "This can help us to understand the training dynamic even better, and shows potential issues.\n", "Let's load the TensorBoard below:"]}, {"cell_type": "code", "execution_count": 13, "id": "c4ffadfc", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:42.374084Z", "iopub.status.busy": "2021-09-16T12:40:42.373589Z", "iopub.status.idle": "2021-09-16T12:40:42.375710Z", "shell.execute_reply": "2021-09-16T12:40:42.375247Z"}, "papermill": {"duration": 0.028238, "end_time": "2021-09-16T12:40:42.375807", "exception": false, "start_time": "2021-09-16T12:40:42.347569", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Uncomment the following two lines to open a tensorboard in the notebook.\n", "# Adjust the path to your CHECKPOINT_PATH if needed.\n", "# %load_ext tensorboard\n", "# %tensorboard --logdir ../saved_models/tutorial8/tensorboards/"]}, {"cell_type": "markdown", "id": "c7b8f333", "metadata": {"papermill": {"duration": 0.023692, "end_time": "2021-09-16T12:40:42.423585", "exception": false, "start_time": "2021-09-16T12:40:42.399893", "status": "completed"}, "tags": []}, "source": ["
"]}, {"cell_type": "markdown", "id": "e608ca53", "metadata": {"papermill": {"duration": 0.023833, "end_time": "2021-09-16T12:40:42.471015", "exception": false, "start_time": "2021-09-16T12:40:42.447182", "status": "completed"}, "tags": []}, "source": ["We see that the contrastive divergence as well as the regularization converge quickly to 0.\n", "However, the training continues although the loss is always close to zero.\n", "This is because our \"training\" data changes with the model by sampling.\n", "The progress of training can be best measured by looking at the samples across iterations,\n", "and the score for random images that decreases constantly over time."]}, {"cell_type": "markdown", "id": "a1184e57", "metadata": {"papermill": {"duration": 0.023719, "end_time": "2021-09-16T12:40:42.518253", "exception": false, "start_time": "2021-09-16T12:40:42.494534", "status": "completed"}, "tags": []}, "source": ["### Image Generation\n", "\n", "Another way of evaluating generative models is by sampling a few generated images.\n", "Generative models need to be good at generating realistic images as this truely shows that they have modeled the true data distribution.\n", "Thus, let's sample a few images of the model below:"]}, {"cell_type": "code", "execution_count": 14, "id": "f84da507", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:42.570336Z", "iopub.status.busy": "2021-09-16T12:40:42.569866Z", "iopub.status.idle": "2021-09-16T12:40:45.522168Z", "shell.execute_reply": "2021-09-16T12:40:45.521632Z"}, "papermill": {"duration": 2.980431, "end_time": "2021-09-16T12:40:45.522288", "exception": false, "start_time": "2021-09-16T12:40:42.541857", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 43\n"]}], "source": ["model.to(device)\n", "pl.seed_everything(43)\n", "callback = GenerateCallback(batch_size=4, vis_steps=8, num_steps=256)\n", "imgs_per_step = callback.generate_imgs(model)\n", "imgs_per_step = imgs_per_step.cpu()"]}, {"cell_type": "markdown", "id": "96a6dec4", "metadata": {"papermill": {"duration": 0.260591, "end_time": "2021-09-16T12:40:45.862389", "exception": false, "start_time": "2021-09-16T12:40:45.601798", "status": "completed"}, "tags": []}, "source": ["The characteristic of sampling with energy-based models is that they require the iterative MCMC algorithm.\n", "To gain an insight in how the images change over iterations, we plot a few intermediate samples in the MCMC as well:"]}, {"cell_type": "code", "execution_count": 15, "id": "8a53c31a", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:46.095876Z", "iopub.status.busy": "2021-09-16T12:40:46.095395Z", "iopub.status.idle": "2021-09-16T12:40:46.738218Z", "shell.execute_reply": "2021-09-16T12:40:46.738604Z"}, "papermill": {"duration": 0.675706, "end_time": "2021-09-16T12:40:46.738750", "exception": false, "start_time": "2021-09-16T12:40:46.063044", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torchvision/utils.py:50: UserWarning: range will be deprecated, please use value_range instead.\n", " warnings.warn(warning)\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDQ2NC4zIDk3LjI2MTM5NzA1ODggXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnichVXLbtswELzzK/bYHkpzH+SSxxhpjebm1kAPRQ6Fq6QOYgdOgub3u7JlhLJV6WCBGpMzs+RoifDgZlcI9y8Q4MF+b4CwgNl183ezbr4t5rB+ccHwrZMknm302I2KekrIRQ0Jvbc/zu2ccdrkhdHdOwxegdVHYY0tVVSP78BjB5Tgw4njsKQGjPTO7aFPJZK8QCQfUZNoiDnDcwM/YAezK2qrQqsKrarQq8pZVXtbrdDWFvmMdr2F2VeE6ydYuiXsT1zBSmr5gs8doyGOk0fOxLGusALFh65KN7ddeXN7ewb4FIyLyZeYo6gWQaD2zcTdfAWzL2imYHV32P3Vb/cTPuBHuIXVjfu8ckt3MOFy9KwcqdTiFTgmrsUHFIwiiHlSnOlSHY0/hZxJavkaHdNHyp6kiGSzKpMGkgwYyOzz0X9toEJHDah6yaJClp3p7S/p0gAx+9AeHtUGanTMABHbmsBSWgfT5095wEEmz2jZTz0HFTrqQMmrxiTM7XlNOkjh0gEz+thPf4eMR58sfqG0/0/KloHwcbZv9hidWrpCR+UVLXyJGJO1jkkHRAPpE7b+RIemUze3Ch1zIBQsfRgp2rfK0w5iL351jC3ulBSzCRVPmpHGiBbNrnn+9bp52sHmtRvVzAQ3x7vg0OX6N8F5I7/o0u77eXPfDjR3mzV5JZzmvK/7D9PS/QMFFmA7CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKNTI3CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ3ID4+CnN0cmVhbQp4nE1RSW7EMAy7+xX8wACWrMV5T4pBD+3/ryUdFO3BECNLXOLuxEQWXrZQ10KH48NGXgmbge+D1pz4GrHiP9pGpJU/VFsgEzFRJHRRNxr3SDe8CtF+pIJXqvdY8xF3K81bOnaxv/fBtOaRKqtCPOTYHNlIWtdE0fE9tN5zQ3TKIIE+NyEHRGmOXoWkv/bDdW00u7U2syeqg0emhPJJsxqa0ylmyGyox20qVjIKN6qMivtURloP8jbOMoCT44QyWk92rCai/NQnl5AXE3HCLjs7FmITCxuHtB+VPrH8fOvN+JtpraWQcUEiNMWl32e8x+d4/wCVT1wmCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTUgPj4Kc3RyZWFtCnicPVJLbsVACNvnFFyg0vCbz3lSVd28+29rQ1KpKryJMcYwfcqQueVLXRJxhcm3Xq5bPKZ8LltamXmIu4uNJT623JfuIbZddC6xOB1H8gsynSpEqM2q0aH4QpaFB5BO8KELwn05/uMvgMHXsA244T0yQbAk5ilCxm5RGZoSQRFh55EVqKRQn1nC31Hu6/cyBWpvjKULYxz0CbQFQm1IxALqQABE7JRUrZCOZyQTvxXdZ2IcYOfRsgGuGVRElnvsx4ipzqiMvETEPk9N+iiWTC1Wxm5TGV/8lIzUfHQFKqk08pTy0FWz0AtYiXkS9jn8SPjn1mwhhjpu1vKJ5R8zxTISzmBLOWChl+NH4NtZdRGuHbm4znSBH5XWcEy0637I9U/+dNtazXW8cgiiQOVNQfC7Dq5GscTEMj6djSl6oiywGpq8RjPBYRAR1vfDyAMa/XK8EDSnayK0WCKbtWJEjYpscz29BNZM78U51sMTwmzvndahsjMzKiGC2rqGautAdrO+83C2nz8z6KJtCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicTVFJigMwDLvnFfpAIV6TvKdDmUPn/9fKDoU5BAmvkpOWmFgLDzGEHyw9+JEhczf9G36i2btZepLJ2f+Y5yJTUfhSqC5iQl2IG8+hEfA9oWsSWbG98Tkso5lzvgcfhbgEM6EBY31JMrmo5pUhE04MdRwOWqTCuGtiw+Ja0TyN3G77RmZlJoQNj2RC3BiAiCDrArIYLJQ2NhMyWc4D7Q3JDVpg16kbUYuCK5TWCXSiVsSqzOCz5tZ2N0Mt8uCoffH6aFaXYIXRS/VYeF+FPpipmXbukkJ64U07IsweCqQyOy0rtXvE6m6B+j/LUvD9yff4Ha8PzfxcnAplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTQgPj4Kc3RyZWFtCnicRY3BEcAgCAT/VEEJCgraTyaTh/b/jRAyfGDnDu6EBQu2eUYfBZUmXhVYB0pj3FCPQL3hci3J3AUPcCd/2tBUnJbTd2mRSVUp3KQSef8OZyaQqHnRY533C2P7IzwKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIyID4+CnN0cmVhbQp4nDVRu23FMAzsNQUXMCB+Jc3jIEiRt3+bO9qpSNO8H1VeMqVcLnXJKllh8qVDdYqmfJ5mpvwO9ZDjmB7ZIbpT1pZ7GBaWiXlKHbGaLPdwCza+AJoScwvx9wjwK4BRwESgbvH3D7pZEkAaFPwU6JqrllhiAg2Lha3ZFeJW3SlYuKv4diS5BwlyMVnoUw5Fiim3wHwZLNmRWpzrclkK/259AhphhTjss4tE4HnAA0wk/mSAbM8+W+zq6kU2doY46dCAi4CbzSQBQVM4qz64Yftqu+bnmSgnODnWr6Ixvg1O5ktS3le5x8+gQd74Mzxnd45QDppQCPTdAiCH3cBGhD61z8AuA7ZJu3djSvmcZCm+BDYK9qhTHcrwYuzMVm/Y/MfoymZRbJCV9dHpDsrcoBNiHm9koVuytvs3D7N9/wFfGXtkCmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTggPj4Kc3RyZWFtCnicPVC5jQQxDMtdhRpYwHrtqWcWi0um//RI+fYi0RZFUio1mZIpL3WUJVlT3jp8lsQOeYblbmQ2JSpFL5OwJffQCvF9ieYU993VlrNDNJdoOX4LMyqqGx3TSzaacCoTuqDcwzP6DW10A1aHHrFbINCkYNe2IHLHDxgMwZkTiyIMSk0G/65yj59eixs+w/FDFJGSDuY1/1j98nMNr1OPJ5Fub77iXpypDgMRHJKavCNdWLEuEhFpNUFNz8BaLYC7t17+G7QjugxA9onEcZpSjqG/a3Clzy/lJ1PYCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MyA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIwID4+CnN0cmVhbQp4nDVSS24FMQjbzym4QKXwT87zqqqLvvtvaxO9FUwwYOMpL1nSS77UJdulw+RbH/clsULej+2azFLF9xazFM8tr0fPEbctCgRREz1YmS8VItTP9Og6qHBKn4FXCLcUG7yDSQCDavgHHqUzIFDnQMa7YjJSA4Ik2HNpcQiJciaJf6S8nt8nraSh9D1Zmcvfk0ul0B1NTugBxcrFSaBdSfmgmZhKRJKX632xQvSGwJI8PkcxyYDsNoltogUm5x6lJczEFDqwxwK8ZprVVehgwh6HKYxXC7OoHmzyWxOVpB2t4xnZMN7LMFNioeGwBdTmYmWC7uXjNa/CiO1Rk13DcO6WzXcI0Wj+GxbK4GMVkoBHp7ESDWk4wIjAnl44xV7zEzkOwIhjnZosDGNoJqd6jonA0J6zpWHGxx5a9fMPVOl8hwplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTggPj4Kc3RyZWFtCnicMza0UDCAwxRDrjQAHeYDUgplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMzID4+CnN0cmVhbQp4nEWPSw4EIQhE95yijsDHH+dxMumFc//tgJ1uE2M9hVSBuYKhPS5rA50VHyEZtvG3qZaORVk+VHpSVg/J4Iesxssh3KAs8IJJKoYhUIuYGpEtZW63gNs2DbKylVOljrCLozCP9rRsFR5folsidZI/g8QqL9zjuh3Ipda73qKLvn+kATEJCmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDAgPj4Kc3RyZWFtCnicNVI5bgQxDOv9Cn0ggG7b79kgSJH8vw2p2RQDcXRSlDtaVHbLh4VUtex0+bSV2hI35HdlhcQJyasS7VKGSKi8ViHV75kyr7c1ZwTIUqXC5KTkccmCP8OlpwvH+baxr+XIHY8eWBUjoUTAMsXE6BqWzu6wZlt+lmnAj3iEnCvWLcdYBVIb3TjtiveheS2yBoi9mZaKCh1WiRZ+QfGgR4199hhUWCDR7RxJcIyJUJGAdoHaSAw5eyx2UR/0MygxE+jaG0XcQYElkpg5xbp09N/40LGg/tiMN786KulbWllj0j4b7ZTGLDLpelj0dPPWx4MLNO+i/OfVDBI0ZY2Sxget2jmGoplRVni3Q5MNzTHHIfMOnsMZCUr6PBS/jyUTHZTI3w4NoX9fHqOMnDbeAuaiP20VBw7is8NeuYEVShdrkvcBqUzogen/r/G1vtfXHx3tgMYKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1MSA+PgpzdHJlYW0KeJwtUUlyA0EIu88r9IRmp99jlyuH5P/XCMoHBg2LQHRa4qCMnyAsV7zlkatow98zMYLfBYd+K9dtWORAVCBJY1A1oXbxevQe2HGYCcyT1rAMZqwP/Iwp3OjF4TEZZ7fXZdQQ7F2vPZlByaxcxCUTF0zVYSNnDj+ZMi60cz03IOdGWJdhkG5WGjMSjjSFSCGFqpukzgRBEoyuRo02chT7pS+PdIZVjagx7HMtbV/PTThr0OxYrPLklB5dcS4nFy+sHPT1NgMXUWms8kBIwP1uD/VzspPfeEvnzhbT43vNyfLCVGDFm9duQDbV4t+8iOP7jK/n5/n8A19gW4gKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxNSA+PgpzdHJlYW0KeJw1UTkOAyEM7PcV/kAkjC94T6Iozf6/zYzRVh7BXIa0lCGZ8lKTqCHlUz56mS6cutzXzGo055a0LXOAuLa8L62SwIlmiIPBaZi4AZo8AUPX0ahRQxce0NSlUyiw3AQ+irduD91jtYGXtiHniSBiKBksQc2pRRMWbc8npDW/Xosb3pft3chTpcaWGIEGAVY4HNfo1/CVPU8m0XQVMtSrNcsYCRNFIjz5jqbVE+taNNIyEtTGEaxqA7w7/TBOAAATccsCZJ9KlLPkxG+x9LMGV/r+AZ9HVJYKZW5kc3RyZWFtCmVuZG9iagoxNiAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNyAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggNTYgL2VpZ2h0IC9uaW5lIDcxCi9HIDk3IC9hIDEwMSAvZSAxMDUgL2kgMTEwIC9uIC9vIDExNCAvciAxMTYgL3QgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDE1IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDE0IDAgUiA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxNCAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNyAwIG9iago8PCAvRyAxOCAwIFIgL2EgMTkgMCBSIC9lIDIwIDAgUiAvZWlnaHQgMjEgMCBSIC9maXZlIDIyIDAgUiAvZm91ciAyMyAwIFIKL2kgMjQgMCBSIC9uIDI1IDAgUiAvbmluZSAyNiAwIFIgL28gMjcgMCBSIC9vbmUgMjggMCBSIC9yIDI5IDAgUgovc2l4IDMwIDAgUiAvc3BhY2UgMzEgMCBSIC90IDMyIDAgUiAvdGhyZWUgMzMgMCBSIC90d28gMzQgMCBSIC96ZXJvIDM1IDAgUgo+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTYgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjU1ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDQ0NyAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgNTMgL0xlbmd0aCAzNiAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NDcgPj4Kc3RyZWFtCnic7ZwJOFRv3/idWQwz9p1SUbYW0YJCFIlUZN/K0oKyk8pSE7KULWtZkiK0kEhFSUWLColCSSuS7Psy3nNmDGdmTr83z3Nd7/O+/3/3Vdc1c+7zPed73597+S73YCL+Lf+XC9N/WoG/5d8qTP9pBf6Wf6swgf9PRt0N1Q7PfvSqLj7xVYiRncma07ejgo8f87HzjrtycXf0k+o7DhGXjQ48PKznE5oYQiRmvptLKSASo97MReB5EDGgfC4Cb+OIxOtz0ukqkRj/di4Cj08QgyrnotKbCCKxcE46XSISU373NMQv96b5nYioiVSIu1P3qT06sfOUhtlm7ujX50K8vD2tPONv5pil/Rh4bXmmcNPez+5rvUIvnCISi6bmUl4QiQnjcxHoPEkM+DYXAVIqkfhoTjqVEYlppLkIfA0gBv+ai8AYOKZezUmnQiIxZ04CDdP89HcUpbinZ3o57l6p+zjE4vDu+dFVucfkNpufDbC2c7n4anjwlb7vOTO/Dgfx3d5xwX/5/Un5H+TneSQ/J7m09PSB7Xucj9vqxXrxO6dlnNDcZuRsqqRnl3i19OGNo3E5iQnJnrs8wovOMPKbbGv9MfLb1/zl92flX+Z389G1e286Wkv8tB7liUiuK45lVTCJDnHapb+YF6Vq5uu21zWwpKxi+N7Ck/fjb3ZdZOQ3+rC04veN+8vvz8q/zO9c4tbwb3HuWoGlSX5KWoa+lmiuRa722m5EP499ayUklq3e43700uPSu/mRxOKzeu60/IY+Xj5hr7lxo56p9T6ngJjiZobXMPJrv3su1NfHLyQy+VbdEKNeCPzGvz3MTruYkXWnEqnXkfiR+pvulT289/wd4sKAyG/0fcO713Vf+5EEkPn1d/348rETUeB3/Eb7W3uR16rf8SP1DkwgjjUqv9NECc9mN5018b1Rjspbjc02ABhOK4st/rGRwX5qS/DCy01s950tvVTwJXTXkyghGzi/iZHmEg9NCU4CKxun8KKlmmZhRd96h2lfQ8dvYvRX5SnbbSrKKrqGdoHZb1t7Run0YuA32t9SFnvI8aCLR3DKhy9tQ/SzmZEfqev7m4KIM9Gnkq7Vfe5gHCOM/IY7W98X5l3LyC6pbunoYxBg5DfZ09la9/JRceGT6o/t3QwdjMBvuK+j9eu76gcv6pu/do8xvIKR38RIV8f3z80vq5s+tHQxNoLKz9VunUWIg0/nwNANH2Z+MTFBQFhGxToxLXqxAOfpVNVd/ktWRKZejok9ryN5K9XIA86vvcpGVYiABqCCQqOxOHaJjSkVtI2h49dR528oxMHKgsMy43B4juU68dV0ejHwe3114/J5/NwsrHgCG5ecwYPPdAIM/MYHQwxXSszjwIMCnJL7HzA0nZFfuZfuOglhdrBwCzueZxBg5PcrwdNAUZoTKtymEQzEGfmRHl7Yr7dOiBsSUAp9x/AKRn7fnp48oCnFy8nJxSUaXMIgQOXnYbdc3TogqbrqeY4/nkPA2lpDWlTTPtnHXkdne8Edlb3RB7xu3CkO9D1upP34wfkwOL+qC+AExTDNFAAlY5Ty8B/5VWXqybFi0WAhQ+eRO11Jpxcdv9GfFz3F+AgsOAwahQIAAdnMajoBBn5f8k1l+bkIOCzABABc65O/TdBJ0PMbeR65WUqEi40FVArNouLdQb/EMfBrKbPRXCrMw4IFVUKvsmoZpBNg4Nf36dQ+Ben5bDgU2ApR4zKGRZSe30Rv8cnt65bw41EACkUwTBynn+NUfkfshRfJX34YHxWeckyAlf1uedp6vIH72c0rQ5Nzv9YoOKRXNTY1N7jtsjjg3/BrKgvO7+qehewAhRwEAyyq/hll/8gvy0GEAN2MZkaRBRYEP6NTjI5fd9X+5cwocGiwYiEBDFvwHToBBn6lxotYwVfg8aBOGGa8y336xYeeX6efDicGQOG5MSgMBsOu9bKTToCeH6kgcCEnCsBws4DrDpZdvKKNToCB38dbm+aD2nBxoDBYNF44o3OSToKe38iHAFk8GsfBg0FjsSyCTv30g5DK7/q9rCALV/+kkrcu21cbuGRfDj/hrb3zYNz1d6/utzRGJGQ8uhd3u9JEU+d4xqsvtPzCZAgYMj8Ax4rSu1eUnxVkfTCKVjMafm3Z5ktYyDOPMI+wOL/geozfFvtTtL1Fy2+swkKaHRzkAG6xMEfcnavxx7cax7yhEaDj1x9lLcYGvgMlLsXnUJJ/0c9SJ+4+bdPp+D1MWLcQmkjC8os2FudfPrpPK+w27fyg49de4aCCxwAA+9qlUpm3r4S46nve76ARoOM30ZFlOY8FADDya6VPlOQlOplZPvhIqxM9v6+HNnCgUWhpdXnLkpuXXWyN8ppoO5bKr7iuOcPKxPZc2Sdb9bX7wxOjjpyO1NHSz333sb6ysiw+Kf3hncBLRbYGBhE3n735cn6W30SfNzeaPPtQGJElUiemRvrbk7boeIzQvIeG33s/RTwK4s0sqrpKv2+wqzpTZr0hrbVJw4/Ulr2Mi4yPV2ej4uuxztpbq5da0eKg5TfWZqPIBvYtlqChvyGN1PepwEzENom2r+j4XXYX5QB5s66y0PWY6Pue58tjFk5rVdLx+5CjtQBUCjd/t7HRt9GOB+eWqMd8oBGg4zfadHoVuOxg2U1tjYtJPW9SNorHV9DqRMdvrE5bHAWgWTWdrKNJ/d/S7AUDS2gNNyq/84U/o/iVjYJDQw+bM2u679ZbpaxhrSOaXO6e2W29QtQ4+m1jXqDDy8q0nPwH8Tqus/w6c/WxlNWTVcj5/DdoDycFYzAKzTS7OQ2/58pCACSCXeLw/Ae0qk08xTHzNtLoRcNvOGY3NzO0Notvff9zcHKKNNG7HINPphGg5ddSumEhBFxwbXnvIGjmkcad0JyWtH1Fxy9Cn4AB95g1cUNDI5DATRTf6h80AnT8XpyQApVCS1kNDA2RQIHv87llaQ0MOn59N2wgM2++yruhIXAdnBw1Zll9klYnOn4t+Ys5wQkulzk8DBropLEU9Cob2m2Ayu/W08IoOwszq+DcCz48iqYn3NbZ+mVG7SGGOngFa8hr6hrFJIfFX3x776D9gThfbRi/gRoHfsh4wSoYEfNfQXpNdPowMYn63W6HrdU0/GoMFkGL54qtgbmtFL/hORaFjyqD60XDbzTbmQC+Q0TjSEoPxeYeXAZgvJ/Blzdafr+qVwmAe9+GvdFfp9cBZwCn855mPtHxO6bKikYt3xr5dPr7LYAg8YHGDaLj99BBDAcIrfbNojyE9GMeXpQ2qkHH71esARuaVco6bvohkyYYsaM0AnT8SBUJC9hRohvCa6YvpAFi22mNJCq/2s+uMU8DrKVDvlelSy1Xyk3dkvh8pLXUbuN+FZT0Jv/N7KrbzHK7GtMl+Hh3WxvQ+A+nl7OAs4nt8I3p78O1TuBiinWrhDWehl+T90pobuxJo/Y/6RkWQK9xh+tFw2/8RTAOHLjrYmechqFlAEonoBsmQLf/tUuzAwB/2MOZC84ASjH3C/wVdPxcl+LQKJtZLW8BGBFaE4aOX9EOUTywxpX6TJAfluMG/H56fu1eGtwYXmPqPaQJExS3K40APb/sw/PY0Bt9Zlz3NIBXZYBGgMqvsjk15UJm+IKTbXZGoW5WqrICoWVeyT+fFqb76jofORJ3bavR/SDLs6eN/KL32tgdhvP7Wr5NCmDCrvCd/t6VthNyIiyzYSsoDb+hj2HbQNNQTKd3Ws36C6BVucgGrhft/tdXbr8YAHjlqLtFd4MkgFppAbcW6O2XSEMUwCwRMv11os8BAKR8a2AC9PwenQH3JpGt1EDCWD7o1yQ9hwvQ2y9PFDEA+6LXVAU65mNwMTTmBR2/sWY/LgxWIHpmXTJBERyGaewEuvWzo3g+AeDWG6XekwZwKSHzu1aSkxiYm7YktE1jU2GEoyAL1v+23alPFUXJPoZHTwZeKNEzrXCUOX16z8VSB4ejx2jiZyPHzWSFCDymtb0Trd9bGx4fUoa2Q6PzvbO30PkPRURFMRxhSfm3qb6unm91mUdAfoJmNMY8rf/QkmIog8OxJzVOjg0O9zY+vrIAACR02mECdPxG7gfJcQAYmzeDU+MTE5/ry40AYNG+Z/Cm0/H7WqY0D0BJvyQ/tLO5OgYAOALvwgXo+I312EmDYySLbLMMf254JITGBlTAzQsG/+HSenBVcHkNrTuTrR/rtFGsth00QRh6+7Nx80IAWPvyJ/S562t9EMCp0E0DnMpPcXmK1/rsm7rnvprt7bzqyYdB26XW5B2UFZCaz6brVuwouNPKYMW8/Ku2N+q9TlXE08Y/x0aHj8szY1hv/vJy9VrKiSXbo6aXfs9vcmw4X5IdxeI0dSc998B8AhbyJfT+gR9pfLhZHA/gZAe+lVffkBUG/VlAZF0rTIDe/wNBa4O2HtuT8bZfXeaynDjQM9Cj2WLp+JEmhyMh+zMQ+nJWQxjsaYJDJlyAwX8ffQUuIzh1qGE1Zss50Wis/UX4nszAb7xPASTO3wB+HDy8lRcHYLeX0jyRnh9p+DBoIrGegz5ftJ/HBrDLv++F30DlZ2hYGrnFw2ff5c8OHu3JVsLcQkevnfbYrqGycvU6v6R7dqzyKhuND1Q+S4kN22UfcIIh/5Drwc4EWPkpKynzkp1BAFD3+TlbzRi/rvWRBgCFYDtTayV2DGTOcOq2wkwrxvh191ENJiZ+oq/Lkf2CBBT4jnnK/8QPvBJrxgyg7INPhkevFiR7dvr/xA8spftFAGBTcFzSJWtp0KcF8HvS4dWM8bNWZwUAWHAyPj3nlKIQaL2idyXCLR7G+NlEyE4UgPOIzs7N0ZMF91u0Vi6NjcsYP7thx8PEtDPiasFdj82gR8Qi/Qze6tn4S9CXKxZKqicKPvoEfArTWiwue6Z4nexqOxe1nS4FT+7sQnPNVwgs+/a9yVNps/oaBwZ+H+8KockBGBQ1FCO5A9b/jPx663VA0AAzK54FA4XEUHzaL2DAGfmNNwZhQa+RhVuEDIOJaYEaPN6BkH/4epcTtHqwwmJSHDhohIgYPIRXM/LrfrcOGkqiS1WXcEEf8LtpgqCM/EabPMGlA7VQSVtVGA+9wij8n/lNfbrKigbQAlsMTKQEQN4otbTv8GpGfh3V0hiAmXPHLkd1SVAnzMLCJng1lV9EuL1PwqnwjEvpmZez828l+aqpq3BK7/I+5uTmvWOTpvZ237iH0R6fnlqHpIZYCVgz8BvpvLSXEkCj8pOzgXUvI7+JodJoNtDMAdtAlgIIm2pgnYOQPxp+m7GSiQmNZcGhyG/hXwVvOgK/0Z/Xd4OrDzsbAQtFe9AC6sXwakZ+oE4R4JTlFRRix0Jq4Y0S4dWM/EjDdTlLUBj+xeLz2bBQOND49H/Db7Q93wiN4Vy2fJkABwaF5tFJoWklI7/xgYdhKCxhzTqFxYJYDDP/6ts0MQIqv7hobbuwtOzCc8RLWcl5pY/Ob1VZvULN2t456Mg+GcH5Sian00uj3J9mrvLLDHfZ4MSYv52ojxHjZgVmw9hLjRpmXRWk/G1rhZwgNKOmo944xUKYeY+Uv+1tMl2IgQKmlFHCJf0CZt4j5f/GvkRIsKJYsCjyKEHzrMn5BQOGlP/rKFvOh2LnJDCj0GhwL9QNH4A5sUj5v75P2xdiOIV4OXFYDIDG6QX0wx6JlP8bawuQxhPmzxPmZGPFoHk3x9AkUpDyf733ZPhYlkgsFODGs+AEV19/C6+k8ktIsFwvGPegKnijc0DGMYf35baBKTWpKkILr3qIgDOeT9pYX/R4uYsWQcXcKn40E+H8C+lDqr4UepYfjj/1xUwlYv59IM9VEDfDD+DTgj0TMf9OehzHi6PkqUB+zNwHM2B1iPn35pvLyPejWcHpwTrPNQs2PRDzt713d0G3AwAzMxqFlT/wvHu2DjF/O/kkCoOGtj4CaLehFfe2wwwYxPztZONN0HhGY9CEeWwYZlkXmiQSYv62q1QPwKBRGBYRPmZW0cBSeB2V393KDOdlV5+c8d5t6xofFX7jalzmtXA7UVWTRGtuo4NOxupermujXppt2R5wVsO3B+H8BLh7vN7JR5lOvJIs4EKHt4yfqUPkN950kg1DRscixgUuoFKB5TONRz4/8amQvC8BKIyoCIDBa/h9mzF5kPl1vJKf5iciwYxl1/WDRWAQ+Q3VOFKWczaBZdxooS0JLbN1iPxIb89D+zeKmVtEVgw9f/1NGA/k/PvnB4vITeCUlJfHCytlNcKUQOTXU74TWnSY2WTk1wtwa4Z/gSW7qfwax36c31T63Hj/URPLgPM3QuKfPCrYLIHaGx6gz5P2qjrWPCFaL+ml9vaIZ41ytk0piOfPJvUhGOAGJbONC+ozUesZxxP5/MtkIkQbjcXxbxZHARg2/eAuqgAyv6EqNihQjmPlUFPCotALTCo6qI1H5tf/WQGCwczKpbCdAw0sNfgxux4in59ocYcEMCxCK0yXYFFL9r6cVRv5/MSnbGhIYThFl1luYsHy+hTO5uiQ+X2vEIP4sc1TMLflZcV658FC/cj8Hu+EdGIXUjH0kCbgLcu6ZgWo/O5VXzu0yvyA71bQWlMwdKtv++GvrrZu5Q4b4qVXqcdUDZzK82SWK15IMCntPmy27gASv7YSyG8Hdvi8uOQuggY4hPBiNtQeReQ3fM8F4qcSUPos316SBYUWWH7mwXQdMr/nZ1lBgYWxhTWNt7TFWXHCasmvpx+LzO/9VRkIX3BuTUO9kyYnB79R/Ez3IvLrzjGC5sb+gqq3H5MOiPAKmQTNLLnI62ehP2Qcrb1bWd/yKGTjwsW6wS3UOmR+T6OFwFdwX69saGk677JipVpk+UwdIr8vkcrQdAitaWz+WhCtqKh+pngGIJXf1Ttprmu0De3XYriElm2yauzqCtLerqVs7hRe3Bu5W0Tv1Nu7EkLzim+7PWhN3ce9e5bf5FB/z9ePTR9bXhfHguahsPLJvIlnpxQWS0tLMrOvyJ2+i54fqb+rvfF5pAETE17Jq6hn+F2i9caV3Gzcu6lJBUZ+pJ7v9UluzEyArOkT0AX6eMxCTZBX6ED6dPci8Bvqbsj1EQUdDfWST+DMuuCrI8ErafOIioCR3+RIS8VRFdAVVU3+Ds7TZ2mGCrzy+o+pFgYCv4mfb0PMUUzo1c4doB7tJUf0F8gb5r6bfiwCP9JQ0/l93AAgsbUF3CvGqi7tXbNsV9wb6jaAwI/0+d5ecBDyqRZDT2157LJ9xf5TdVRXi8ovNPryMR1bEywrJ4ZHS0f/Utm32rLjdiqxdwrLqnavWeDf/POxkpxCZVtP5e1PZ+H8hhoq7wW6W7p4Kipo8DMxOYxPkqY+3T8ddu2QBgEAqP4TPb/xyqJkc6WlwkxMMj3k6OzkxPATrfkA2vS3/Ebuh2tL8qOYsI8p4dzJiR4rORSrROdv+TXcNl6BBg2d4+PTAuPBG3BYwu3pakZ+/Z8PaTKD1q0yZUUjTU4UbxJgnu8/XY10finTkBe0pdjqKcsyabLNZCWLhMH0No7Ab7xh1wpwvUUnTguQJo5v5lq8sX66GoHfmJ8OGjSPdk6vG6SJm8YSYtupd1H5ObneTtinqaK9euGa9aqKa7YfzHrw2NViS/LDp4XnvG3kTE58fK6usdXbx9P5gPtu5YMz/H5UhXnZqa2VWbVGSGgBHsXqBl3s/Xz/flPZRZCOjt+NaugKDb/u8uzYA5ZbpIV5CQBWjupkTLZe0QQAmaMVZBuflt9k6+3UU+bqi0AHBcVCTfBMjRV6ARiewzcp7aLlN/72QZq7iTQ/tCEHzlx9do4Thd6TTNn/6fgNtBXEOimSs4YqM5vk5+y1KLYt5yghDwZ+NYX+BlLQIQ32meTl0E17NNeKC5RAOQO/9ldnnJfyQbvfuZlrFXEc3GJx5RRNGM5PfMwPVhGHBHbOXGvOk+OS8XlG6TcqP1Pz8mwf2WVE0xVWFmuWLcZLuFy+Z2O440JF3WXHyLCNigpvqnRNd69cyC4pLyBr7U3lR3p7XWspPzN62u9Dc3tR9Gj/AXaILA5gwrtkQ99p+H0+s2udEDvFayAowIJmTngUChvXAg1+uvMT1Ue0l5ItTwDDDssJ5BOwGB4Xiia0/IYLQnSFMRgosIOD5UjbF+GAeZqUED4dv44aN3UWcjABpQpz+kxYARH1avJHBn5ZXsJs5MgAJyz5nIxHcxpmUdSm51d3YS0vxf+BnQVoEmBlsZpOStDz6ytzEWch62Qwe3FiE4HdIIGiyWz+4epR1dDwdXLL1sovcs44ZMC3YKmT+fKTl4P2qWfkWwbmd7dfTDpje+hO1ZuoQ8aeM/Ov66mZLAcGoPITdGkjD+5xKL/8/izAxJUHBWtp+bWcMl3NjqXwW7QD5pF9LxcGgCxyQJ6W3+AzN2VRCj/hdbBzL331uii8I/kjPb9r/qqCZF+RsDZl9vL4u2CAS5HiRNDx+/JkvyontHqiJG1gIf5Ptzl4ZSmOLMSPJiGY7iJMPoY1b/2n2Ys/Xyixq1FeycCvNlVZBKLBJgOjNFzpgtHyQ+bX/8hDihU6D7foAOzq20z2jX6U5AuVX37JCctVadfNN61cKqsaUJDqIbRIxkRXzjvKdefKU+mhmRX93aUJ3iZuJe9bE31MZvkNvT28Y6Xapo0bVXnxaADNa178fjaKeRvkl1MHfaI9v5R1Yr+O5sZ5LAATOBmaYfH0TlEASEVYP4frYw8aayyFojUCqyvgWXQLFMtecgqNjt9oWaqTgSqU18CvjBuEYUoH2FYh8mt/c9pluwB05GmxJSwpR2rm4pamnG5k4HcnxnAZNKqEFZpnBSaHNdkUKHE3Bn7Nt1y0oUw0XiJ79iJpIBi7wROZ31BtghU/FJkTtYddHXnGqexOWdNnzl8fExRTulL7OVJZXCU1LrrqpoUvkWPBDrPdezYJyFqMNBR2DTTtYRWWNU++u9dqdv0Ep8Cz7PiuycnJzq2S4EDBLNQKyZ1ZfUB+BDeyPrT2C4lEAiUcF6BB110kGpaU+wXy87wFzWB6+wWSuMAFTloAm/AK1uvmAM6iGVqCGewXUKKJCwWFZJ0bYDGRiwBuGSWxzGC/kEjjGtAxG0CxDbaoN3NziFFURIh/ToaRD7lxVsMESFr4pdHkT0j25ws+FugVZ+EXI1nW2FN6B+SXTWtTkQYVCeT9D375Ne9a26/kT1R+x7yUt1js9qoM1gjJfHM3p6bI+ULhftNtdoeykwKDzg21v3v3JNxS+djJo2GxyVnFMbP8xtqbaqEOH8kP0xMGUKIaIXnT/EbDTQCA+xrZtELy/+5FKYG4eOyTa6mNz3EAd8VYhP2PUt5EbABXH/SemIau6Su17tIAxroXejKS/9cdbcIPItcMq6LmXPr8tACCHOL8I3dWthc7ONrFIx60Ugdhih0LlwRly2WYf2CpjFgAjikWvys/qPtAhfsSzjUJlL5B8P9+nN0C8bNI+UI9ntDho8Wl6oY8/8AykeMOLbkrY191TWs7Eb9XRN2JEryfyR+5W+xxk1xZFKBf09X7uri2xP/2mwziNoeAhqa6B4/7utrK0gws7B+XRh499LQFMf45We0vDU4ntRNZPf39AwP9/e3ybABa6D157CPxG/1iJcfExLbZ8/KXfkqxQwMo7DVyJRK/kZ/2y8E71tvcrJ8WuAwaKDjKyoLEb7InTpyFiWnJ1sxnfeDdfb19HwRAT0r5t/zAjVYAnB7cmhHPOqHn9/X1bUdjBORfkiuR+I11rYKMqnUHXn0D74b+ncGyCW5Oo1Qi8CONe0P8pLc9aqa0oa+Kj1t4m/9v+U1NPYBsJD618/Xd5Bf0/trIs0SPbv+7fM1bf4nizkt5RXEBe4jJrY0XvHftsPTcsulcTqGhces5bVtH14xHSkvV9e1/vIlB+P3mxPsYPtBewOD5RCQkJdevk5RcAhpOvHJt5C5C4vfl/jywc1EEHpHFkpTCBfLXpQQjkPh112+CLBJWzvli0wIiAIrDkbKVIOYf3vsyg1smM5vIQvBmKSlJqSVYZg43ioWEyO9bIXTCD0MQWCghKUV+BQdu4f5yyhaNmH9okYNmByv3oiXkV0hJCqFlDD9QBJD4TQ54QifIcWwLxKHHS8lIiRNk9CumjQZEfmWQVYXBC4tJSi+VkpKRluIR3XPlJwU4lV96TpipiLrlufzy1FMH3Y5WVec7qitt99xl7ksMXLXmpo+qrqFtWvlScT2XU19LDx6l4zc52Pnpgt1M8o8J4OEGbRMUmmWDUzf5BqT83/1IDhQASzgxMWFZedYHv/stv095azFwAQDAcojIpjz/Lb+h0gPMlFdQYtIAmplfbG1y1cRv+dVncjFP5yMpdj5WSHJbVPvYb/l11q5hRc+8A/SzmfmkzIOmz7Qg8RvvOMTOjKGkwMgq4XlXmYdQM5lI/MYf8BFwlCwpCoNCYfA8q/USqI+l8jt+Ms9LQm9fWPqdmtonTnLBl6vsl8uo2l4u0FnBw84jr++stlrW/byWcXJt1+MIBUc6foPvi8/ykm0Fas9C2mEI87Kmb2DkN9C8VxKPpeXHJaoeQO0XBH6lpovZ4PcDKO5VZtSMAhK/Tv+tfMyoWX5MGLbN+2NbpmuR+BUFL+LGwvgxc+p536JmFJD4fbytPn+6GeS0FppPI/h56+/jZ1ND9b7LePDkeyll0YbklzO1SPy6769dLED+jQx5hKCEFS+8ZIh/evtcvXr5WtIWo71Rp633u5fk7A0IEZUxulqZfPro3gOpPkbr1fbY7U2NsYm9cSktKRTOb6C9IMnL3kyAk5NPRFbTdr+tppqynY3ianmtPRepuWI6fqTRFyVZB3WVFJXUtA86O5vs0HE4YWtp7hGcS20uA7/J0e8Pw/cZ6+ntdHB1szXQs47yc/MOvVZS+Q/x68F7oGK2Vg6e7o4GRuahp4lhMQUP31DtBiR+n1/EnXB3PHjY09nM2Ox4XFhU6p1nn7unKxHXzy/5yX5eLoe9XW3NLN3OJiSkFVT9oIaUkPhNdD3Pjjp5+KiPm4PZrn0xiYkZN97NelxI/Ea/XU+LO3H8uIejhaVt6Nmz6bmNswJUfl6HM6umfpYuEJd12C1ok9yUI55QITBfO7Pq9YtHyamfYjYqbjm5S/ttsbpDQEROI03+qKsp0E5x/XppKRm5deZ+OflXwPZcy7a33e1/eeYe+vNng9eTTx11tdvn6BV49/79s+FBhR+uXDhf/HrmMB0Dv4mht8+KkqLCI6JvlT3KiYnM6qt/WAk7SoDEb6Kj/l5ezsVbTx8XxcSntnW+b4P/xhIx/zD+vbbs1t1XFfeTE5Mbx3/0wOuQ80ek/qqKBy9flF1JSXs4NUHz+yCI30sGgSlS24dXNa8fFqRcukH/A7Lf/P52sLPhbWN50fn0K610NVR+WXknbo2/L9Lcue9OCvv2YxGh3rmly0TFl20qKq/UXnnUeOUqFbOL73IuZvjuCwoJojl/NjnyqOjG6+9fv3z99v1H18DgQNevzoGBnx0dXbOONkP+YaCv51dnx8+fv7qHhof7eroHxwf6+4ZmE5MI+YexkaG+HrAMDg8P9PYMkMaGR+BTGil/NDk2NDjQPzgyPNTb2zcxOT4BPzuJyG9qYnR4aGh0ZBg0PcemaO7/3e/fSeDd4L+Bvn66Hx3/jt/UxPjo6OjwYF//IP37f8OPNDk2BjW/f2CCrobKL/HsvsinN89t2+Vx6fT8HZ42+45k3Ngku0jNJPpGrZepta7SDiuvszeOR1b4GJ8PtzpEs/9Nvq+v75n6x/L37xf8WfmX/36Bnb78BgNzc317P1NdDbM9fOI2gbFuhtKJtzSDvgw2amkbxFypcFi80qHTff2HjIV2/w/9/RfSHzP838xvz56cGwUXovdZWwYEubi4Hg89e6ssMjYj/WJe4d2GpjvF94ruVhak52UEH/H9WupI7z/8t+V/Mb8/L/+b+Tl6fv3V/TTX3944/nykn9ftspIX1ReKvl2K/fI8vbyBNDVZ/bisprnjvr7/hc9VaWH/c/z+dHr8f84vJUk3fjjyZOePZu3tT8Otrf3L3bfGR7gd9trmdunuy0nS5Gil0ppV0ltTTvrnv+zP+Dv//qT8q/z+fEmftV/i9YJrgoi1tZVGpoWnHOyO5LroR5w84H1Iz/ls5s03dXX1NzXUVNbsTAryTSmoSyISM+rnUm4SiZG1cxF4FkQMeDwXgbpYIvHanHSC/v5g3VwEHp0gBj2fi0BtBJFYMCedLhGJyWBb/lytEuLfv//5f70w/acV+Fv+rcL0n1bgb/m3yn8BC+vX0QplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjkyMTAKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjM3IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTA5MTYxNDQwNDYrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My40LjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My40LjMpID4+CmVuZG9iagp4cmVmCjAgMzgKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMTg1NTAgMDAwMDAgbiAKMDAwMDAwODA4NSAwMDAwMCBuIAowMDAwMDA4MTE3IDAwMDAwIG4gCjAwMDAwMDgyMTYgMDAwMDAgbiAKMDAwMDAwODIzNyAwMDAwMCBuIAowMDAwMDA4MjU4IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDQwMCAwMDAwMCBuIAowMDAwMDAxMDIyIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMTAwMiAwMDAwMCBuIAowMDAwMDA4MjkwIDAwMDAwIG4gCjAwMDAwMDY4MDAgMDAwMDAgbiAKMDAwMDAwNjYwMCAwMDAwMCBuIAowMDAwMDA2MTg4IDAwMDAwIG4gCjAwMDAwMDc4NTMgMDAwMDAgbiAKMDAwMDAwMTA0MiAwMDAwMCBuIAowMDAwMDAxMzYyIDAwMDAwIG4gCjAwMDAwMDE3NDIgMDAwMDAgbiAKMDAwMDAwMjA2NCAwMDAwMCBuIAowMDAwMDAyNTMyIDAwMDAwIG4gCjAwMDAwMDI4NTQgMDAwMDAgbiAKMDAwMDAwMzAyMCAwMDAwMCBuIAowMDAwMDAzMTY0IDAwMDAwIG4gCjAwMDAwMDM0MDAgMDAwMDAgbiAKMDAwMDAwMzc5NSAwMDAwMCBuIAowMDAwMDA0MDg2IDAwMDAwIG4gCjAwMDAwMDQyNDEgMDAwMDAgbiAKMDAwMDAwNDQ3NCAwMDAwMCBuIAowMDAwMDA0ODY3IDAwMDAwIG4gCjAwMDAwMDQ5NTcgMDAwMDAgbiAKMDAwMDAwNTE2MyAwMDAwMCBuIAowMDAwMDA1NTc2IDAwMDAwIG4gCjAwMDAwMDU5MDAgMDAwMDAgbiAKMDAwMDAxODUyOSAwMDAwMCBuIAowMDAwMDE4NjEwIDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMzcgMCBSIC9Sb290IDEgMCBSIC9TaXplIDM4ID4+CnN0YXJ0eHJlZgoxODc2NwolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-09-16T14:40:46.157923\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.4.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torchvision/utils.py:50: UserWarning: range will be deprecated, please use value_range instead.\n", " warnings.warn(warning)\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDQ2NC4zIDk3LjI2MTM5NzA1ODggXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnichVXLbtswELzzK/bYHkpzH+SSxxhpjebm1kAPRQ6Fq6QOYgdOgub3u7JlhLJV6WCBGpMzs+RoifDgZlcI9y8Q4MF+b4CwgNl183ezbr4t5rB+ccHwrZMknm302I2KekrIRQ0Jvbc/zu2ccdrkhdHdOwxegdVHYY0tVVSP78BjB5Tgw4njsKQGjPTO7aFPJZK8QCQfUZNoiDnDcwM/YAezK2qrQqsKrarQq8pZVXtbrdDWFvmMdr2F2VeE6ydYuiXsT1zBSmr5gs8doyGOk0fOxLGusALFh65KN7ddeXN7ewb4FIyLyZeYo6gWQaD2zcTdfAWzL2imYHV32P3Vb/cTPuBHuIXVjfu8ckt3MOFy9KwcqdTiFTgmrsUHFIwiiHlSnOlSHY0/hZxJavkaHdNHyp6kiGSzKpMGkgwYyOzz0X9toEJHDah6yaJClp3p7S/p0gAx+9AeHtUGanTMABHbmsBSWgfT5095wEEmz2jZTz0HFTrqQMmrxiTM7XlNOkjh0gEz+thPf4eMR58sfqG0/0/KloHwcbZv9hidWrpCR+UVLXyJGJO1jkkHRAPpE7b+RIemUze3Ch1zIBQsfRgp2rfK0w5iL351jC3ulBSzCRVPmpHGiBbNrnn+9bp52sHmtRvVzAQ3x7vg0OX6N8F5I7/o0u77eXPfDjR3mzV5JZzmvK/7D9PS/QMFFmA7CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKNTI3CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ3ID4+CnN0cmVhbQp4nE1RSW7EMAy7+xX8wACWrMV5T4pBD+3/ryUdFO3BECNLXOLuxEQWXrZQ10KH48NGXgmbge+D1pz4GrHiP9pGpJU/VFsgEzFRJHRRNxr3SDe8CtF+pIJXqvdY8xF3K81bOnaxv/fBtOaRKqtCPOTYHNlIWtdE0fE9tN5zQ3TKIIE+NyEHRGmOXoWkv/bDdW00u7U2syeqg0emhPJJsxqa0ylmyGyox20qVjIKN6qMivtURloP8jbOMoCT44QyWk92rCai/NQnl5AXE3HCLjs7FmITCxuHtB+VPrH8fOvN+JtpraWQcUEiNMWl32e8x+d4/wCVT1wmCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTUgPj4Kc3RyZWFtCnicPVJLbsVACNvnFFyg0vCbz3lSVd28+29rQ1KpKryJMcYwfcqQueVLXRJxhcm3Xq5bPKZ8LltamXmIu4uNJT623JfuIbZddC6xOB1H8gsynSpEqM2q0aH4QpaFB5BO8KELwn05/uMvgMHXsA244T0yQbAk5ilCxm5RGZoSQRFh55EVqKRQn1nC31Hu6/cyBWpvjKULYxz0CbQFQm1IxALqQABE7JRUrZCOZyQTvxXdZ2IcYOfRsgGuGVRElnvsx4ipzqiMvETEPk9N+iiWTC1Wxm5TGV/8lIzUfHQFKqk08pTy0FWz0AtYiXkS9jn8SPjn1mwhhjpu1vKJ5R8zxTISzmBLOWChl+NH4NtZdRGuHbm4znSBH5XWcEy0637I9U/+dNtazXW8cgiiQOVNQfC7Dq5GscTEMj6djSl6oiywGpq8RjPBYRAR1vfDyAMa/XK8EDSnayK0WCKbtWJEjYpscz29BNZM78U51sMTwmzvndahsjMzKiGC2rqGautAdrO+83C2nz8z6KJtCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicTVFJigMwDLvnFfpAIV6TvKdDmUPn/9fKDoU5BAmvkpOWmFgLDzGEHyw9+JEhczf9G36i2btZepLJ2f+Y5yJTUfhSqC5iQl2IG8+hEfA9oWsSWbG98Tkso5lzvgcfhbgEM6EBY31JMrmo5pUhE04MdRwOWqTCuGtiw+Ja0TyN3G77RmZlJoQNj2RC3BiAiCDrArIYLJQ2NhMyWc4D7Q3JDVpg16kbUYuCK5TWCXSiVsSqzOCz5tZ2N0Mt8uCoffH6aFaXYIXRS/VYeF+FPpipmXbukkJ64U07IsweCqQyOy0rtXvE6m6B+j/LUvD9yff4Ha8PzfxcnAplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTQgPj4Kc3RyZWFtCnicRY3BEcAgCAT/VEEJCgraTyaTh/b/jRAyfGDnDu6EBQu2eUYfBZUmXhVYB0pj3FCPQL3hci3J3AUPcCd/2tBUnJbTd2mRSVUp3KQSef8OZyaQqHnRY533C2P7IzwKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIyID4+CnN0cmVhbQp4nDVRu23FMAzsNQUXMCB+Jc3jIEiRt3+bO9qpSNO8H1VeMqVcLnXJKllh8qVDdYqmfJ5mpvwO9ZDjmB7ZIbpT1pZ7GBaWiXlKHbGaLPdwCza+AJoScwvx9wjwK4BRwESgbvH3D7pZEkAaFPwU6JqrllhiAg2Lha3ZFeJW3SlYuKv4diS5BwlyMVnoUw5Fiim3wHwZLNmRWpzrclkK/259AhphhTjss4tE4HnAA0wk/mSAbM8+W+zq6kU2doY46dCAi4CbzSQBQVM4qz64Yftqu+bnmSgnODnWr6Ixvg1O5ktS3le5x8+gQd74Mzxnd45QDppQCPTdAiCH3cBGhD61z8AuA7ZJu3djSvmcZCm+BDYK9qhTHcrwYuzMVm/Y/MfoymZRbJCV9dHpDsrcoBNiHm9koVuytvs3D7N9/wFfGXtkCmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTggPj4Kc3RyZWFtCnicPVC5jQQxDMtdhRpYwHrtqWcWi0um//RI+fYi0RZFUio1mZIpL3WUJVlT3jp8lsQOeYblbmQ2JSpFL5OwJffQCvF9ieYU993VlrNDNJdoOX4LMyqqGx3TSzaacCoTuqDcwzP6DW10A1aHHrFbINCkYNe2IHLHDxgMwZkTiyIMSk0G/65yj59eixs+w/FDFJGSDuY1/1j98nMNr1OPJ5Fub77iXpypDgMRHJKavCNdWLEuEhFpNUFNz8BaLYC7t17+G7QjugxA9onEcZpSjqG/a3Clzy/lJ1PYCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MyA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIwID4+CnN0cmVhbQp4nDVSS24FMQjbzym4QKXwT87zqqqLvvtvaxO9FUwwYOMpL1nSS77UJdulw+RbH/clsULej+2azFLF9xazFM8tr0fPEbctCgRREz1YmS8VItTP9Og6qHBKn4FXCLcUG7yDSQCDavgHHqUzIFDnQMa7YjJSA4Ik2HNpcQiJciaJf6S8nt8nraSh9D1Zmcvfk0ul0B1NTugBxcrFSaBdSfmgmZhKRJKX632xQvSGwJI8PkcxyYDsNoltogUm5x6lJczEFDqwxwK8ZprVVehgwh6HKYxXC7OoHmzyWxOVpB2t4xnZMN7LMFNioeGwBdTmYmWC7uXjNa/CiO1Rk13DcO6WzXcI0Wj+GxbK4GMVkoBHp7ESDWk4wIjAnl44xV7zEzkOwIhjnZosDGNoJqd6jonA0J6zpWHGxx5a9fMPVOl8hwplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTggPj4Kc3RyZWFtCnicMza0UDCAwxRDrjQAHeYDUgplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMzID4+CnN0cmVhbQp4nEWPSw4EIQhE95yijsDHH+dxMumFc//tgJ1uE2M9hVSBuYKhPS5rA50VHyEZtvG3qZaORVk+VHpSVg/J4Iesxssh3KAs8IJJKoYhUIuYGpEtZW63gNs2DbKylVOljrCLozCP9rRsFR5folsidZI/g8QqL9zjuh3Ipda73qKLvn+kATEJCmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDAgPj4Kc3RyZWFtCnicNVI5bgQxDOv9Cn0ggG7b79kgSJH8vw2p2RQDcXRSlDtaVHbLh4VUtex0+bSV2hI35HdlhcQJyasS7VKGSKi8ViHV75kyr7c1ZwTIUqXC5KTkccmCP8OlpwvH+baxr+XIHY8eWBUjoUTAMsXE6BqWzu6wZlt+lmnAj3iEnCvWLcdYBVIb3TjtiveheS2yBoi9mZaKCh1WiRZ+QfGgR4199hhUWCDR7RxJcIyJUJGAdoHaSAw5eyx2UR/0MygxE+jaG0XcQYElkpg5xbp09N/40LGg/tiMN786KulbWllj0j4b7ZTGLDLpelj0dPPWx4MLNO+i/OfVDBI0ZY2Sxget2jmGoplRVni3Q5MNzTHHIfMOnsMZCUr6PBS/jyUTHZTI3w4NoX9fHqOMnDbeAuaiP20VBw7is8NeuYEVShdrkvcBqUzogen/r/G1vtfXHx3tgMYKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1MSA+PgpzdHJlYW0KeJwtUUlyA0EIu88r9IRmp99jlyuH5P/XCMoHBg2LQHRa4qCMnyAsV7zlkatow98zMYLfBYd+K9dtWORAVCBJY1A1oXbxevQe2HGYCcyT1rAMZqwP/Iwp3OjF4TEZZ7fXZdQQ7F2vPZlByaxcxCUTF0zVYSNnDj+ZMi60cz03IOdGWJdhkG5WGjMSjjSFSCGFqpukzgRBEoyuRo02chT7pS+PdIZVjagx7HMtbV/PTThr0OxYrPLklB5dcS4nFy+sHPT1NgMXUWms8kBIwP1uD/VzspPfeEvnzhbT43vNyfLCVGDFm9duQDbV4t+8iOP7jK/n5/n8A19gW4gKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxNSA+PgpzdHJlYW0KeJw1UTkOAyEM7PcV/kAkjC94T6Iozf6/zYzRVh7BXIa0lCGZ8lKTqCHlUz56mS6cutzXzGo055a0LXOAuLa8L62SwIlmiIPBaZi4AZo8AUPX0ahRQxce0NSlUyiw3AQ+irduD91jtYGXtiHniSBiKBksQc2pRRMWbc8npDW/Xosb3pft3chTpcaWGIEGAVY4HNfo1/CVPU8m0XQVMtSrNcsYCRNFIjz5jqbVE+taNNIyEtTGEaxqA7w7/TBOAAATccsCZJ9KlLPkxG+x9LMGV/r+AZ9HVJYKZW5kc3RyZWFtCmVuZG9iagoxNiAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNyAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggNTYgL2VpZ2h0IC9uaW5lIDcxCi9HIDk3IC9hIDEwMSAvZSAxMDUgL2kgMTEwIC9uIC9vIDExNCAvciAxMTYgL3QgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDE1IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDE0IDAgUiA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxNCAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNyAwIG9iago8PCAvRyAxOCAwIFIgL2EgMTkgMCBSIC9lIDIwIDAgUiAvZWlnaHQgMjEgMCBSIC9maXZlIDIyIDAgUiAvZm91ciAyMyAwIFIKL2kgMjQgMCBSIC9uIDI1IDAgUiAvbmluZSAyNiAwIFIgL28gMjcgMCBSIC9vbmUgMjggMCBSIC9yIDI5IDAgUgovc2l4IDMwIDAgUiAvc3BhY2UgMzEgMCBSIC90IDMyIDAgUiAvdGhyZWUgMzMgMCBSIC90d28gMzQgMCBSIC96ZXJvIDM1IDAgUgo+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTYgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjU1ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDQ0NyAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgNTMgL0xlbmd0aCAzNiAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NDcgPj4Kc3RyZWFtCnic7ZwJNFVf28CdcyfzPIuMkRQqZCikookMlQwpUyGUMYmuIVJK5gwlQ2VqQkQZylSKiNAgSYUGlDLf6zsXdzqXv/retd73/b7Vs8q6+9z9nL3P/u3n2Xs/e59Lh/0r/5eF7j9dgb/yLwndf7oCf+VfEjrov6PlBg1l/3B7bGJSUrzj9uQb94+bObm4JmWFH1APjbFKbH1aElrQ5OPkfL3hQpCDDxZ7pe1PJB+LjWj+E4XHwdjAqj9RaI3BYq//UZ1ysNjYVkgRkt+5f2tbZQA2uG728+9otDWfxWILpj8R/8AykD8RKgH9S8dik2m+halM17d15lPb/Vl+hyxWrZALv+QYlnXjRqbL5oziWv/dbq72aUUXD8mHRm9N+Pz1+YnKfjdbr4c9SScsPLHYwqk/kSdYbNzEnyh8PYkN/PAnCviLWOzDP6pTBRabgv8ThfeB2JBvf6IwDvWp+pmPCxY0k6EAi82aTc+vQflN+yy/0sfYjJcDg62PS1++bmhuCnQyTqz+WGRzLHqw477L9qVaFk0V8n4FZscHShPs9hgc+S/n91tYyPzwuN8qgswPN/ZbGmR++JHx+TJRgSLzGxmmzTmHctssv7KnIYnFpfeuJ5+7ev12WnxKPDa36kGas53z9dwMCz1TG9fiAqdTCevNChNP+R6x9v4v5vcPHZdaiPxw45M0NOYkSuI3OjoOKwSHm5yjXBK/seHxSXgBY3O1B5Hf5PDIGExhYmxsjicj2t+diuCjzlbmFvraRlb+esuaPrVW5h/18t+qrGbgqLP16tW0+IzmNG9RiS12PhlJAdj/Yn64ibkAznFtlh9+4geNNeEn4YAI0j3LD//tO1xhfGxknBY5id/AZ9jtIBoDP+eoJpHfcO8oTGPy5/fvc/QRIr+L2fnHVZwcRfenx2K3amtdruiJd/KMqXXZvGOz1plrRWf27jDYecjvgPVen/MtT26c+8/ym5zLk5D8J82Xk2M/f83RuiT7g+PDTwx96/sxQqNA5Dc1BqOLn+zvfdc7QFME2f7gN8MPfGjvHBiDK5D4TfyC2evEj3etzX0/5uUXHlfkI+3rx2VblHNeV2fjqezOICPPxBdehnb68pcf3PFWVVZStg4KO255Mr25vihyPn7jQ58/9/X0fh78Sd1g8/KbHO3t6fnU2z8Cq9m8/CZHPnV1dn0Y+AW7Pu/8BffrU2vzi46+n/BnJ/KD+0r8UN+zupqmjp/wCpP4wcwA/+trfdW9qvrBUZgCiR+Ougj8eP/zsvySuk80YxyRH9yJ4L6+vncrt7J9GN5HiPysjPft3l5as3uTRlhqvs1yr+BTXs5Hj3oYG+ceV3by9VmH0XW5d+3ClQCZ1NdmFr7+8/CbeJ3tcMDGxNIx9nor1UxtXn69j8xNjPX3hFTC/MW8/HpqjBTXqK8Ph5c/Jz88Ho8bLLbhkJJb5ZkD91dE/wl3VMNFwez8opLbr3fBFLrnnH/iJ0cqIyXYufmk0uphX1HMP6k0Rl5H67CysnFgK+CzGvL8hUomvl+wYWFhYTWsgJdP5LfXYLve+qJSzQ36praBbrbme8yO+JqZ7bc4WBKmczJ8mxyHR0SC114PT5eiVyEh8aGU/MZ+tFRfv5YaF5cQF+m/T0VltexqTRNr/2t1A+T+SMuv/2luclxsTIib/DJZaXl9j9hbjym/puWHH2zJSoyJOXlkKY/AIlFDl7SyFsqnn4vfz9tpMdFnHJUQrJy82nap97upjHau9UNVbmxUxMHNIIaZdenBmII+KoW5+L25kxQd4WrIAqIxbDYhd3qovpyL3/f712Ij/A3FABBEGvgVtlJ/Owc/3NPCuKjwnUoAJCt8ch5TO1YiP2ujtUoKmTn8G934WJUCbyuLSxw6o6C0wzm4Nn7nzWxeAambBeZyTPI2z5ubG59Wx1LyG+yM91CXE0WimJF0M0LPjgQZ9LAUJkjDD/8iSI0bCQIzgmBiYt3gT9mYtPzGX8UpsgJEYeAQs73wg/KGtPzw3boSUAkgiIAUUEzye0s+UX49B7/JI2sQxBJQHMu3PqZSmIMfPlufa0YDKoV76fZqqhvOxe/N3hWo6fwAiOCRMk2mrsEc/MaCDVHEdmISMfSjHkqJ/AoLlmz0qa7z2a2qo7XSJVpaxsjRNznUUm/H7XOmx4K8HfdqKgiISmtsNDyZ/eXHYDolvzteGjLszAxQfYBZfgiohggeGceLpDwwfn1NztuWsKNBEg4EAsm1KqCBnIOGX9EZEyUWJFkBxSC65nozRUvC+b27fsyMh5HQrkgkAmotJIvoplv9FM0F5zdSe/nwUjYoP4hkYmRAIlGsAqYXKO9Iw6+3MWg7LwaqO5qNg50BiWEVNAujHNJo+E0MpXuKsiEAEMMlyMtBz8AstCn0LWUGWn51scqCUJ0YOcWFeOnpGUVUImsovybyq2+U1gusrL90aJ2pkZzd6TXaHg6OORectdRjg2ycfS6e91+EAAQVjAx13ONfvOu9TOaHH4nX42cikqMQEKPqRZrZUfPDN2Up8M10WwQSBAkYIQVuvSsfSc4Bxm/s69mdImwEBTQjBg0JRATNFXhrgORC4fxG605oi0N3RqDZ2ViYGJkwSICe/0zjMNn9wPn9SDusxAIASAwbJz8vNwcrGomWcemhmJPQ8GvK3boEcoMYdoHFoiL83AwYzHKTdxTDLA2/wdfuG+mRIIpZWHr5EnEhNgaUuGH1EEUGOL/Jb9es+VAA1FllVeWXCvMyswhYZVBOYoj8sq+brhXHpuYWP/DUZ7WIv15c76goa5q5R1neMjE84nZJ2RougGd55uOe3BMrtA77kvn9bDi0CAXMwQ9q6g1viB6Oih9u2H8FZsb2QA5uBnoMGknQB9GKZl/m4fcyTJWVABxEi6yRFhUSWcyCgVpaYmv863n4TTw/L8tKwEfPb7hBebmC5hIuFAKU39VEKoGG36dtS+kh02DmMzLT09E00BFjAZAyB5vm54cP1WeBfAhacLPrboNN5tZyAgCDUEDxP/B7cFSGDQEiBNWCXPeZ7vfaJIOk5wkoohjR4Pz6Io0EoCI4l5wIdrO38bVV40Tx2dVQjBtEfj7ex4+6Z2fbOLkbaQpvsvMLDNmtud27zHq9V/D5gOCriWedzLQsvTKKGu8krdW2pIh/fjiziQ1BwIVk4l21ydVSa6uuhBA3BvKlAKjzkehNqPh9ubOTF/oezapq7O7r5+3laWuqIcUJNZ3oht65+Q2VGoqiARSLtrlHUNSp4ICgoAOGUmwgh5IvqXlh/H5esBNjRjLJWxw5kZoQGX4uLsRTZTEgtCavbT5+TTlaEmimRXtdT6ReTboQl5JwUl+WXlCneJCUB8bv8xNXHWYG5h0HA5LzMlISr+accZDnYt+dMi+/8c7MgxJszGq2/tFleVnpWYWJAUqijHvCKaZhMH5D7Vg9HmbR/Z6h98ryc7OLc6O3yLBsjKAYlYn8tFSvPuzouotBMUjKrVwmraYsrW0SkP7C0fDFnROubrGuWzLzIwqbLsRl3i+zNDQkxz/xjaoC04aEYBRY5Z4y1nQ+K3WHpjwHhmBRGweJVaPkh2+2kSZ4TjZxYl99/SB051LIoLhXfpyTH+59MhcGQLCKRD2edR34tpvG4gC4eG/tPPy+Gq5axMa8yPEJ0TtNDHhuABgEz5bPxy/eWkWckV+z8TPpyuW9bCyLLr4heSsYv8ZwUw0OTqH8btKV+kMSaNlA8j1h/IaKwi0leRYFvp1uFUK2fk9tYNkeiiETxq+72m09P//m9gHSlchdaHF9ikkrkd/tsuggNwfz5VvsJaV2m+zOjtPbuC0u9aqxZpqz6Go1JTnp9Pyws2f3+T332lr8pCWRyA9fEc6JIfDbERSXU9/Ri//54XNvS3We6SoCP6G9D2aLoeA3WXpGmh0A5E/nPX7XP3NpZPB94RlhNIDmuUAcnKn4DZ82ZWPmD7n66CPJcwz31ZmzgIwi2cSHp+b3ttxEU8EvvfrVD6Jzwk92JDCiGZSDSHmo+eFSPXZ4ZFY0/iRbQ2+tEAajd5TUvDB+rYlHDucW13wlj5BDr/VAVq1T74hpGL+fpcnHb5VXd5JjFROvgwGWpRfJeWD8OmvP59RUPx8mTx26CpmYBFOaSHcg8qtp8TnsqKVq6HhCTcP9sFteirmRSeSlTCfzRHvhNesUlDVz8jyPY03cnx3Sqv/0K4PID5fphgKhBYOg371n78nP0R1mtZiVjo5Z5vrsFQp+44kHODBIKcvKrxQVneoo1hSG5sceubNpSn5jnw9t4JXTLe+kVJgc8pdFI+hj3hMtkopfe7GtuVneG+poxX0pVgS/0wjxIoxf3lnP9PfUEZQvKzmBpSak+QWMX+etyPgPsIiWFTtKxrKFmILxG6nLS/4wShX8m0xhxfAdKyGl4fb3PLeNOsAw2cbHQB9yn/RgRH4njiz2+Ki65G70oaScklupSpongoPMIkdHv1fdTEhJy6zqbM0Wd8087BqSUXD5+uM4Ir/JRFsQsjSZ0EYcZTH4iY+X9ejoUGzpsxco+I3sVUQhuGp6YSGlsYJjhBnPqdk0Jb+OsgMGa4pGcbDFWm08NwBYJs6aBzW/gXfZVWPwMGlHxGqQbdez77NJmP/88mkUrvDz5HZASIdk8zB++MkJeKgcF22CEdYlLYPg8xfcJI3C3a28HPtySGkYPzweXiV81wYxhONlGn5u+7iMc/TUT4cGXUm2T34QevJCXIT5+eGO+huFD/wcDme+e35Vwi7ew8EyvjD+XHAAyf7SD4HQTFOtsg9Wr/6K3XR0GIFrs2lKfnvkkGzSr+ChP9zTcwAd/c7Y2SQlv84H9hY2T6bg0nqdj47O/e5sEBjG70NzN43Ch2trQRbLj0Qbg/Ebpt0P+JVkCfBrzsdvDsFnODEIaD4lJueJn1HIZIX1IrZdV0npeeJnFPJx1zLQKoY0ZSXyO2jEpHjAfJfh0UtZQZiAtx9bb15Ltor4evdyXPHHnTJywa+fpMqaB/rYrgnJO++q7kTid8Mb4gdug5cyOvjaho6OSZroDin57ZZFLFrbB9fAdVwE6BidicAp+XXX2jtFv4YrTL0rhWZO5zpmHwXGD2be0/K5WAdksSclF95/H77uCPBq/AG/qXx/Rr41dcTUwvwmHnmLsWy/TEovzK/HXgk0CaPhdypgy6GLtxKl3Uq2KChqGaadVk58Guu17WLF/ZRjCacX+768dMbP51jGnSe3UmK9dZxJ889v+Qg6kHPXd+pALL7q4hYxOjrpEOL0npKf3dpF66zh7TD8SV8CoON4RBwUKfn96rtV0klrHUGyaACRRrQman6Tc2x3T96TZUPwuZDSC/Kb+KTGD0jqzzf+0cr4oK0wQtyYtKZZkB/ua6IQA7dzPunCgvz6H4mxII5co/GfgVj3c/lNRdIuBbpqVurSIa68YZWhtkoXa/PjPa9f5ndrOBccc8ypoGnkcfZZLyM38vqvhhkJcuk9bHrV/Y3c5XGXXYWY6ehW3SaG8Cn4jR43ktCxaf86QmUh7+4vpqND8JGcHiW/iV819b3wvZlf7aZ0dGhO4gRp4fMvuLZIgA4l60u6sCC/zmIeAFCxJ8WwF+T3+aEWAK44/JKYXpDf6AN3ACF4upJ0YUF+T5KZEJiwCpr555YtTxuf/OgwtvdPK+o8LG5rKr7biVf9TEmFQ1z77VNMplknA6IsNCo/To31OjpQ7r83LOcCAIZFa7Z5ZPwgTXPHtwsQpjUb+4k7lJTrh4cRkoqagRkvKKPPUzFKDHR0LNJzr/8mastKeqeopUFbkI6Ob2M5Mb0gvyF9SYCO6cAV0oUF+XlLoADAJYe0y7ogv3wxJgDc8WCAmF6QX88KTgC9lGJ5tyA/U06QnqeW3PWJ/HYYPbmJLby9Y+M6S0c/XUl3GzG1jct0XM9G2/tdrsph13R3dzIxNqttvNvQlpfgcZzMr2WjCAAgmQUkVQ2PXsibdVvj65gAOgC5hWRjFPzw77NEhUV1zdxPRcfefkns3MH8SDo68U2kYZGK33h+4rHY1KRLGVfySI66ejEzAMgcf07MA+M3/qym/FZWZs7Nz8T1xYASO4DiyyKHvGH8Bj5C69bruTeJARrc6AE2EMUa2zbf+n0K39Nekp97g2gM+JFrrGgkjwN5EULDb/hLXdmt2wVEjzzcLkwP8Gn1kX0LnB9uuKM67+btztnk2MBWepB3ZSM5A5GfuVVdhIZ/iIEMml9IYLFs8BFxMcn1m3fsP+jg4NjawLd0vY2piKHfo7tu6dW/qu2Pkvm92r8MmN1AAZVt+mcmyKMrCRE1JiNSMVTxswcizCAHF5eYjNz+Gx9neOA8CQHTNQ6k5qHiN5p6VGO9zpp1m7favPk2c3YBd4+wM6SeR8oD4/frUrjfPmODnXsbZ08CjPeKAiCjVC+ZGIxfx6Orp2zMTPdmj06fLMKPfdsDLUgXUYQzYfxwY09uuNuamp/4NTo+PdX/mgiAmKWB5BxwfpN9TVHH9u0/+G50OrCP/1zLAYDL9lDsCMH44UZ7Ck/ZWO4vma3T9zdaICi5gxwDJPErafv5qelpWViwn6Y0g35QQlzo3vVc/LLmPrcL7z69q2e4zz+2tqE+wN2tqP3c8UPHyPyGnuxnhpqSiR0B/eFR834MucWeR0shfhivXFIxVPyaraQAJBKFoWfgltjsRnBP38ugwQxEhb2fe/9h4sl5SQFOZhY2Nm65jTZvCA9WGggCCKH9g/PtPwzYbVrKz8HBySWrtu/2FB43WhUDDWZL9PvJWWD80o+oSvNwcXKJr9oVBC1mf7VHqwCAqNlTcg4Yv+fnjVX4uTm5BBX1rHvwk/1vz5sCILsDeTEH5/f9ylFdKQFubp7l6jtL8FPD3ZdcGBFou6R545/4yksmyot4uLmlVhuexk9Nvr/jKorgNEiiGEuI/J59a29tfdt4OsDHTBOpH5p2JQNrzSiwwepIQkFzTb7n/h1uMR+flx22c0wp8jjs6UfmN9Z9iAXiJ7dJQ1meDWTSibpbU3MjSoTAzzu19f0XHC2/FhtpAORRV1ddKQIIaZbX1NTcDV0HzV4YQt/0E/0pNb8HwXzMzMpqaspyCB6FjOqamsoQM8L+g/1nkuuh5tffsmkpu5yKitJKZoblvjVVleVnrVkBQN6sl9xYVPxG+/x1+cRVlJUUBEAJg8rqqvuX90sCCBnH+fYfcP23zGWFV61erSAF8MrfqK68c3XvGgDNd7xoPn4/X7vricivXqUgR88oEVRVfT/70AY0mtUzl2JjlIrf+Pd4J6llKxUVVnAjF++qqn6YhdXmQgla3KDoRER+j5p2mzu9emsuxxAVCi7Vv16SV17AtaH4sC7n4cH7hV3uSOV9D3x13V2M16m6B5wLJvMbeW2HAUBU9JfxvkeakEtDolAo5Mx+0GIF/RNXh2n51WnwA/RmExMjrf6QAoqggADp6FCc2gfyiMMTFb9ffptBhHDvxMSn+0wzJUAaEL9lO9M75+Z3x58VBDO/fu1+sZJCAaFs1UDuu1T8OlJXgoDNp57uZw6kOgEgs7wLaYCF8fuVZw8CS9redTVcRIIIyJsQHgJgX3Ly/nz8GlI40Kgbr1/W3xMl7D9DAimwSvrfnI9fT60SA+DSXF9XZYICETPNBNJL2czFLznJUN+wsj3R0+Soo/S+c0VZx67k8it4r5NWPPn9nNcFIxb7M/cCTKxsDhw5nhlOsf8wNTngSg8KmBSPTo307OEhBGOIm4EAK6+E+q6ZeD4Vv0cqPPRG0YQuHC+AIWWH7E9EwfRQWDstv59O6sAae8gx/2qSIB+hoAO5l+7wiUyvnYNf4h4GoU2VI6NDnzdzkRQAUFDeNSQtv2qSlt8j+8X0yqd+/Bz6HMCDmrk9HQDS86/yiL1V+nSIlt8393WAuH53/1BfsRAzOL0JDf1n4l1nn1jwsO3LHPxiLOh55Mp7v/S8UeUkVYlRQNMsJb/27ec5+JW7LGJcHNT58dMHV0EkML3LDYBowdU7o3KfdM4eQSXy83Ay2bI+99mzOwlWBpvOPqxI3nbuiri4BheLUfzng+tNVQSibheFWBlaHLtU9DxaxJry/IQ3AygXN93qzksQ8E14zpn5HBW/2lXcLJHTOxPZcsyUG74ggplb7A6eht/QXkXALp0wzHeqCFAqIBlFFQxi5+B3Uh2tGNA+CU0AzMTIpzRAELVIwcD57Cgtv0Ildjbba6M4/GSUFD2RH4BCAmIqB08kfqLl92ntIkDr8JeRqcn6FTyQXRAOaiBADD0gpH341I1WWn54GwlQ0uDp4Mj4r22LQWKFMKwY9u0ukfeaZ07CUfFLFqdnUY/pGRmfCFqGAWfPK6DYmOnX708ofTtGxW+jzpPq2yd80oofbV8upWXR/ORMWnao285lMmHephGxtmY7r1eU2Mgkp2Ijog9YUK7fJwc8ORHC1lWE6IM5FwjfhGe7+ISGX52mAKNrFmE6FUWPoMRBcL+YhI84OL+fB1UBk3BoaPxWzYWiUkBgmLicp+e81PxC16KXuUGNOPhKFU22PwSIYGQXFFv/k4Yf/g6BXxZ+aqzDffZoANT5EJDDYuGRkF7xhIYf/qOaILD5+DBuqvsGO/2MxRLcKApg5FuqsCpmDn4WguAK21eT+G+tKxhnT26B9PT0CBSvpJLK/pnWoeJ3QQjDbX5jDD/SZcOImTn0hGBiYUCDnCJrNTe+peKnsrKmNMPV8VJmru6qdbomOTdj0q4Ge1htN0rx1jgZf8LPLz75go3WxaQj7kd2GFG+f/Sjeh8bgks7orzi3m1tZsjCJWT5FsusVhSatiwE58yOHiU/XPUqboz+sfLy0jxHQh4uGeHFCirqi2ccHXj+9SSc35DZCkDDrqi8/GYEwTYYZCRFV6lpLxNEEXo9av/PSRp+x1egRI1SystvX1pCuKeolKjCWq01EuzQpJeJZcW3STg/3C1pZubtARXlxZcNCbXgkRKX1VqvtUyYgYmZjZ3zIQ7OD9clzwWo7rtXXnbVDxr6QCZJSal1OhtWy7Ays3Dzcs1s4lLxw5lwAUuMMisqbiYJI5EgUlxKSmW9nqYiHwsLpwD3hjFafrFcKA79Mw8r7mZsQkCeQFBaZuWGLZuVJNhYOUUXibTiKfktFzxsoWHmdtNrrbZpcrjvCu1T8bFaW13S75WGbdjhUNNQryu78mj2EXOtZRIGu/Z4kPm1GUnO9Ax6NGJ6UArNMPNKeFJ2YAWBDZNw9EM4v5ECegTB0hAIxLS5brl4xLv4xcejejN98mTDBJzfgCwjoQSCAuHgnWjiad/aLtw1W67pTrnr3TANP/MZc0BOawAov0jfe1/G2wJU2TDQQCL++gec3/jlGWOYnoRA0y/DiKCM0Ymh5IPCbPRQFyn6Bec32gaNYSh6RgxyugSE7Nmw84M4XFmUojAT9HTe06tUSn64kQ0AQM/BwoRGTg+VLIHnw9twuI5sE3keJgBQGsbD+eHDodGOi4eHHoWYPuJlHRdfisMN3fVTEedHI5kaxin5LWPXsI0K9tLfpLl2d1ZC6BpFjS07Pc/kNHZ1X93ll/A6083c/FBYSkqMr9PBC75rSfsPU8N1OsIzzowAAymgvLek5W5Vy+cPZRf9FQQA3pXVH+H8+m+woWe8JWSt7BIHL754WP2+/1f17aidsmh6/rQ+PJzfoDov0QWiOMQcI1vqa3uHpl6VJvho8yGFD3+foOF3gIM46iGZpUzDaxpru0fxA4+zI3evRgho9Y7R8EuHXBrUPwgo0OJaEQXP6tpxUxMvyi+5GbPxLKkaJ/Ijxtcn3nCDhFOP08XwywemN9Q3Qpk+NF07c1CSTSJseoFOyQ8/AfFDMaCnz3KyKjrF1TU2DEBrwteFl4/psoruGJqcgx+AZGVmJHQQ5ArDyPvPm6EWmeh6lBXuzC0g0zBMyU+RU8ixIfUok9R69V03UyM3yyE5liQVNvb8xJc5XL73IUzBAZsYfb60NPfypeeJsnYkfoMPVflnZ5CEw32ye9O7ZsMJPzotFABx3ZmZGGX8rC+Xl2F29EYgRLRLyAcgE42YOVfeo52//NgqQZjcQQaFZhFfV0bee2rxWcGscXrmtlT83MSQ0xEhSEFoc8oQKQT2K8GKabXVEB7ObyKTh37mqC+aiXO9F8X7KQ+iROW2NE7f4D0FP/w7QfTMI6AxTKvMSXHbqamugg0SuskjhN1m6vFvM2rGI6DQjGKWZWSFkTpfMS3nmcAglf88h4bGO8jAkRgGDtMYssLU2/ylq/Wahij5Zd103rdrn72HmbaukXtUdkW0lOLag+Zr9Wz731VcifP18dq0O//QGofIN5XFQ/cp3t/ENWotIvNDcOiEvZt9dtxYTSjCMHwIzm9q4uEKbkKfYmAVlpDd4dVPXlEPtq81zp7ZgqBe/3lsYscgmdgF1uhZHE/vIZ+3G/sSZl33kZYfLn43OxqAJrNqpp7nGnrINPCDD9wfzqap5p8Pdy8CQUZe0RUWR2NfkI8wQQ7mQ9St7pkJKyW/qc/QnBhEC4hIGh5KqflEsYSbGCrI7BqYriP1+iFgIwqJ4RaU3GCbVvqJ4mA+fqT1VnvfTBej4pe/ixtEsgmIrbKLLe4aoKjTeH9Vc/co1fiXlJGMtTQysjbVt7LeF3qtvcjMfK+PmciSjX2NqTdvRMYlunjf2b/E5WxR6cO2fGeK+FmnhzorhoFzkcQyOVlZ2ZXWV0nWgf+QweV2Z8Yaqc6fNZkricgqqG42srJ1DE2n3Kn7Ye3TNDj9iTr+meKlpqGpb2LuFRSdW0d53nUqL+bLTONSv39bfFpTXdtgt7V3ZFbpN6oT552ZRFuh4td6Tl9JZ/ue/c4xuVXUW40j5cRwIxW/7zEHVq3farHP7kxaA2wzuu05bop2/jKV46+iuW2XhR02qRH2Zs63l8QqUvGrj9isqmtkbu2WXNVN/dbDRB+xjkR+GusG2jK3LQFV3S6c3e4W/fZDa3F2WQC3wJoPcZKP8ZO9Lb0dOaZiGbHrLjZgw7LPkvnhhk8r8otp2/knJcdHR8U+oCymVp24M0m1fhioDrGJyXnaT/s24mhOxewn2PndrvTq1xNzvVn74xsx+k8d/xwprPky9s+v4hL4UexB1t4dGMX/owb8/FL+F/w/K8DPD5a8xMMP8cAEFv+sezyBw/1jGaT998T+3naX3SeO7N+uo+iaUnnNLKfRfweH3NYHMQbPBwormz9U7zIwiI9183Jbb3yM8v2xycasgNM3Kx4VVrxoaW6heuGj7+Yc+7eQ1+ttqGjp+EL79uIUroOoD+M39Kr3+5xPMT73/vvU5Pte6g1iWoHFr/u6F3qpHcZvqIv2FU9Y5aj5TXwYXEABvv/wuW+hl8FJv1/Q1tXzNcS3McpQjH+xQ2p+INOlLhMVES2zjEiPls6wa4867yzWdYhKTDRbKbUG9vsTuIYXE/iJp7QnVMjy/+L3J37j/Au1LHz+BS6z/H6/WqTzu/lbtu8Ji48y4g/ELlfd3lrt4OTsfbEj9ohpQHFKSlWw3rPHO+TFn7ytNRKpKaOMXxNkjDCGzfk+PlH+8vs9Wfj8EkyI/NKu2Jub+ITnYE3zbh2wtqssj7Ha6HhxoCA+MPbWcZ870YcLs7Yt4cl/dH/Xshd1MSH/xb9f8Nvy/4pfcNSbguPGDtOG9OlZyenktp0c63xaO3vwdUFaav4VE4GmKxdx2PnF7DeoKIo8+Zff78i/kZ/boQMRD2Kjyouun7nwMmy7roHNMnb1AzmxSWNdJZmpyRdiEpIvexpvsb3i65h3N//8X36/I/9Gfi6W0t7vC6+lh3vomNdasi2WWCIkqHcwzv7gcN/Tly03Q21yq16meW4wv3LK91rBg5i//H5H/o38buTbuZ8+HPg9auW6LfZGWmJrT/v6vWyvNtYcylbTMr3z/GtbifM2jVtJ6vbYoJOhgX/5/Y78r/hlzvXrNvMKkV9Gtof3CVf/5+GbjM3s95lp74oICX1cnb/PsD5ug55ZRllL2VWHXfrZF/QOYU+cwBIUXvyJ5GGx557/icKjYGxg5Z8otERjsbl/VCfC7w+2/InCwwBs8OM/UXh+FovN/6M6pWOxSc1/Uql72L+///l/Xej+0xX4K/+S0P2nK/BX/iX5H3oVbwQKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago5MTAzCmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagozNyAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MDQ2KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDM4CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDE4NDQzIDAwMDAwIG4gCjAwMDAwMDgwODUgMDAwMDAgbiAKMDAwMDAwODExNyAwMDAwMCBuIAowMDAwMDA4MjE2IDAwMDAwIG4gCjAwMDAwMDgyMzcgMDAwMDAgbiAKMDAwMDAwODI1OCAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDA0MDAgMDAwMDAgbiAKMDAwMDAwMTAyMiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDEwMDIgMDAwMDAgbiAKMDAwMDAwODI5MCAwMDAwMCBuIAowMDAwMDA2ODAwIDAwMDAwIG4gCjAwMDAwMDY2MDAgMDAwMDAgbiAKMDAwMDAwNjE4OCAwMDAwMCBuIAowMDAwMDA3ODUzIDAwMDAwIG4gCjAwMDAwMDEwNDIgMDAwMDAgbiAKMDAwMDAwMTM2MiAwMDAwMCBuIAowMDAwMDAxNzQyIDAwMDAwIG4gCjAwMDAwMDIwNjQgMDAwMDAgbiAKMDAwMDAwMjUzMiAwMDAwMCBuIAowMDAwMDAyODU0IDAwMDAwIG4gCjAwMDAwMDMwMjAgMDAwMDAgbiAKMDAwMDAwMzE2NCAwMDAwMCBuIAowMDAwMDAzNDAwIDAwMDAwIG4gCjAwMDAwMDM3OTUgMDAwMDAgbiAKMDAwMDAwNDA4NiAwMDAwMCBuIAowMDAwMDA0MjQxIDAwMDAwIG4gCjAwMDAwMDQ0NzQgMDAwMDAgbiAKMDAwMDAwNDg2NyAwMDAwMCBuIAowMDAwMDA0OTU3IDAwMDAwIG4gCjAwMDAwMDUxNjMgMDAwMDAgbiAKMDAwMDAwNTU3NiAwMDAwMCBuIAowMDAwMDA1OTAwIDAwMDAwIG4gCjAwMDAwMTg0MjIgMDAwMDAgbiAKMDAwMDAxODUwMyAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDM3IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAzOCA+PgpzdGFydHhyZWYKMTg2NjAKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-09-16T14:40:46.343108\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.4.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torchvision/utils.py:50: UserWarning: range will be deprecated, please use value_range instead.\n", " warnings.warn(warning)\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDQ2NC4zIDk3LjI2MTM5NzA1ODggXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnichVXLbtswELzzK/bYHkpzH+SSxxhpjebm1kAPRQ6Fq6QOYgdOgub3u7JlhLJV6WCBGpMzs+RoifDgZlcI9y8Q4MF+b4CwgNl183ezbr4t5rB+ccHwrZMknm302I2KekrIRQ0Jvbc/zu2ccdrkhdHdOwxegdVHYY0tVVSP78BjB5Tgw4njsKQGjPTO7aFPJZK8QCQfUZNoiDnDcwM/YAezK2qrQqsKrarQq8pZVXtbrdDWFvmMdr2F2VeE6ydYuiXsT1zBSmr5gs8doyGOk0fOxLGusALFh65KN7ddeXN7ewb4FIyLyZeYo6gWQaD2zcTdfAWzL2imYHV32P3Vb/cTPuBHuIXVjfu8ckt3MOFy9KwcqdTiFTgmrsUHFIwiiHlSnOlSHY0/hZxJavkaHdNHyp6kiGSzKpMGkgwYyOzz0X9toEJHDah6yaJClp3p7S/p0gAx+9AeHtUGanTMABHbmsBSWgfT5095wEEmz2jZTz0HFTrqQMmrxiTM7XlNOkjh0gEz+thPf4eMR58sfqG0/0/KloHwcbZv9hidWrpCR+UVLXyJGJO1jkkHRAPpE7b+RIemUze3Ch1zIBQsfRgp2rfK0w5iL351jC3ulBSzCRVPmpHGiBbNrnn+9bp52sHmtRvVzAQ3x7vg0OX6N8F5I7/o0u77eXPfDjR3mzV5JZzmvK/7D9PS/QMFFmA7CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKNTI3CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ3ID4+CnN0cmVhbQp4nE1RSW7EMAy7+xX8wACWrMV5T4pBD+3/ryUdFO3BECNLXOLuxEQWXrZQ10KH48NGXgmbge+D1pz4GrHiP9pGpJU/VFsgEzFRJHRRNxr3SDe8CtF+pIJXqvdY8xF3K81bOnaxv/fBtOaRKqtCPOTYHNlIWtdE0fE9tN5zQ3TKIIE+NyEHRGmOXoWkv/bDdW00u7U2syeqg0emhPJJsxqa0ylmyGyox20qVjIKN6qMivtURloP8jbOMoCT44QyWk92rCai/NQnl5AXE3HCLjs7FmITCxuHtB+VPrH8fOvN+JtpraWQcUEiNMWl32e8x+d4/wCVT1wmCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTUgPj4Kc3RyZWFtCnicPVJLbsVACNvnFFyg0vCbz3lSVd28+29rQ1KpKryJMcYwfcqQueVLXRJxhcm3Xq5bPKZ8LltamXmIu4uNJT623JfuIbZddC6xOB1H8gsynSpEqM2q0aH4QpaFB5BO8KELwn05/uMvgMHXsA244T0yQbAk5ilCxm5RGZoSQRFh55EVqKRQn1nC31Hu6/cyBWpvjKULYxz0CbQFQm1IxALqQABE7JRUrZCOZyQTvxXdZ2IcYOfRsgGuGVRElnvsx4ipzqiMvETEPk9N+iiWTC1Wxm5TGV/8lIzUfHQFKqk08pTy0FWz0AtYiXkS9jn8SPjn1mwhhjpu1vKJ5R8zxTISzmBLOWChl+NH4NtZdRGuHbm4znSBH5XWcEy0637I9U/+dNtazXW8cgiiQOVNQfC7Dq5GscTEMj6djSl6oiywGpq8RjPBYRAR1vfDyAMa/XK8EDSnayK0WCKbtWJEjYpscz29BNZM78U51sMTwmzvndahsjMzKiGC2rqGautAdrO+83C2nz8z6KJtCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicTVFJigMwDLvnFfpAIV6TvKdDmUPn/9fKDoU5BAmvkpOWmFgLDzGEHyw9+JEhczf9G36i2btZepLJ2f+Y5yJTUfhSqC5iQl2IG8+hEfA9oWsSWbG98Tkso5lzvgcfhbgEM6EBY31JMrmo5pUhE04MdRwOWqTCuGtiw+Ja0TyN3G77RmZlJoQNj2RC3BiAiCDrArIYLJQ2NhMyWc4D7Q3JDVpg16kbUYuCK5TWCXSiVsSqzOCz5tZ2N0Mt8uCoffH6aFaXYIXRS/VYeF+FPpipmXbukkJ64U07IsweCqQyOy0rtXvE6m6B+j/LUvD9yff4Ha8PzfxcnAplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTQgPj4Kc3RyZWFtCnicRY3BEcAgCAT/VEEJCgraTyaTh/b/jRAyfGDnDu6EBQu2eUYfBZUmXhVYB0pj3FCPQL3hci3J3AUPcCd/2tBUnJbTd2mRSVUp3KQSef8OZyaQqHnRY533C2P7IzwKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIyID4+CnN0cmVhbQp4nDVRu23FMAzsNQUXMCB+Jc3jIEiRt3+bO9qpSNO8H1VeMqVcLnXJKllh8qVDdYqmfJ5mpvwO9ZDjmB7ZIbpT1pZ7GBaWiXlKHbGaLPdwCza+AJoScwvx9wjwK4BRwESgbvH3D7pZEkAaFPwU6JqrllhiAg2Lha3ZFeJW3SlYuKv4diS5BwlyMVnoUw5Fiim3wHwZLNmRWpzrclkK/259AhphhTjss4tE4HnAA0wk/mSAbM8+W+zq6kU2doY46dCAi4CbzSQBQVM4qz64Yftqu+bnmSgnODnWr6Ixvg1O5ktS3le5x8+gQd74Mzxnd45QDppQCPTdAiCH3cBGhD61z8AuA7ZJu3djSvmcZCm+BDYK9qhTHcrwYuzMVm/Y/MfoymZRbJCV9dHpDsrcoBNiHm9koVuytvs3D7N9/wFfGXtkCmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTggPj4Kc3RyZWFtCnicPVC5jQQxDMtdhRpYwHrtqWcWi0um//RI+fYi0RZFUio1mZIpL3WUJVlT3jp8lsQOeYblbmQ2JSpFL5OwJffQCvF9ieYU993VlrNDNJdoOX4LMyqqGx3TSzaacCoTuqDcwzP6DW10A1aHHrFbINCkYNe2IHLHDxgMwZkTiyIMSk0G/65yj59eixs+w/FDFJGSDuY1/1j98nMNr1OPJ5Fub77iXpypDgMRHJKavCNdWLEuEhFpNUFNz8BaLYC7t17+G7QjugxA9onEcZpSjqG/a3Clzy/lJ1PYCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MyA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIwID4+CnN0cmVhbQp4nDVSS24FMQjbzym4QKXwT87zqqqLvvtvaxO9FUwwYOMpL1nSS77UJdulw+RbH/clsULej+2azFLF9xazFM8tr0fPEbctCgRREz1YmS8VItTP9Og6qHBKn4FXCLcUG7yDSQCDavgHHqUzIFDnQMa7YjJSA4Ik2HNpcQiJciaJf6S8nt8nraSh9D1Zmcvfk0ul0B1NTugBxcrFSaBdSfmgmZhKRJKX632xQvSGwJI8PkcxyYDsNoltogUm5x6lJczEFDqwxwK8ZprVVehgwh6HKYxXC7OoHmzyWxOVpB2t4xnZMN7LMFNioeGwBdTmYmWC7uXjNa/CiO1Rk13DcO6WzXcI0Wj+GxbK4GMVkoBHp7ESDWk4wIjAnl44xV7zEzkOwIhjnZosDGNoJqd6jonA0J6zpWHGxx5a9fMPVOl8hwplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTggPj4Kc3RyZWFtCnicMza0UDCAwxRDrjQAHeYDUgplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMzID4+CnN0cmVhbQp4nEWPSw4EIQhE95yijsDHH+dxMumFc//tgJ1uE2M9hVSBuYKhPS5rA50VHyEZtvG3qZaORVk+VHpSVg/J4Iesxssh3KAs8IJJKoYhUIuYGpEtZW63gNs2DbKylVOljrCLozCP9rRsFR5folsidZI/g8QqL9zjuh3Ipda73qKLvn+kATEJCmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDAgPj4Kc3RyZWFtCnicNVI5bgQxDOv9Cn0ggG7b79kgSJH8vw2p2RQDcXRSlDtaVHbLh4VUtex0+bSV2hI35HdlhcQJyasS7VKGSKi8ViHV75kyr7c1ZwTIUqXC5KTkccmCP8OlpwvH+baxr+XIHY8eWBUjoUTAMsXE6BqWzu6wZlt+lmnAj3iEnCvWLcdYBVIb3TjtiveheS2yBoi9mZaKCh1WiRZ+QfGgR4199hhUWCDR7RxJcIyJUJGAdoHaSAw5eyx2UR/0MygxE+jaG0XcQYElkpg5xbp09N/40LGg/tiMN786KulbWllj0j4b7ZTGLDLpelj0dPPWx4MLNO+i/OfVDBI0ZY2Sxget2jmGoplRVni3Q5MNzTHHIfMOnsMZCUr6PBS/jyUTHZTI3w4NoX9fHqOMnDbeAuaiP20VBw7is8NeuYEVShdrkvcBqUzogen/r/G1vtfXHx3tgMYKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1MSA+PgpzdHJlYW0KeJwtUUlyA0EIu88r9IRmp99jlyuH5P/XCMoHBg2LQHRa4qCMnyAsV7zlkatow98zMYLfBYd+K9dtWORAVCBJY1A1oXbxevQe2HGYCcyT1rAMZqwP/Iwp3OjF4TEZZ7fXZdQQ7F2vPZlByaxcxCUTF0zVYSNnDj+ZMi60cz03IOdGWJdhkG5WGjMSjjSFSCGFqpukzgRBEoyuRo02chT7pS+PdIZVjagx7HMtbV/PTThr0OxYrPLklB5dcS4nFy+sHPT1NgMXUWms8kBIwP1uD/VzspPfeEvnzhbT43vNyfLCVGDFm9duQDbV4t+8iOP7jK/n5/n8A19gW4gKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxNSA+PgpzdHJlYW0KeJw1UTkOAyEM7PcV/kAkjC94T6Iozf6/zYzRVh7BXIa0lCGZ8lKTqCHlUz56mS6cutzXzGo055a0LXOAuLa8L62SwIlmiIPBaZi4AZo8AUPX0ahRQxce0NSlUyiw3AQ+irduD91jtYGXtiHniSBiKBksQc2pRRMWbc8npDW/Xosb3pft3chTpcaWGIEGAVY4HNfo1/CVPU8m0XQVMtSrNcsYCRNFIjz5jqbVE+taNNIyEtTGEaxqA7w7/TBOAAATccsCZJ9KlLPkxG+x9LMGV/r+AZ9HVJYKZW5kc3RyZWFtCmVuZG9iagoxNiAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNyAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggNTYgL2VpZ2h0IC9uaW5lIDcxCi9HIDk3IC9hIDEwMSAvZSAxMDUgL2kgMTEwIC9uIC9vIDExNCAvciAxMTYgL3QgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDE1IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDE0IDAgUiA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxNCAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNyAwIG9iago8PCAvRyAxOCAwIFIgL2EgMTkgMCBSIC9lIDIwIDAgUiAvZWlnaHQgMjEgMCBSIC9maXZlIDIyIDAgUiAvZm91ciAyMyAwIFIKL2kgMjQgMCBSIC9uIDI1IDAgUiAvbmluZSAyNiAwIFIgL28gMjcgMCBSIC9vbmUgMjggMCBSIC9yIDI5IDAgUgovc2l4IDMwIDAgUiAvc3BhY2UgMzEgMCBSIC90IDMyIDAgUiAvdGhyZWUgMzMgMCBSIC90d28gMzQgMCBSIC96ZXJvIDM1IDAgUgo+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTYgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjU1ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDQ0NyAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgNTMgL0xlbmd0aCAzNiAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NDcgPj4Kc3RyZWFtCnic7ZwJNNRf38D/v9mNfY1E9iQi0qqFiFJRWbJki0qU7CEZS2SXfQvZkr0QkiIpRQhRiShLREi2jOkdzIwZmZl6n/M+z3nP0/cc5/h9772/+733c+/3rr/5B/VX/j/LP/9pA/7KvyT//KcN+Cv/kvyD/Tsip2rmU1pqZ6rjE5mWktrwyM9AQXLrAbdANcVd8Sk7dsprG1la2SYWteWGaLu3xaFQaW1/IgUoVHDznyR47oXyePInCVrDUaicP7IpC4WKaP2TBFXuKK8X81mRM2F5QHMgClX4RzaloFDxlGO0LuU1//cAx2+XkIy2V329hbF+UHL5neyhtuATshwiyoGJh3dKZOWv4uJXO2l+6lRa5cCDWG3fr8ko1L2ffyK1KFTk7J8kGLqK8uj5kwSYGyjU4z+yqQKFSsT8SYJPHijv4T9J8APbpl7+kU2FKNTt34uJs/wNjt85fXkVraRUtaSRrLyhNLvcnPDEJPuE4a/DXZWxhXFIGuYbhWGqXKanj/jkvit1dP3L73fk/5IfTvD8zp+xcfUNcJULaM/Oa4u1Ckl++Pz5jeLhurJb0d6ullKS0jbeUeYqZ09rJtY1pmjYUOY32NTUQaqhwg89PPJtjkRDjd/c8pr/L+dnaV3e2GWrLGiWnpWT53fBPBKbe11di6/JNkVtkXXnz58RkomMi3OwcaibznWWOE2ZX01AQDppzVDhN9H4+v0PEg01fj9+LFP8d/D7xV48Pz9PpfNpPuYsEqr2ofUPs7S0HJ/1mfgNWGzTsLy2U+yQuXegn3NYWlxWy8D3/iofD0r8Jqq8lZWtSWlR5tf79KKd2xcSFWV+k4PR/lGvSUtGmd9cRXhyJamKCr+PsbdLpkk0VPhNZ2SUD5JoqPGrTHvQRqqhwu/9zft1aBINnl+4H6+Cs689k+BOC98n97MOyh/NbT54eVhP6Lht1EFZxQvRJTk212Li8z+8fTvenhk4z29maupXJDOjPc3hBiKSBt9Iwsjzw3z/0v0ofu9BvW4Swyjwmx7vaa020TxfRfoe8vzmZgZ6OsJOOqeTqinwmxvr/1hu5hbznURLiR82izcOlxO6SJSU+GGGP3+KtIuvJtVS4jc+1FNsFVVI6njw/Hyv6m9BGlmK25eXxmwV4JA4WZEQ8rj2lQo9cou1ne2Z62XFmT662+R3y8juK7gT5TPPr7n62adfMmkO3cvHiIQftHvaR6zG8Zv9leG3PJtNvJzMeyxfkCQgzw9Tny4vyc8ierFyhkRNnt/Qa1N5ER4ei2RSNXl+s0NRxzatW6PrN06ipsSv2VheQkDR9y2JkhK/CVddWTFJzxJSCyjxu223W3qd7U2SUhP4BVw7Jg4ydVQJaq6/a6S238Y7JTamrDRHXcrI1sfPQ9un2NP29Am18/oiMnudHU45zvPrarz/fnnB228eXcfFywyRUPEMTSxtJATg+E1OLjeq59lpBV4RfhqhvR4RKWUNhMZFlt/YB18jkfUS9BwKTknpRc2jeD1Zfj/67l6R2yjNy7LnVHJG/rslD0eW30hriN7mbRKsskfjb+e2DBDikOU3/TXbS27nVk5RtYDMu88HCYUgzw/9LP6o4l5BblXbzLsPh6cIevL8PqVbqe+X4dhncutOyeclv47nFxa4mx9s6WOZ0fOxOS0uurnsQkRmerSbgWlnXUZq8G7H3F0isjqOBdfX7T0kvX6tyTy/yb4ckiEIg8F8yzAE2AXlhUAAUmztau1gQtFx/EZHf5IIZu5hAJKWY89OEADiFxE+6T+GDyHHD9NxV4KGWfowEwjEISyuFvGOEECGH+Zb5SkaVmGN9QgwnYCkQmodIYQcv7m2ZBZWrmNK9DCosNTu6BqCXyfHDzPUsp2ba582CxwksHm/a903fAA5fhj05CXBtZuNhGAAxxYV06alNkWOHwZTJiYsZqwAg9LJKmrVfCUE4PlFp1eEqKupawY9fZB5CeVXnKBpfcXa5GBi7ffcIwq7d5oFnjhqpn0sJz/8yvkjDk8i5vnNzQx+/kxwiBOBHl4n5CV4aeVV2GlBAJiOBr5qbwK+elfynxWJoTaKMiJg4cMCLCAAoKGlXS2Z3YwLXIlff0V6yJG90vTMKpLsMDAAR9Ktkoh/j6velfjNdj+7fU5ddg1s+/bVDDAQBEnPutllFG/DSvwwYw+CjPaIQ0V2r2VHQkF0jGyy+gQnuiI/zJvic+o7mLnkxVbTI0B0zKwbjhDczsr8xrp8zJUFGbbJrGGhg9CwsvNppRHCyPB74HN0K+NGOcHVjHAYMxvncU9Cm8LzC4x/d9tUYYeCTVJCsJmte6qfvO4ZIz2Noq7+1EPi63bqONhYWRjq3sjICHQxDfmYgJt/fuvF+oq52bHhvq5G7QMHOQEEkk7xMDczDQQMIJnBIl74ovwyf5ka7Ys8r7sFgEIB0eOiqxlooGAYE4Ih6iku/Fd+mNGX0fY69Ag4iPnIFh42RgQETEcL827EvXYFfrNjVUkO4qxQMERujwA3BxIORtLT6g/gB5CV+H1tC9LmR4CAdQoifGuYkVAEA8umIfzKdCV+0yOFfuJsIDC7opQQ/2oGJIKGaQ3BjJX4YabePVAXh0Ng27YICwuyMtDQ0Av5EkJX4jc31ROixgSDisltEFvPxUyDoBHWJLhoPL+Dcud09u6QYmBiU9bQOnXBURMiKn8ubiAryPl2r+0hBTm5JyUSfnXa+5TOom56HbTCrx/myz/WfSvg1J512MYEgomvASA8AcYSqxjAu88yyeR347JZzm/2+U3jbXwICMDJDQIj3IMPSHHT8+kLM2a34iL8ym8ixWEXNy1AxwkDQQ1zLdTWcTAe2AZce02+//XUmO5khkJZkQBEpvy65XYRelklxPEu/Bi8Aj/MTSteRggtEgAzZea5HNvCyi/Pzvce3wFX4teSrCwMgUAAENzjeYidqgiXLD+4BF+7K/H7XoPawwzD+hvIgabMIG251Rs20DgRQlfiN16juYEeAgZD17ysSbLR4hOSYNxCmMTg+SlucXC57Oxoqa8qt3O/4j5lGUBMySi8P8PrmHultaHned3WZ1vNQ4I8zpjbVmeddyZe/33vywrQVdwsKCAsIO5whJFnV9mtADdnW9/wVTymT3BxlvP7UR5/TklGSlzU9ByP4KbcJ/GB7k6XvbfQmCbhIvzC78fXW6jD8tsljjrLCsv4teUnXXO7dOkYoH5tAlf1v/LrfWGjpyi7x0VdZLPJh6f5YT5OtmbITV7tuOCV+GV5HVXYYnlaVFblZVvpzeuuNqZrWT0f4EJX4td+1+bEDp2LWzcrZfVVF9245qQnC7LKwLWplfhNv0+zPXjE+oCU0pWhliepEe4GSjSqifgRcCV+0++umR65YLBJXq+3t74w1cv8ACv/jVfL+MlvCch8UdvYUxGzWUBmLRMzHWjToeNX3yZfFlXzM7drzvbqb1aW29vWGnXBdLgjxZ+Y3/TIvXBdbY09SppHTjYGCu6zGF5oHT1P1gKgBFyc5fxmCuLdDI+fM9OtqturZrG4avpSpAIgdHERfuE33lcQZnHJ+UzGuI2GbfniS3KtAJp1Q7iq/5Vf/6tA/0B7P/RtLceERVDlvgxwllJc8Er+82F63DWn3jf6jmEjC8+t/hvA7JdwgSvx66/LT/J4OuVqF/Nm4XnURxXg2IebUK7oP6fflkdmz6ZeiK1YVMSeQrKKNOBCVxz/vjcVpH5rPB+SszjtLDfnohGLxoXh+Ykzb7Mt622/5OYtsxXleOHAtrUuGeZa8vuVd7u1+Lo9uh1al8N/8GJVkoRn/sWAuihifpjZr71tra8bm163tH3/9PxV++xCrTSnrwZBE3FxlvPDDPd3vXnd/rZtbPxVS/uiXT1xuwHak7gIv/BDzwz3tnd+eDeI7njdsVi503FGAO168vxmvn/69Knj08/B1s7Pi5o7TnRI7jJc8Er8Rgb7P3bOTLZ19i52oYZLwlBBd1zgSvx+jA997h6b+9DRt+iWR5z3AYLHcPP7FecvmMmRvi+Ygff9uLl4sCaMW64FF7giv7nvQwOz4+09XxatvafDTi+fggvD89vIvOHMnYH3BvaB8vuDouIczA7GVjhpi+46eDKmLzU8K9LjwU1BVauiCPHAMpWLDdFUzx9mPt69yg5GJuLGfurnD5i+aicpEIsJ7vE3zh++vnNVBZilcTX6G/ufk93hhkhG0Ue4R+r7n3M9dw14EDL+uMff2P/80mSyGdhkTIkfqXzvdtwHET6GX/dT37+e7Y5TYmY5kYd7xPPTPHDBI6ap6drt565n9VPbBwc+3E9LuREem1JT+3K0y0JJND7FUG599J2HMVeOmCf7U+X3znDnWiiCO2FssYao85txUIKBoRvscY/U+WEStJkg/4hr45rxb/B7eVIYCvBp4muUOr8xCzkIiPEUvkZ/g1+YBhIM6Ifiivob/Cr014ABhdB+3CNVfpjek5sgYP5E/DoLz8/ayic4sqQssbRB70RSalrirbLKisTY6KvuSXGhTa0hhptjbvmiLlleyY4JvOLh70mN39SLHfyMTKKm93CTb6r8ZkePCgM0nJdycc/U+aE9NsCgjJYZuLb+G/weSbKAGVVu9eIeqfLDDMrzAHQiGfi6os4PY7MegmALfobzOtT5YXI20kM4ztTgp7jU+XXKcgLsu+sGcM94fsGRiTGhscklNU2y+7+kmCufCKx+HxsWcfaE1aljmdVl7geisu6+aNq7PzgqoSDBnsr5LWZ2oIgTCeY5VNjU/3v80OPd4nAQ28aXuNGM+vnf7IQpANDy3SHkSYUfBj2bCwJB+ewIGmr80FMfOAFgtTxhO4Uav7nZ6aMAwLLhGV5BjR8GPR0FAAjJEIKGGr+5mRYGACRu+Mv6r7xaT3GTqWdlXqx/QMydMm29ABfn2gjR3aqnDE+o6wamlV6zN4qrKQyQ3XGgoKomkiI/dLHlnvVwge17LnSP4jKiwq/3qo40klFqt2YvPhIVfrM37LavAvPJHCMcQlDj99BXURTELqIVTdBQ4dcVrisNQ3KrORP2bKnwm0qy38YCZt1+kbCJTY3fw9C9ggByrX0xQUOF39s4DUkIgtUy4Zf9l/JqYyXx4xY3QjwToy/cen5cN8rCoD6S77Ch5in7/YquGQ3XL2l459bGCwpI5TyqjqDAb7B3JGorHQgspX0qaAzfTijxm/wy0XJMGAA41M6gCPujlPihxyYnrPeAAegeXXe8b6PMb3oak3icHgA2HvEqIigp8ZtDYxoNhQCAS8klmbBWpsJv3HoPtjPtPhOPd9BU+cXp04BAkupJ9QQNFX7Vp3kBEJ9ydAXBbjw//8hHaW6HZeDbzgVfFrFOllKrdDO6GelY/EDCrtPdzCog81mZx2mt1CA5CclIf2MH8vwwsVerLbFmwY4E1CydhlHi9yrhZR4LAgwSi25ZikKJ31jJq4+bEABAF141SygIJX6Yd+1T56EgAHCsmFy6pkGJ3+TXqTsQbIKDL4eXElDkh/k5tBYCAXE3flk6x6TGTwcKBoPzxtFLVqzMjxB+Ew4BQW1+oJdswvO7W9vdUhl2kUsjyP6CRWDyBdvrAQGpYVbBAetN72vJaTpGB/t7Xo+1vOh/SCo06VYAGX7oT1Ux2spntkEAALz1bBvhNIE8v4medDfti4YICAgkeLl2SU+eH/pRorm962oIAKLzIDKCAr/2IncXPzksDZBVEdGJOnl+ow9u+ASaAQAAUa4cWlKT5zf3vCDs+lVmEAjCUzuwpKbAr73iRkSINBgMo71DfMhIvv8NV92KCtEFgcCMTsRG4/m9wxrx403eJut8Q7OCrNTbEadiijNDzU/riWhGSLDJmXjr6HmWd2491npuc3AJ2fuDU9U+sqvoGbB9AwAE9hOd7pLl9/nZYUk2DhZsfNAaY6ILDmT5oSevHOFZww3DVi6DDVFZyfNDF54T4+FnxOYAO3t76aiNPL+5Dof9/MLcIACE2F/0eUlPlh9m6vppMRE+GAiA8z8hsposPwy6yG2nhDAjBIxkz/9KFECW31ybl9omQQ4AgHKjiC8q4PnVDs1VX35Zc8DMO+X+YGXC2YsR2WX1ja/ctSVsX6E0GSXULgelVTXuOXzbw6a0vidxid/c1NLrJj6FadPBafmxOEDQ49FEh31k+RV5stPRcHHCIBBWxQqiE/iV+WHmfr5OlGGGs/Nj28gqmRdE103I8vuabsCFpONjRcLYNmd+JQJG9vzvgT8vM5KXmxbJIIcaI7KaLL/m27Krmbj5GegZdph8IbrfQJbf55wTvKycQquY2KT0G4hxEPgtM2ymzJOHk4N/DSvrWpNM4gA8v7wXk/ctX9apnPIqrB0r9j1rG5V9LzOvxMtM6szdy9owfgX/xFv5dwwMPX1C6mqKQgj8vvR0LGKa629/UZ5zdicIjOBlAkBguHEukWNY4jc1OoLTj3x6+bza4wR21c7GDoPBOY+0EQFf4jc3NTGxWJqpgeYXT2+c4wIARm5GJIxX7gNRXRHx+/Hjx2IC9GBHfU3RRaw7h3My0yFX771PXPQlfhjCeDLW/6q2+qo+HAJZxcHAyH4giDjBSvymhtoansXbccIRbNyMzBwqNhNEgSvxQ3/tanp+x1EagmDk4WDj3X2B5AbTCvwwY59f11V6aYERdKu5VvFI2BcRJ8DzMzbvqo57/UTyTEpBRedVlcR429ibnHRMlyI2C6/eIEa3bn9Kfril5uObot6PB1P5T+H5zaWHBC5aOBFuzMVIAwbNe0JsL4fTnC4iuv2zxO99efGLRV2x9zo26HwCrFMHI5lWq71dkd/E25aWRU5v4/evhUMWs6BhWCUi957IGy7xQw/29y8GfEs/v4ERBgXPJ4DTsgmQ4zeDXyJgKqO2rEVis8CaBGfgEjy4Mj/MUu2+zVGXoMXihkKhMFp24cO21PiNFNrvYqfFzo1AYBjjKmnV863EXQ3PD0OY02Bmq2KVhbA2YQsBRrJJ7nMkGbnw/FQONCcePWOwWis84nZvpGFMsNGtShXZDU7Bigr7RXl5hTZomFhpKBclKSS8GSrRssVOeCbmhmP9XY8rK5zxb2+KiotQEUcu1NR8ZYGwBZI5GV4yjW/Yi/wwRbGulgZaxm6P+5LjEk12MyMWY2MFimCScIwuJKy1cPyaMlB2Z83MLiUOFsWnoA6spQctZgCGI+k5RVDx5YThCcdvqDDE4aLVeQfv7vqE1Fg1aVYosJAEm4CWfYNVbPXSxeJFfrOP0xzt7R0vuTzvTklON1PiYoDMvx9bABp6VimdG9XvCZ0Tx6+1+IqD4yUn59uDhckZV9SF2OEQYD4+DEHHskn1RlknoVXh+A08DHJwdHRyCeyrSbodo7mFmwY7WwNAEAQ9E5eEanBB/5KjWuQ38zTV3sHR2dml4V1qRuYpFUEW6DxvKA0tA6fQUdc7vUvXiPD8Nm+scmFBwmkOXnMN/5R0MdRTo7DTVv+AyzWtk+dEODYJsTOvVVNSTI3RKRzsr/JyR6Gy+r63yQsz0CHBzMJFqTvl98LnqwmKnd9iawzbgAE44x4nwiVCLL+IydlJW3k6WjoIjMmnRk3hINdCvULAUPhCj0IwCahaE8byeX7d6NkMPWYIAgZFHGiw3ndUZiEetuxQCBgGg9CySB4iXf9VYtBvLkqCEHAwjLvihvIxVdhCFmBsc8LyA9NwScgHlZPyw0z4aQIQKLYyIyrU1bQ5gPm+BwJgNBBsGginmFLwfcIIuMhvLt+Rcd5umG7DBTUdyfl+BMHyhsNg2MJzih5xLScUYpEf5rWn+LzdiPU1oap6hyALNmELgW1RtGCWDeoXXy6N+4v8vl3XwMaBwWiS7h7TNWIBwbFvBoEQtIz0NBCajQctapbGfcL5A6ee8joBAbHzubf9pUWEAiI8n3RY2MV4+9Z5cPNIXT4mzCl0PLZG55Deo0HNPbLmKJSz9UVjPmZss0PSMvDz0NLRgRDscPqTKgJGxpD5Jo9tLqsNCE0Xy+9qYrj3HkHwQsvjFGSgo4fOF2SLPIfcmVXz/8HgNALyhO6E5ed+pyTpjAzNwsvohdnpmZDz0VZv5eAxlJyvZSickVcOfxi0wO9O58Po3auxHRrbkIR4GRgZ5hPQiK2CqSpB5iki6FgVA4j5Jcx2lKuJL3ZpbkFGJtZ5m0Ccq4BNxxmxCQA4PbuCBWEYWOA3/tpJDrFgE7MIOyMzzUIWjBCu48JwMLYp0q7Zqo8/DFrkN9efp8K54GYQQtwMzAs2QWihcFU5JBKJhciz0WhpRrLAb6ZHV3LRi/HwMTKzLlQnDArarMyMRECh9KvX6/n8sv8iL7Zrz979ykrOuSXXmQW3XHIxL2k7aZN7ybotSHjDvqTT6+UU1eLrj6m5lNQrbdtpMc9P5/D2XQrbt27bvHGh6wEcu7fuDXXVCbsuKbx2HR8WE7sewS4sP+/UQGs1JfkdMhvWizAuJIDyr5Mwtz9wIUpelE9YlBYC5tyC34df4HfvfojlIfmdkmIiAqv+Ac3XJwefqPI5FbVAXTFBoQ2sYBiLGGEwn+dX0F0UobFfbuN6AT5uKLZngwFaTgEp/QNbL13cICIkthYKQWx0JeGHfldipLpDUpSPlwcJwrpAEISNW1BZRdrgsux6ETFRBIxG7Djp/aXxVq+jO6TX8fNwMYGhMOywRM/KI7VXWsn1kPg6MSk2OIPAHsIVtwV+mIFiHXkZobVcnGzYNo1NAKfnFNy5WdbBdONGyU38cCZu+RhSfj/6zqhKC/GuWsUJA8FgMOyMkHWNzGZJYztZyY2bxOkY2XefJ/gEPD8XS0EVt6RklG9Eza01Z2IVRcB+NXJGJfvFcuIctM92XBVPSthr4W/h05LoZuvg6oZCRZec1Yt8gX5bW5oTz7mAY1dWed33kZ6Rvswg20gXMADiMSbmFzZY7HKjdLbraUZC6PaFQYnJLSr3/eeWjv68aOfguNUAwC5N8CTz/vPTYFxKXv+nkpvXvU5gZ0QwCHDcOe5BV0N9e2nsVd+bCgCA5CHdv56rKipo+Xw3wcvZnBaCQCBAYic94ltqH9c/SQj2SbTBMl1jTcwvEdNTX/Sw4168q4OVAIienhbMePisV0XtoydPEyODE+PYwSAOOdL969mRp/mvHoZdtjLeDmZipYWBN6lcSHpWfP9FVmRwTK4CPZh5Hf4OFn78+5TzuDrA4aTeYTiSmZEW4JI+eaWipOj5/cjo+DxbFgRSfGmWtOg/J+8WVgTYaGlqssPY2ZihNLzK5wru33nemBCXmHuDl36l/WvTY5tMb+Vnh2Q8CnfVOu/mYaue1ph+tz4h7FWamYHdu0h1S9fYrOKMm9E2+oYauy1RqNjy5NSmgZ+jA92vHijyAzuNHePaPg7Oznyf+d7eUNVcpMyH8Coh5hc+9uFp68ef4/3v6u8ZCkPX6pk7ldZ2jE0Oj37raH7a0GSyDTBIJviqhfnLZMubjonx7ta6LNd1jPQnTC0Tils+jX8ZHOluqa1r8z1CKx1LuEG8wA/T2/VhaLzz9YvMQAlOuMqps55p1a+HB3sHe1sb6lrv6AqwXF9aIy7w+z7Y9Wmks6m6IEKRByFhYnoh9u6L3sGevv7WpobWxgvyMKtM0v1PzEx/5+DHxurSAAN+Jk6DU2a+NyrbPnd/HGhvamjpCDvFoHST0Ahx/L6/7+uvf1Ic7ijMQXfYyMQxoKimp7urv7u5+XXHPUdR/oTl+5/ozs6PLx8XJvlsXUO/1dDI3D310YePnZ+/tL5u7WjyOsRw9T5hYMLzOy6v4PSwICu64qON8ZWLFtmZQSXNX/vb3nd9zjGycG9ItlYyamtpfHX3nOZB9R0sBijUjapXHYtz3KFW8z0wi+zmTzNEn4B98tjF9Gppej8/f5nCB483o5TodySX1fYtNO25hdXabMJZeNQkYS5NWD/Mzf2cm34Rr7KGI/ZO5du+hQiLH489DubS/EHIkrB+QM+/7lW2qjjSJbv0eceCDYsT/o9Je/mJVvz49QMGPTuL/lhosgVxMOPO/Tai72imi+yZiFZbhPUDGo1Bf7vjKsfGfzOnsL4TbwFmfiW/xmYpAfH6ATP7OE5eiM49Lf9x8/SSTYPP9u8gWnIQ1g+zGAz6Xb6WNJ12YmZRA9Em3kyL+9qlO8hL329aZqL275C3wTq4eyoXfDTO3Yn3O35MLyg1uaJ/oFY1qElbXktFwcPX/X696v4Tdtj1w+Q0rhPPTWRcOxn5cHyWeCEzOxR6pIRwOXqBH6HPz00V++k6VfVNEe3D/vw5Wn0i5zNBQbL/gpls8jXWrXw/OUO8hzPZZxP6g8z+9URPoNXh7DfjUz+IjJodi3P6tvRIvP+Cnhq96ajq8354fIZoOwTz/dkloivmxOt39MxoJUrH7OPw92niD0pm+r0Llp5I1n9zE91+VjqN38YnZ4hsQk9l3yQqFPH6HTPzLclNP2P02/dpYptmXiURNUI8Py+/JGsZdZOrmQ1fX5m4R9pdexzuoHPSwicy/dGHjiqd4Oqju7SUd512TanvdLZxukKy/zlbmuia37x8e6zMh9z5A2buadKV2DffliXovv5siAw/zIckn6utX5YlmEkrQZPhh/6Wdt316efl22NVmWT2P+fQsyVRqJzhH8tSdGf1Lj0Q85tDo1sTA6LHl5d6smjpow9SfpifY1nx1z//XC6NNUStmHj/E/MTU5ERVvtLgs91RBsjhPO/Kk2lndUDNdfUO0crCvPHJz74mxVW1ccFNxRcc/cri7Hbrhjh7aBo/bXl+Ux3dhAJP/Tz0ozOX7IhEVJ+s833SxopRSflh/U0A2XPX6MpJli2/znztLZ9gnzseSHd/8S8ffl5nELsn7/sn31pGJ0jH3lelu2//Oj6Sj7uoizbv/76dXkLXC6E86OqbGe5M1fSXY5ZnDcIKSq5/yDEKSi50tM6200+IKelPNnOKTc1xiOqublt4pGO3RK/kZJoV13TwNpvFHMi4jfXXBpgZXq7jlJ0Un6DrYnXzt199/v8pvpK4q5kjFG+MEXMDzP5Ji0ygVoCEn4ThRmJfZRi/1zOr64k/zWFyAtCwq+/prrxd/ml3X8fuWPDDl9rDUGuNda53tfiQ65dCi5xMIk4xRPxuP1VddLNitKCvIJH9W9HMtaZLvHrDdSVZJFwKe+j2BaJ+KFLr6tu3nTzV79AIsT8OivOHN11q4tyYyfmN94WZHoiinIGpPxGK61tvKcoxyflN3rNw7uLUmyszJDwywuNrKFmEwm/Nxm5Fb/Lz1Rjpyirk7+6hr6bjeQmaY61uy5eLyotzcspLqtIDj6T+Lapsf5p5fOKW75XKvPcUUv8Jtsepp/QRd16MkMpG2L/OTnakxda95GyXcT8Zqe+vK/sp9I5iPlhfox+GRijGP3nsvFvamh4hFpdkfCbGxkZofaLGqT9b2JsbJpC5AUh4fdjfIJakyLwc7LRO6GdmW+goxMRbraDW2zbMc+k7MyUsJD0tJs3gjS8ilteVz59fi832edyViKKiN/sl476sOu3Hv0yfyER0vO/udcPu6hcpFx2/jf58ZcvP5fJf8fvF/wieH6POzqHf84NhFuppz9uN2NDRaXcr3B3sNotY6fOG5B4TMu4uj6xstXdLTEu2sdR3/7v77/8jvwb+ZXU5sdfjUxvcZPQMvfczZ6RjYrPrShOMzxaGKRx2UnGqepdpbGzv/mlrJKKp8nGjn/5/Y78G/nde+xnrnjCZSJYVGqnlgTXg3sWvsmf2isdzDtKUc7mYv4jrVnb1Ey0rAufNXcVWzj/5fc78m/k1/peQjfZ0KispiH0euk52bCwC+6hjwtCQpPGxgae3g2yUpcVoTsZLn7kWV6ckeOy9d9vyF9+vyf/a34PHu09nWRxNrnoSURYnuNRX187lP/tRO+g2JeNDfcyQm2O7tm28Vyoot6dhEBDu1R/FCr19Z/IXRQqqOlPEtR4oTyq/iRBSxgKlf1HNs3//mDLnyR47I7yev4nCZoCUaiCP7IpBYWK+6MEZai/v//5/13++U8b8Ff+JfnnP23AX/mX5H8AScl9/AplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjkwMzYKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjM3IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTA5MTYxNDQwNDYrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My40LjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My40LjMpID4+CmVuZG9iagp4cmVmCjAgMzgKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMTgzNzYgMDAwMDAgbiAKMDAwMDAwODA4NSAwMDAwMCBuIAowMDAwMDA4MTE3IDAwMDAwIG4gCjAwMDAwMDgyMTYgMDAwMDAgbiAKMDAwMDAwODIzNyAwMDAwMCBuIAowMDAwMDA4MjU4IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDQwMCAwMDAwMCBuIAowMDAwMDAxMDIyIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMTAwMiAwMDAwMCBuIAowMDAwMDA4MjkwIDAwMDAwIG4gCjAwMDAwMDY4MDAgMDAwMDAgbiAKMDAwMDAwNjYwMCAwMDAwMCBuIAowMDAwMDA2MTg4IDAwMDAwIG4gCjAwMDAwMDc4NTMgMDAwMDAgbiAKMDAwMDAwMTA0MiAwMDAwMCBuIAowMDAwMDAxMzYyIDAwMDAwIG4gCjAwMDAwMDE3NDIgMDAwMDAgbiAKMDAwMDAwMjA2NCAwMDAwMCBuIAowMDAwMDAyNTMyIDAwMDAwIG4gCjAwMDAwMDI4NTQgMDAwMDAgbiAKMDAwMDAwMzAyMCAwMDAwMCBuIAowMDAwMDAzMTY0IDAwMDAwIG4gCjAwMDAwMDM0MDAgMDAwMDAgbiAKMDAwMDAwMzc5NSAwMDAwMCBuIAowMDAwMDA0MDg2IDAwMDAwIG4gCjAwMDAwMDQyNDEgMDAwMDAgbiAKMDAwMDAwNDQ3NCAwMDAwMCBuIAowMDAwMDA0ODY3IDAwMDAwIG4gCjAwMDAwMDQ5NTcgMDAwMDAgbiAKMDAwMDAwNTE2MyAwMDAwMCBuIAowMDAwMDA1NTc2IDAwMDAwIG4gCjAwMDAwMDU5MDAgMDAwMDAgbiAKMDAwMDAxODM1NSAwMDAwMCBuIAowMDAwMDE4NDM2IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMzcgMCBSIC9Sb290IDEgMCBSIC9TaXplIDM4ID4+CnN0YXJ0eHJlZgoxODU5MwolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-09-16T14:40:46.488312\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.4.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torchvision/utils.py:50: UserWarning: range will be deprecated, please use value_range instead.\n", " warnings.warn(warning)\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDQ2NC4zIDk3LjI2MTM5NzA1ODggXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnichVXLbtswELzzK/bYHkpzH+SSxxhpjebm1kAPRQ6Fq6QOYgdOgub3u7JlhLJV6WCBGpMzs+RoifDgZlcI9y8Q4MF+b4CwgNl183ezbr4t5rB+ccHwrZMknm302I2KekrIRQ0Jvbc/zu2ccdrkhdHdOwxegdVHYY0tVVSP78BjB5Tgw4njsKQGjPTO7aFPJZK8QCQfUZNoiDnDcwM/YAezK2qrQqsKrarQq8pZVXtbrdDWFvmMdr2F2VeE6ydYuiXsT1zBSmr5gs8doyGOk0fOxLGusALFh65KN7ddeXN7ewb4FIyLyZeYo6gWQaD2zcTdfAWzL2imYHV32P3Vb/cTPuBHuIXVjfu8ckt3MOFy9KwcqdTiFTgmrsUHFIwiiHlSnOlSHY0/hZxJavkaHdNHyp6kiGSzKpMGkgwYyOzz0X9toEJHDah6yaJClp3p7S/p0gAx+9AeHtUGanTMABHbmsBSWgfT5095wEEmz2jZTz0HFTrqQMmrxiTM7XlNOkjh0gEz+thPf4eMR58sfqG0/0/KloHwcbZv9hidWrpCR+UVLXyJGJO1jkkHRAPpE7b+RIemUze3Ch1zIBQsfRgp2rfK0w5iL351jC3ulBSzCRVPmpHGiBbNrnn+9bp52sHmtRvVzAQ3x7vg0OX6N8F5I7/o0u77eXPfDjR3mzV5JZzmvK/7D9PS/QMFFmA7CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKNTI3CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ3ID4+CnN0cmVhbQp4nE1RSW7EMAy7+xX8wACWrMV5T4pBD+3/ryUdFO3BECNLXOLuxEQWXrZQ10KH48NGXgmbge+D1pz4GrHiP9pGpJU/VFsgEzFRJHRRNxr3SDe8CtF+pIJXqvdY8xF3K81bOnaxv/fBtOaRKqtCPOTYHNlIWtdE0fE9tN5zQ3TKIIE+NyEHRGmOXoWkv/bDdW00u7U2syeqg0emhPJJsxqa0ylmyGyox20qVjIKN6qMivtURloP8jbOMoCT44QyWk92rCai/NQnl5AXE3HCLjs7FmITCxuHtB+VPrH8fOvN+JtpraWQcUEiNMWl32e8x+d4/wCVT1wmCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTUgPj4Kc3RyZWFtCnicPVJLbsVACNvnFFyg0vCbz3lSVd28+29rQ1KpKryJMcYwfcqQueVLXRJxhcm3Xq5bPKZ8LltamXmIu4uNJT623JfuIbZddC6xOB1H8gsynSpEqM2q0aH4QpaFB5BO8KELwn05/uMvgMHXsA244T0yQbAk5ilCxm5RGZoSQRFh55EVqKRQn1nC31Hu6/cyBWpvjKULYxz0CbQFQm1IxALqQABE7JRUrZCOZyQTvxXdZ2IcYOfRsgGuGVRElnvsx4ipzqiMvETEPk9N+iiWTC1Wxm5TGV/8lIzUfHQFKqk08pTy0FWz0AtYiXkS9jn8SPjn1mwhhjpu1vKJ5R8zxTISzmBLOWChl+NH4NtZdRGuHbm4znSBH5XWcEy0637I9U/+dNtazXW8cgiiQOVNQfC7Dq5GscTEMj6djSl6oiywGpq8RjPBYRAR1vfDyAMa/XK8EDSnayK0WCKbtWJEjYpscz29BNZM78U51sMTwmzvndahsjMzKiGC2rqGautAdrO+83C2nz8z6KJtCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicTVFJigMwDLvnFfpAIV6TvKdDmUPn/9fKDoU5BAmvkpOWmFgLDzGEHyw9+JEhczf9G36i2btZepLJ2f+Y5yJTUfhSqC5iQl2IG8+hEfA9oWsSWbG98Tkso5lzvgcfhbgEM6EBY31JMrmo5pUhE04MdRwOWqTCuGtiw+Ja0TyN3G77RmZlJoQNj2RC3BiAiCDrArIYLJQ2NhMyWc4D7Q3JDVpg16kbUYuCK5TWCXSiVsSqzOCz5tZ2N0Mt8uCoffH6aFaXYIXRS/VYeF+FPpipmXbukkJ64U07IsweCqQyOy0rtXvE6m6B+j/LUvD9yff4Ha8PzfxcnAplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTQgPj4Kc3RyZWFtCnicRY3BEcAgCAT/VEEJCgraTyaTh/b/jRAyfGDnDu6EBQu2eUYfBZUmXhVYB0pj3FCPQL3hci3J3AUPcCd/2tBUnJbTd2mRSVUp3KQSef8OZyaQqHnRY533C2P7IzwKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIyID4+CnN0cmVhbQp4nDVRu23FMAzsNQUXMCB+Jc3jIEiRt3+bO9qpSNO8H1VeMqVcLnXJKllh8qVDdYqmfJ5mpvwO9ZDjmB7ZIbpT1pZ7GBaWiXlKHbGaLPdwCza+AJoScwvx9wjwK4BRwESgbvH3D7pZEkAaFPwU6JqrllhiAg2Lha3ZFeJW3SlYuKv4diS5BwlyMVnoUw5Fiim3wHwZLNmRWpzrclkK/259AhphhTjss4tE4HnAA0wk/mSAbM8+W+zq6kU2doY46dCAi4CbzSQBQVM4qz64Yftqu+bnmSgnODnWr6Ixvg1O5ktS3le5x8+gQd74Mzxnd45QDppQCPTdAiCH3cBGhD61z8AuA7ZJu3djSvmcZCm+BDYK9qhTHcrwYuzMVm/Y/MfoymZRbJCV9dHpDsrcoBNiHm9koVuytvs3D7N9/wFfGXtkCmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTggPj4Kc3RyZWFtCnicPVC5jQQxDMtdhRpYwHrtqWcWi0um//RI+fYi0RZFUio1mZIpL3WUJVlT3jp8lsQOeYblbmQ2JSpFL5OwJffQCvF9ieYU993VlrNDNJdoOX4LMyqqGx3TSzaacCoTuqDcwzP6DW10A1aHHrFbINCkYNe2IHLHDxgMwZkTiyIMSk0G/65yj59eixs+w/FDFJGSDuY1/1j98nMNr1OPJ5Fub77iXpypDgMRHJKavCNdWLEuEhFpNUFNz8BaLYC7t17+G7QjugxA9onEcZpSjqG/a3Clzy/lJ1PYCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MyA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIwID4+CnN0cmVhbQp4nDVSS24FMQjbzym4QKXwT87zqqqLvvtvaxO9FUwwYOMpL1nSS77UJdulw+RbH/clsULej+2azFLF9xazFM8tr0fPEbctCgRREz1YmS8VItTP9Og6qHBKn4FXCLcUG7yDSQCDavgHHqUzIFDnQMa7YjJSA4Ik2HNpcQiJciaJf6S8nt8nraSh9D1Zmcvfk0ul0B1NTugBxcrFSaBdSfmgmZhKRJKX632xQvSGwJI8PkcxyYDsNoltogUm5x6lJczEFDqwxwK8ZprVVehgwh6HKYxXC7OoHmzyWxOVpB2t4xnZMN7LMFNioeGwBdTmYmWC7uXjNa/CiO1Rk13DcO6WzXcI0Wj+GxbK4GMVkoBHp7ESDWk4wIjAnl44xV7zEzkOwIhjnZosDGNoJqd6jonA0J6zpWHGxx5a9fMPVOl8hwplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTggPj4Kc3RyZWFtCnicMza0UDCAwxRDrjQAHeYDUgplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMzID4+CnN0cmVhbQp4nEWPSw4EIQhE95yijsDHH+dxMumFc//tgJ1uE2M9hVSBuYKhPS5rA50VHyEZtvG3qZaORVk+VHpSVg/J4Iesxssh3KAs8IJJKoYhUIuYGpEtZW63gNs2DbKylVOljrCLozCP9rRsFR5folsidZI/g8QqL9zjuh3Ipda73qKLvn+kATEJCmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDAgPj4Kc3RyZWFtCnicNVI5bgQxDOv9Cn0ggG7b79kgSJH8vw2p2RQDcXRSlDtaVHbLh4VUtex0+bSV2hI35HdlhcQJyasS7VKGSKi8ViHV75kyr7c1ZwTIUqXC5KTkccmCP8OlpwvH+baxr+XIHY8eWBUjoUTAMsXE6BqWzu6wZlt+lmnAj3iEnCvWLcdYBVIb3TjtiveheS2yBoi9mZaKCh1WiRZ+QfGgR4199hhUWCDR7RxJcIyJUJGAdoHaSAw5eyx2UR/0MygxE+jaG0XcQYElkpg5xbp09N/40LGg/tiMN786KulbWllj0j4b7ZTGLDLpelj0dPPWx4MLNO+i/OfVDBI0ZY2Sxget2jmGoplRVni3Q5MNzTHHIfMOnsMZCUr6PBS/jyUTHZTI3w4NoX9fHqOMnDbeAuaiP20VBw7is8NeuYEVShdrkvcBqUzogen/r/G1vtfXHx3tgMYKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1MSA+PgpzdHJlYW0KeJwtUUlyA0EIu88r9IRmp99jlyuH5P/XCMoHBg2LQHRa4qCMnyAsV7zlkatow98zMYLfBYd+K9dtWORAVCBJY1A1oXbxevQe2HGYCcyT1rAMZqwP/Iwp3OjF4TEZZ7fXZdQQ7F2vPZlByaxcxCUTF0zVYSNnDj+ZMi60cz03IOdGWJdhkG5WGjMSjjSFSCGFqpukzgRBEoyuRo02chT7pS+PdIZVjagx7HMtbV/PTThr0OxYrPLklB5dcS4nFy+sHPT1NgMXUWms8kBIwP1uD/VzspPfeEvnzhbT43vNyfLCVGDFm9duQDbV4t+8iOP7jK/n5/n8A19gW4gKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxNSA+PgpzdHJlYW0KeJw1UTkOAyEM7PcV/kAkjC94T6Iozf6/zYzRVh7BXIa0lCGZ8lKTqCHlUz56mS6cutzXzGo055a0LXOAuLa8L62SwIlmiIPBaZi4AZo8AUPX0ahRQxce0NSlUyiw3AQ+irduD91jtYGXtiHniSBiKBksQc2pRRMWbc8npDW/Xosb3pft3chTpcaWGIEGAVY4HNfo1/CVPU8m0XQVMtSrNcsYCRNFIjz5jqbVE+taNNIyEtTGEaxqA7w7/TBOAAATccsCZJ9KlLPkxG+x9LMGV/r+AZ9HVJYKZW5kc3RyZWFtCmVuZG9iagoxNiAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNyAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggNTYgL2VpZ2h0IC9uaW5lIDcxCi9HIDk3IC9hIDEwMSAvZSAxMDUgL2kgMTEwIC9uIC9vIDExNCAvciAxMTYgL3QgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDE1IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDE0IDAgUiA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxNCAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNyAwIG9iago8PCAvRyAxOCAwIFIgL2EgMTkgMCBSIC9lIDIwIDAgUiAvZWlnaHQgMjEgMCBSIC9maXZlIDIyIDAgUiAvZm91ciAyMyAwIFIKL2kgMjQgMCBSIC9uIDI1IDAgUiAvbmluZSAyNiAwIFIgL28gMjcgMCBSIC9vbmUgMjggMCBSIC9yIDI5IDAgUgovc2l4IDMwIDAgUiAvc3BhY2UgMzEgMCBSIC90IDMyIDAgUiAvdGhyZWUgMzMgMCBSIC90d28gMzQgMCBSIC96ZXJvIDM1IDAgUgo+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTYgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjU1ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDQ0NyAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgNTMgL0xlbmd0aCAzNiAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NDcgPj4Kc3RyZWFtCnic7ZwJOFVr+/D3vI2ZqUyVCJlnQmalpBSZhRQayJQpbVOSzFOZIkmkAYVGTqPQoERzEkWjIuO2fWvt9tqzd+t7r/973v/3nfs652qvtZ57PcPvGe77fp4FhvtH/jcL7O8uwD/ybwns7y7AP/JvCQz439uvtSE3OjQo0N/Jdo1vpIGpr5dPVUFwXikuu6a6qjLJx9O/KGanr69PTJKRdVAUDlfePWdpufm0DodLezJnha6urnvxuNhbc0hJ+ZWNw52Ze5kAOY3D5XSxTgYVqbv7ZgwuvpVFMYiFh34+ScHhLtC95F/nUobDFc41LTHBVRI/B9e3N7MTY/bt8bZbZ+odvlQt0HFz8+mgiobk0r9qzl1K8fQOKU8M2O3nvi9Bw9g3HIern5mzfPlAaMPhcqfmrkGY+ZKAi+2buwKgUoTD3fgThZlmHO4Y4U8UemNxB77+icIk0Kfu/1GZLuBwlX+k8IzEz27tfm/j1pGhovXnbl3LC7ePHQ7Qfngrp733SX2apXlZVtyrLtyZjlfX/DdvPJezMfCP+BGmZ/6Q38yf8iP8e/zmhJHwh/wI/0l+4cHb3DZ3jI1cOdDcnHdwX0Txz52q96/FldVeqKuNiz6Zse9Zq2Ny/cfmDes3F8St9v8jfoD8T/P7bxx//1F+dQ3WHgeefP70/lV75ertsVXXX3so3Dvr4bYl/MzQz/6zh3c3V0rZRg9ellzt5edkuPMP+RFa/1/g9/6/bP6cnqHwu/709uWz5ee/dV2rLCt0VzNdtc7GsbIg2MMj62BIy7OK9PDNXqWB1nbGXPrOfknXs/6/HH//bfxAgfhdezH85FTG0ed36wqLajxkFGUETbwKs6P2BJftX9vQUZF3QHf943izhcIShhu3H/12/B9+c5H/IL8jp3tSFqcU2R6+XZ7qHnesMmaRlGpJQfTd3psVCVfvNV65lppx/XjCYtMrPjq2sZ35//Cbi/wH+ZVWHfZUCUtLPXWjscQl/3ZekIq0QvnxhIabCVnVnW8eXynPOhzns0He4XGsY0593zFm/KZ+zk7oH35zk/9rfjXVooq2LhFfOi93XN5S/zXQ3mz58uqqtCOHFYNfjkwON+7P3K8pgtIOeF+WNEzAn2TGb/jNCIEoTLKZhd+s6WflB6RmrjArv9kUZuVHmM2f+Bf8mGfxr/gxr0Td7PyYlwni57c1ZI+vooxj4MGUvHPPfxb5CatYppQ2BKxVdSuqrq0vS/UPq7RfXPd8tCr2VLYPE/994mKG20b7g7vNdsUUNH8cxYNeH0WY8hs5nrA1OXnV9rTqWwyFY87vycXgxBInh2P17aMM1WPK71tXxoFzUdYnbrwfYXjGlB/+y+WkmjK7I49+4RkUmPOb7s87e31HTA9jBrPxmzpb3ZHu3z/B+GS28Xf/zJP6ne1MHpD52Zjvj/BXkDRxDvSOvPvocbbPQlP3Q0cq7TVN3GIPJCYdiLXb2rDH/N1Ib0FgpN8qev9v6nvf0wQ3OSEhG1W4iqV32tUHnc9f9/wcJydg5Dc+8O7hrjUytrZouc1hWV3PX/R8GKFKwciPMDTYUxGrbb1j4Xzf+KIHL970fabuIUz4jf14f6/SyTrSmNM/s7HtxbsPPyapHzPym/o18PqvgzbRuwQ9Stufvuz9OEZTZkZ+hPFvfd2Xt0SmKFpcaH326t3AOM1jJvwmf3148yQirMxB7XLHs1c9n4ZpHzPywwMN1V0cXh6vXPTs2cueT0O0jyF+Vkocyh7ReS+PbRFROb9VVlg7qaCiIUpgsVbWgV2rNflFZTi55eLu9V2zW6myyK4+g47f+yJHRX5OJAKBRsKRKAwbJ9c8YfUNNZ3kBIz8Hu4xlmZDI9FoOBKNZePin2/gVf+e8piR36+yvUbzsUg0BoHAsLHzzFd23P+duikZ+bWW2ypzYtBYFBzDziUgarrt5Fvqx4z8PtwIWMXNhsZiEKDC4lW+rTRFYOQ39ih3sygHBsuGRHNx8SwzDXhI3aWY8eu57qHOhcWyo5GcXFwy+tEXaB8z8vvcGWDJzY5lZ0Oyc/EsMgg/TtMHyfy22WC0whxdMlMTNKzu7pBWWH88Pzt+E8bU3W+ri+t6YVXr5Ro+5/q6rsbbGQYdOnUYcPjHv/S/fNrx5PGDxvLMaDtFQSwKRiUIzHzFrCZyNjT8xnovFaUHrpDkRUCp4XAUm4ROYPMIuT3p+H2oKTzkuFKSnZgWFARWSHnViTZKCjp+EzfPZAY4yQkj4CQVNPsibc/SVqra0/Hrvp6J81shhYCTBM29VDeypPMnJQUdv09txYeCNihw/K4BAo7gXaQffbLnOyUFHb/xjprUKB81ESgHOK+46Y7y59Qjio7fixtpsSGUMiG4xI1dT7SNUXUSiN/BME7TLBlOxeALrv7vI5XWBTZmhBpJw/dkKGutjgxbuinBaUfT+8mb11vC7N+0pcTjcOeHntyqLsnJP5LmpsFBbCUIBVEAghx7KpjxI0x/vrJFCkWbHCwc5+Kc9+RFh5ofgTB9a50kBswETiUoTstESk1o+BGmv4Wac6CAYpDeTWwBjMCmAz8oxGj4EfDH/TgQcBpBoEWtCt9SFGj5TbcnStMpwBGiuvXPKQq0/KY/51iTklE0BNVPvaRaaWn4EaYqg1Ew2vRILt2kwQlKFhA/I4UN5hqhaSXpUQaqems2KSqbKojzLlDw8BWQXXskk9u15kicfWjGpdrM2ssbzY134XAX8b/e5QWtN9BYKsKNhFPxm8fPKyKJBX4gxVy+QWs0xG/y440SP1vl+exwCj4BGeklC7mAorHLrf3EyG/q26U0K1VBNgScxA/BpaSiIjUfjkDyy226xYzf5+v+RpL8SMRvfggkWkFTV14chUDPl/M4xowf/l2llYIokpiY2Fji6ivUxTkRWEF5j+QxZvzGezbrLWGHegcCPk9ZS0dSAI7l0XROfcecX7SljABV75PTMFwqhEBza9rmtjDnV2enIgHUAPm7SHDR5fpKQlgUj7x1Rj2530L8zDWcbAyzzlTlxWgt4l8XsnSJobSIoKJDSJCork/VMSGPmoxQPYfg2ircmUb5RQrbiOvf9xM4Fyk+OGn+4OSXEBNXVFIyMDa0WKckAba2YRc0N5D59Z/eqyr6mwQH3zLZZQpKSiab7TZa6S3HAlPWgq5fDPx+daU5chObiY1fRgkQFT13D8+NlspAY7DPr4QsBip+PzuSNYHRCowfwUVKSsoqququ23c4rVVbBEexy0Uw4Tc5dDGEGw3CExRXUQZUlKy9d3tb68qjESglR7JhSeFHmHpeJwrSQ/GILldWUVFWNvDw8bMxVuJBsMs7PGHG73u3JQ+owCFOrISSqpN3gJ2ZshicW9Kjmhm/yZdxC4E2QWLEpcEiKSutcdntbqaxDMkh7JJCHrIQvxs3Fof/Or5XOaVvlbJtyf3lazoK9+3IfNHXvaPg/cM6By//hQs2BR0qOYozljRcuWLHb/sFPzmshSHxw+h4lGYXDI2NjQMy1lUC3peJfUTHb4ZQvJr790hCqLs8ef7kK6AwMTE+/mtABqjdvKNQV6TwexuviyH2P4y8x/0xokyAGmPe4M1Dz/D0/CYbI9jBoQRnW7jzDFlhYnz0NJAx0m0Kmnwo/HquShKHEobHO3eYqDA+MQko9CwAGlyavD5R+I0NuC4ArDUEmm9dRi9JAchhbGQ90AnnNTHhhz8pwwn2QT61o+1A8lGwTEAWIzlAkbijmPF7q70AHHfc4qmXR6EsgBweAX2ZU5/se0D8UpI2ZPSHrhJ1KwncEVJ61T/507kU3NG77X/tjKtqb3IOStY1zt1pW3f5RJCdnYN7KGR/Tprxg/zEXP33HTnb0foA6hffWpzlYDBBi0Z6fjNFllwAPI5dEZE5p798/wLZE4RfXupwGNvmXAZ+z/0VwXpsiNiXfnZwhkqSNIBZelPqL3p+4/lbQAXliOjEC93UCtcVeeFw/ePQskrhdyubHwDOF7Qv5nwbtQH5SUMQDp9f3sHA72OjCQcAwzkq7kQLtd837c4HWMeHbk/T8xu/F8ULDHC9iLjs9gFqhWKgq6Fd70BuBIVfd7U4AHzhXlzSnTdUCoRnAD+Uwr2PpGuI3zrz5NOt5vO5FTcUV+YWlzc+JFw7kplfdabc13vbrVa3hDMee1558d9723LmyO6g+BgyP3spcI3Ru03TsoB8b9wImDDYEtIlhd8xKy5AQ6ifLv3MVJkfHIbg8yRdUvg9dZcBZ9sqeoWZ6p3zYLAF+t9IFSPzG923GsgB4cmg0O60GAYTWXmHdEnhV7mTE1CQ/kGv8HWDLAzGZVtMuqTwe3pAHlzgrzNksWsJEga3iSf1Sgq/H5m24AAPp08/XcoPYDVIgPoUhV/1XnDy1JqkUyA8B9cS8UzImYf4BUX3vHtmq+EXm+NiIR96tuRk4y4jZfNdZ/+K87Xee/hs9cm9e+IN+EpLtu6Jrkkx2QXxI7woBQqMWMUQ+uxLMgDnSKjqFH6fHiwGlr4lH+hrMh5oBLQ6yoF0SeE3+soRBmNfWMvQVgfVgPGHXfqFVDEyv+n3R+Ew5MIgBoV6ITYYjFPiKumSwu9LuxAMLmpK50vP4PulAYcFq5BIuqbwG+1fBYPxSN+lU5ge28wGVELVg7QoU/jhPyYh4JglyfQKQ1koYK6QsX9BukHh96kZGH7iG+j5DbeB/AS3XiRdQ/zSy8d/vgtzCcEl6kuLbMnNOFLlvUJGxzGzbI+nXURmS1NtXKCHmkhSsmdYysNj0PoHym0sAo60psphlDih9UbrgD2Ukd/MiBIWziP/cYZWCKPemkDV2ZxJ19T+w25OBPcy+ngdYSpyERoG417OwA8w3Oah0csi6RRmxk8TfS75ZtI1lf05IMUGl7Glj4D9ejEfWBvY9NJI19T2pwMHXES7jU5hfGAt2GkNA0jLE7X9UsSJZFfPplOYencQ9AlUtkNTJIUfvksQDV++hZ5f/2XAUkeJhTWRriF+z/Djo78mnuqJo9mEjbXUQ7KuFx0M8FovI8hjkf5ssONlX8FePVUlx+ibT99P3vEOo/Br4cfC2W2pcuhoB2f/vkOGMBiK6zjpJhW/USsJ+AJd+vkW/8NXFw5DLQskXVPzS9LEChrQz1UTA3skACdSaytp1qPmd9NIBGvI0Nff5oH8FMKgNZGK37d1y+A6Pr/oNF43CAL8hI60kq6p+YUoo5Y6dNIpfGzUh8GQ2MwHDOvfzMxpZe5560/QKYzW+YL2y87XkBVN4Tf1QkUIYRFJH4W9egiwtPhMRqFlGuKXW1xU23azIctOfP22+B2uJQ0dNcdOxZsZm3vEnyvLCw4O9vUNdFiVdHBH/qXuy+ChQ4jffVk+ONrs5e8wxcSjlr/q6tr+upi0x1wSBuPVriOlouI35qcH515GM/5+9N8t3qshDvgD2yDg1PwqtnJzSNIGmqaf1gfp8AAjP6CO1Nep+XWGyqMkQ2kr3ns3bA0wwDFW10kDlprfMM4KLm5BO38OtR1y5gAsLdm70FxPza/Ik41f8R6NwtT9404SwFQvcmaA9Fpqfjd9xdnkUmnL9PySqwZg9UruH4IwUfhN9/toIpY60o6/gb8CTFFwuJQ9+TbEz97cMuDE4bwPaXq555sKE+8+fX3t/OtyfY+I2mt3dpgunIfWc6uKcruULeKWUlNdcJDC74kJ0O6adW9At2HyW3EqLutIToyvMNGpEPeC3GsqfhOHNyPhgsT045P4SdDufnX7sBHRCeEh23rU/G5nCSLg5USFCfzU5CRod1cEEGM8mFNQRaj59Z82RMDdiArjU/jJySkgj+YUQdCq4d5ObgwqfuM1uxAIqU9gBhNTQBZTQBYvs3WIARI9sllDze9WDhcc0UhUmJzGAzoTE0N5LmAleGUhA4mG38syJQQ8hKgwgQcUgJqPn4sAfQp2jUxymaj8h6ESeyRS4ydRYWp6agpUaIuRBp0QbT+yuQHxu9gUXNh5qTzE33fLJh07X6c1isv1um7GHylKcJHT0deSXxKXE2e5+ERNfnyIU+TlTAq/kUeugEOyVEFNXU3NJ1RpqaSkpMRCITSRn0wCg/8HNPRgFR8WpaCmpqWj5ZzpuX6dhYHSMjFuMD2b+DNoX4ia38gbbSH4EnU1TV291YXxHl6uZnqqS0RAfpyLyWYNNb/J77uWoATU1DS09QxS830DQjeYqS8TQ4Eensp+ZvwIQydluNiU1dT0jYxCqyOCE31stBQkwMaFa3mTp1VqfsNP5fng0moa+ubmbheyIzLi7U00JAXASshvIzst1PzGP68XQMxX01qx1tKi/NT+rELPdZpLF4B+qlDEZWb88J8OCWM41TS0rNeZJl9OSq0KtddVWMgOmDuo8HMM8ZezDVEney4WBQWHWq9UM7NRk5onteJU0d703DArfjkNE0tr3OE0K/EjVeVHU5wDjh2i8MP/2EmMnoFBOk3zeWxoSlCaU6sQsqto4tdNoB0IOLoYlKyrsoSYICeSpCGgTDZLaeLXP4wWgiFMJBubyHYbJRV5fjZSJFRA7wozfjMz4cvRYEfFsHE5emvqm0jyk0KI7GuzmPGbmbmoxgs+55zHYxJqYWSrKYYmxbnWJzCNn830a88HDTROAYHl+9wtXGyk+NDEPBAGmb3M+M1Mu0kgwfjLQiFB/0Art23KYhhilAEpceIxM34z0wVS7GDwT1SM3z5uk2OwiRQbCmwGFMfRh+RyQ/wsdf3LeuLCW47v1TVNt+RB8mnGHhNk47b0CnAzl5Cwy7ijY9aJM9m/z73556lQUvyMKFOffeGkGD/sd5gY4odU8nwEVZeWnzA7eRvh9z8kDfUt0NpEx89YlBy/RRB1SJFQ2TBogNPxi1KG2h8BxklIv2Fw3tRmchoafpf0BagVyBFjxCFyTIKW34DVUqgSSCSkA/yH2faDrEAb/wxQQ8HJCoAGCosCI5sc6lTrLk3885Q+LwIqExKJAnXAWCuXWCtj/DrAy8A5PuloV659Yv55fyOXjeqR+eZmG923elgpmW5O3b9D2/pKQXx9qfOFnrTtqj4Ufl9rnYUAO5B/ITSIiEOPixNwwVR3foL6Ls38eVaSC1y6EFQKMAQSYKK5dRZ+ZhJw6sQwEkGkQjzZBKTjp4Sm6RmkroIQKKbY/DT8GvX4qYLwMKhnIdnze8hzFS2/tTIUhd9tjAF8OaRA4CT5pbT8QnVQcKirE9+NZWcHiIiYUJ0loOF3xpIfQbXpgsaysXMBrxBUekhRgPhlHZwvqZF68U3yypae22l+FYe1Q3OD92UGb7XXF/U80BwgqWtfUt344YZTRXvUFiPy+V0C/m2MjTQGBlusgoEjKfURFAT57SFXhYofvrNYmgfwLThJtfldPCQa+K3lzZzfkIUk1X7F73FL5Kd0iLzW0PKLUECT305qXSQ4AwtVdJDT0PBr0OaHUwTo5Sji5IbhKf9CTkPDb9BGDknZ1wFej8awY4H5USKCEoGj5Re5EkMZ10g0CsM5jxsgvmT9bPxqNgijfxcDzAHNycPLLwhkIWr4hKIA8QsJSfZWj84rrLxw7LCnjfmeAxUN14pLqhxNMyNs1S3unIqzt7Kw9Xp9b0d61YPuN0Ukft9eNZQfMsYCDcUviuRT4SHzY2cHmo1ds5hx/fvZXxuFRf5eL7FoaiaAq6XaDfnQ1Pw+PZDCUvFDo+FYTgTx9/xVTcz5+Qmzo8l9nVOQnVucBwx0IHm2lzPnd1F1kTCRB5iKQ1JMVGkpeInhDTlLttVp+H12M10uyIlCIkHaqOUq8jr64P4hWtL3MXk+pOWX6aknK8KFBdABQ1XZytDcQQoLaCzbNEwhTsOvJc7WYJEwNxoFFJ1tiaPtBm8zTmCRlVpPFdWF+Lk7xHjphqekVFw/lbNzrbZvRFpja1lJ2Rq1uJ0WS1RLM/fv8nay3VhTk5yW1947U0Hkh/9ytzQ60FcZbFguPgTv8nnk6QQNDieYXAo0vUH8Jt7fOx/rAIf2LNAIDBaFYUcRt7iAl0h1M+4fjb+/fVIM83tJQqC4+IQFBYWFhTiIexK8alCAnIbfxOcdUsJ8nBzsGHZuoYUycsvl5FUleJCgq7yOEgKh4ddss1JDYoEgH998SRktfROLVWv0hdnBTXjXTPKZFhp+Q4cCnVeqySwWl5LVNjBz9dzq6bKYDxh/C5zqyZMILb+atGBPGwMNWSVNI1MLX1xYeLSROFAmScvXlMArDb/nVQf3um5cratjZGyyyj0jKTHTdykXAi5u2k42qcj8VAXE1dYGxyWl5rzobfGVDXFBp3cfTd6vLqIqgeAVVVdTL75675CDtndPQWjGFRK/H7XuXCgEgjIJUq02xN/Lc6GeAvH7kO4gC/YhUjIkVngBl9AiHo7fowUtSj4BQ+HXm7bNUpwPjcEAlLHzFIzs9VdsWqmwSBjMg13sPBN+0/01YXZ6qrJLFwktVrPxyo7aiotwMFUFZzcOxX3M+fWeScf5uVqvNHIMyWxtyTlzLX7nGjEwC6qwDO3+O+HT/bPFB/b6JhTcH5jqfDbWlOdlAFQJo73/NXN+BALhZ/+tC0errv2cnh7/NT3YfMCNDSiTZCnZCKM7PwFoTI//uP/wy+j09DSBMNVbH7gYDmcTqHrDYH/GRtmuXRUYtdvDIf+ol7pweIhFVOYqj/zkSGOzdf4+YvIGIUnFbor6ewaPBl59QeI3fMlPDIvh5wCMIwQnL5+QhKKWhUvEOkNNZdnfB1VkkyDLGOL3pTLYap3JomXL5cBPLHwiswsKc7PSw7atlACZYhZ2Ms6f35sbTm939/L2iYg5mJF34kz96dMXzxzfakz0/xYx8/8II69v11edPF56LP9YRd3Vx3ev3rvTkLIdmN0R7NpxzPkNv3p0r+lSbfWZxlsdgwOPX71vrY1VBfmtDJiF38zY59dd7beb2p5+Gp35+n267+GRLUB6lE78W+b8ZsADbh/fdr7sA1piGj8z2lebCB47kChn7j/8lunJT5/Hf1u0hOHHxUpAL+eqeM3A73prmKf17kBHM3X/7Uv4uCMTw3e58ptVtt218UioP8Evv2qTe/iKeab7vuftfjI49fv87siNSHXA/RHn5+PlFJORVTF08ImteJYe7O1sI4AEzQaZOHr//UdTxq6kcGNrO9stUQlpV56Dz0Y+ttb4E/eBMQtaobmHwm9iYHSivORcbUN37zfyUjRVtA2FACdcKEDH8vx1S44IMAOzGRxkzo9RBs6bgh6BWSjZvGBx/nryZhTIT/fAW/IdFuevO6t4gBzEyyn2CIvz1wM3dMBzCMdfMPDLS1cMaN+gapV0u6OzK9kuNGz3FtfYHfpZp+1Wm2aWH9jrs0J7rfpSr+yhHGeNbS25RH6E0YFn+cVvXr98eTu9urnrVe+HgS8/J759Guh/7aoE8lM7DVUF4jc9+m3w6+fevv7+j5+/fvtF5EHAjw0PpuhzADbp/LfQUk5lv+CBeefH8MjIxBTV3urQVWUBGIxtwTnoBit+Y336IsB6pkDZ62bBDz+8HfT6lV2YnJ9gLqNloAsvu+UpdIMVv4kXoNMpGEzZW2HBDz+6CrSvQqsZ4i+FhXYRZy2WacTefz/wOW3NtoCosMiK5A0nLnk6OxdXRvg5y0upKSjsLf1Yslt09ekUkv05PXGX6E99be76MEF1Fmwi2JgTDUdpk090svz+4ZSTIBqOEWey/s0ij20XA/xEzpLzZPX9w4izErjBEUa+wfL7B5wKBgZfZks2J1l+/3BBCnBsF28guygsv3/4oACYPLxbTlPewOr7BychBAzheYTh/Mu17p6UpRrKiHVF7wbuu/MaeJ261trZfmXwS1zupYcXpKXl2ICBrlvUeP/6KVn98GgW3x/hD7sv5YVzGTVB2wws+d1JVeCH88i9hxqUJb++I4Z/NP5mJvI9AH6Lgsk3WPI7H8oNgy80IZ8AZcnvQYA0DCaiSw4RsOT3bZc+DM5lVki+wZJfnAUahjAPZ4hfn2h4jBNRVkTYHj9/KuFwdvURT0fXqmsPs2NrM7z27o+1lTLU4Xc52vm8vf1OUmJ6Agt+hI6aXc7rNkX3Q2s/S34DD6M8TO12kpuHJb+RzqSN6oae5KN3LPnhO4r1pCS2HCPfYMnv6VlVYbhV9FzXP8CIrVrJCVcKhJxe1vx+VXjC4QIhDeQbLPldiGGDI32KGMZfav7N8AXqatxO5clR7mc/fb2oJiaRWf/azeBeorSJ5409Etu8liR2DX1sedJ9+2JREsvv/z6X5x1OPUe+ZP39GL6xODyllLzWsP5+jNCc6baz4BX5kuX3Y2OtPiY6GbPEP5nJ104bBfaA0+SzXiz5fe+wE8GaFpHP9rDkN9EehUaJ5VO+TGHJr7OCG4mOvcSw/tmvT83KSTqYm5cXeHDwXrXvxhUJF/s+fSrfv1ZN+Ni9qz4LMgq3xueH7i87WRYetCeKJb/p4R9DQ5QN7Tl8/zfy/cPX4dnOzzMRwpc3dx8Nkp1r1vxGexuOFz37TL5mye9zd87+7XUvmcevmSrcjHVfFd1BnnBZ8hu9EGugYHSDvGHBmt+1dPn5XEVPGOxPx40ZqQcKymtyIzb7Xs0OsrQJON3W1NZXnbrTxTrz1Klgk/jsnNKT9k7x0RG7g6j33+cmc+BHwFNZQHPhN/Ll3cA4WWUO/D7cuX59kHLIhRU//OcXpXnpbYPM49dMZGrw7mF//6IecnSEFT/8z0vJa6y9Oil9igU/wlRTjrqSWs1bBn6BIacjdE/ebdqnv0RCihclu+3Dq5a1QY/KCt69fmirE5FRtnnX149d2vKrNZT8k8+n/Y98P03dmHPg9+v7ONUrWfIjjA20vvwwSlFh5T/8+vy2/PzDQUqsigU/wo8PT7Ojah78nGX/iFGGB9sKNydVfqV8dcaC3+TPtlOGTolPKZ+8QPxKzjflbolILCrIP5m9ydlVUjfl2DHzqK/dd/KufL1emZl2eHdQ/oG92UUntxjERXnt/S/4/p0wNjyFx1M+QGE5/ia+dQ+OjM+ZH2Hqx4em1o/DlM8sWY2/yaE3tWWP+8Zn2T9ilKmRrosH6trHKIeUWPCbnnx+PSitcZBSJohf7e2mEzG+OzJqHw88DIpLWirnGJVoEov/3rW1ZILwtTA1Ljxw6xrj+z2dUetzI/V3/P38CNMTY3j81B/wmxruHcZPzJkfYB1+ffJ6lEqB5fo39uFe8wBl+LFe/6be3qt61Ee1bLBc//oeFl58TrXhBPFbo2126P3zlz97220tcVZyuXkBh7M3xrwtDFHzvfStPyj9rJuF1oYd7Ze3p1a/uZUc97fzY/gOfg5/v2B6cpp6jmbFj0CYnpyiTsCSHwE/PjY9Pdv3Y8wEP/FrkvqIIEt++Mnh0UmqQkH8NppYJ7Tdvvv0Tt3GdTGbDY7m701K84q5ejjAcmdpy83g5LLtdms8QuvKvJPLb19IA/y/E0//RGoBD+Xxnyi0xONib/6L552d9DeycLhqFi+lVQL//iD9W2hS09+4EYOLv8ciCxq9xyk4XB1rBSopw+EKmOQ8u1zB/fP3P/+3C+zvLsA/8m8J7O8uwD/yb8n/Ab+4WcQKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago4NjgwCmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagozNyAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MDQ2KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDM4CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDE4MDIwIDAwMDAwIG4gCjAwMDAwMDgwODUgMDAwMDAgbiAKMDAwMDAwODExNyAwMDAwMCBuIAowMDAwMDA4MjE2IDAwMDAwIG4gCjAwMDAwMDgyMzcgMDAwMDAgbiAKMDAwMDAwODI1OCAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDA0MDAgMDAwMDAgbiAKMDAwMDAwMTAyMiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDEwMDIgMDAwMDAgbiAKMDAwMDAwODI5MCAwMDAwMCBuIAowMDAwMDA2ODAwIDAwMDAwIG4gCjAwMDAwMDY2MDAgMDAwMDAgbiAKMDAwMDAwNjE4OCAwMDAwMCBuIAowMDAwMDA3ODUzIDAwMDAwIG4gCjAwMDAwMDEwNDIgMDAwMDAgbiAKMDAwMDAwMTM2MiAwMDAwMCBuIAowMDAwMDAxNzQyIDAwMDAwIG4gCjAwMDAwMDIwNjQgMDAwMDAgbiAKMDAwMDAwMjUzMiAwMDAwMCBuIAowMDAwMDAyODU0IDAwMDAwIG4gCjAwMDAwMDMwMjAgMDAwMDAgbiAKMDAwMDAwMzE2NCAwMDAwMCBuIAowMDAwMDAzNDAwIDAwMDAwIG4gCjAwMDAwMDM3OTUgMDAwMDAgbiAKMDAwMDAwNDA4NiAwMDAwMCBuIAowMDAwMDA0MjQxIDAwMDAwIG4gCjAwMDAwMDQ0NzQgMDAwMDAgbiAKMDAwMDAwNDg2NyAwMDAwMCBuIAowMDAwMDA0OTU3IDAwMDAwIG4gCjAwMDAwMDUxNjMgMDAwMDAgbiAKMDAwMDAwNTU3NiAwMDAwMCBuIAowMDAwMDA1OTAwIDAwMDAwIG4gCjAwMDAwMTc5OTkgMDAwMDAgbiAKMDAwMDAxODA4MCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDM3IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAzOCA+PgpzdGFydHhyZWYKMTgyMzcKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-09-16T14:40:46.639358\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.4.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["for i in range(imgs_per_step.shape[1]):\n", " step_size = callback.num_steps // callback.vis_steps\n", " imgs_to_plot = imgs_per_step[step_size - 1 :: step_size, i]\n", " imgs_to_plot = torch.cat([imgs_per_step[0:1, i], imgs_to_plot], dim=0)\n", " grid = torchvision.utils.make_grid(\n", " imgs_to_plot, nrow=imgs_to_plot.shape[0], normalize=True, range=(-1, 1), pad_value=0.5, padding=2\n", " )\n", " grid = grid.permute(1, 2, 0)\n", " plt.figure(figsize=(8, 8))\n", " plt.imshow(grid)\n", " plt.xlabel(\"Generation iteration\")\n", " plt.xticks(\n", " [(imgs_per_step.shape[-1] + 2) * (0.5 + j) for j in range(callback.vis_steps + 1)],\n", " labels=[1] + list(range(step_size, imgs_per_step.shape[0] + 1, step_size)),\n", " )\n", " plt.yticks([])\n", " plt.show()"]}, {"cell_type": "markdown", "id": "d85bacf3", "metadata": {"papermill": {"duration": 0.032666, "end_time": "2021-09-16T12:40:46.804930", "exception": false, "start_time": "2021-09-16T12:40:46.772264", "status": "completed"}, "tags": []}, "source": ["We see that although starting from noise in the very first step, the sampling algorithm obtains reasonable shapes after only 32 steps.\n", "Over the next 200 steps, the shapes become clearer and changed towards realistic digits.\n", "The specific samples can differ when you run the code on Colab, hence the following description is specific to the plots shown on the website.\n", "The first row shows an 8, where we remove unnecessary white parts over iterations.\n", "The transformation across iterations can be seen at best for the second sample, which creates a digit of 2.\n", "While the first sample after 32 iterations looks a bit like a digit, but not really,\n", "the sample is transformed more and more to a typical image of the digit 2."]}, {"cell_type": "markdown", "id": "9b5682ed", "metadata": {"papermill": {"duration": 0.03313, "end_time": "2021-09-16T12:40:46.870416", "exception": false, "start_time": "2021-09-16T12:40:46.837286", "status": "completed"}, "tags": []}, "source": ["### Out-of-distribution detection\n", "\n", "A very common and strong application of energy-based models is out-of-distribution detection\n", "(sometimes referred to as \"anomaly\" detection).\n", "As more and more deep learning models are applied in production and applications,\n", "a crucial aspect of these models is to know what the models don't know.\n", "Deep learning models are usually overconfident, meaning that they classify even random images sometimes with 100% probability.\n", "Clearly, this is not something that we want to see in applications.\n", "Energy-based models can help with this problem because they are trained to detect images that do not fit the training dataset distribution.\n", "Thus, in those applications, you could train an energy-based model along with the classifier,\n", "and only output predictions if the energy-based models assign a (unnormalized) probability higher than $\\delta$ to the image.\n", "You can actually combine classifiers and energy-based objectives in a single model,\n", "as proposed in this [paper](https://arxiv.org/abs/1912.03263).\n", "\n", "In this part of the analysis, we want to test the out-of-distribution capability of our energy-based model.\n", "Remember that a lower output of the model denotes a low probability.\n", "Thus, we hope to see low scores if we enter random noise to the model:"]}, {"cell_type": "code", "execution_count": 16, "id": "303c00f7", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:46.941073Z", "iopub.status.busy": "2021-09-16T12:40:46.940608Z", "iopub.status.idle": "2021-09-16T12:40:46.945276Z", "shell.execute_reply": "2021-09-16T12:40:46.944875Z"}, "papermill": {"duration": 0.041937, "end_time": "2021-09-16T12:40:46.945424", "exception": false, "start_time": "2021-09-16T12:40:46.903487", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Average score for random images: -17.88\n"]}], "source": ["with torch.no_grad():\n", " rand_imgs = torch.rand((128,) + model.hparams.img_shape).to(model.device)\n", " rand_imgs = rand_imgs * 2 - 1.0\n", " rand_out = model.cnn(rand_imgs).mean()\n", " print(\"Average score for random images: %4.2f\" % (rand_out.item()))"]}, {"cell_type": "markdown", "id": "0a485fdb", "metadata": {"papermill": {"duration": 0.032686, "end_time": "2021-09-16T12:40:47.012111", "exception": false, "start_time": "2021-09-16T12:40:46.979425", "status": "completed"}, "tags": []}, "source": ["As we hoped, the model assigns very low probability to those noisy images.\n", "As another reference, let's look at predictions for a batch of images from the training set:"]}, {"cell_type": "code", "execution_count": 17, "id": "2bcb1282", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:47.081066Z", "iopub.status.busy": "2021-09-16T12:40:47.080584Z", "iopub.status.idle": "2021-09-16T12:40:47.233910Z", "shell.execute_reply": "2021-09-16T12:40:47.233401Z"}, "papermill": {"duration": 0.189477, "end_time": "2021-09-16T12:40:47.234026", "exception": false, "start_time": "2021-09-16T12:40:47.044549", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Average score for training images: -0.00\n"]}], "source": ["with torch.no_grad():\n", " train_imgs, _ = next(iter(train_loader))\n", " train_imgs = train_imgs.to(model.device)\n", " train_out = model.cnn(train_imgs).mean()\n", " print(\"Average score for training images: %4.2f\" % (train_out.item()))"]}, {"cell_type": "markdown", "id": "a559009f", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.032727, "end_time": "2021-09-16T12:40:47.300644", "exception": false, "start_time": "2021-09-16T12:40:47.267917", "status": "completed"}, "tags": []}, "source": ["The scores are close to 0 because of the regularization objective that was added to the training.\n", "So clearly, the model can distinguish between noise and real digits.\n", "However, what happens if we change the training images a little, and see which ones gets a very low score?"]}, {"cell_type": "code", "execution_count": 18, "id": "edff6bb0", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:47.372822Z", "iopub.status.busy": "2021-09-16T12:40:47.372352Z", "iopub.status.idle": "2021-09-16T12:40:47.374505Z", "shell.execute_reply": "2021-09-16T12:40:47.374110Z"}, "papermill": {"duration": 0.041215, "end_time": "2021-09-16T12:40:47.374604", "exception": false, "start_time": "2021-09-16T12:40:47.333389", "status": "completed"}, "tags": []}, "outputs": [], "source": ["@torch.no_grad()\n", "def compare_images(img1, img2):\n", " imgs = torch.stack([img1, img2], dim=0).to(model.device)\n", " score1, score2 = model.cnn(imgs).cpu().chunk(2, dim=0)\n", " grid = torchvision.utils.make_grid(\n", " [img1.cpu(), img2.cpu()], nrow=2, normalize=True, range=(-1, 1), pad_value=0.5, padding=2\n", " )\n", " grid = grid.permute(1, 2, 0)\n", " plt.figure(figsize=(4, 4))\n", " plt.imshow(grid)\n", " plt.xticks([(img1.shape[2] + 2) * (0.5 + j) for j in range(2)], labels=[\"Original image\", \"Transformed image\"])\n", " plt.yticks([])\n", " plt.show()\n", " print(\"Score original image: %4.2f\" % score1)\n", " print(\"Score transformed image: %4.2f\" % score2)"]}, {"cell_type": "markdown", "id": "7c540cbd", "metadata": {"papermill": {"duration": 0.033108, "end_time": "2021-09-16T12:40:47.440512", "exception": false, "start_time": "2021-09-16T12:40:47.407404", "status": "completed"}, "tags": []}, "source": ["We use a random test image for this. Feel free to change it to experiment with the model yourself."]}, {"cell_type": "code", "execution_count": 19, "id": "befba0f4", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:47.509319Z", "iopub.status.busy": "2021-09-16T12:40:47.508856Z", "iopub.status.idle": "2021-09-16T12:40:47.685776Z", "shell.execute_reply": "2021-09-16T12:40:47.686151Z"}, "papermill": {"duration": 0.212897, "end_time": "2021-09-16T12:40:47.686303", "exception": false, "start_time": "2021-09-16T12:40:47.473406", "status": "completed"}, "tags": []}, "outputs": [], "source": ["test_imgs, _ = next(iter(test_loader))\n", "exmp_img = test_imgs[0].to(model.device)"]}, {"cell_type": "markdown", "id": "12364d29", "metadata": {"papermill": {"duration": 0.033603, "end_time": "2021-09-16T12:40:47.754499", "exception": false, "start_time": "2021-09-16T12:40:47.720896", "status": "completed"}, "tags": []}, "source": ["The first transformation is to add some random noise to the image:"]}, {"cell_type": "code", "execution_count": 20, "id": "bc21923d", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:47.824609Z", "iopub.status.busy": "2021-09-16T12:40:47.824143Z", "iopub.status.idle": "2021-09-16T12:40:47.919759Z", "shell.execute_reply": "2021-09-16T12:40:47.919292Z"}, "papermill": {"duration": 0.132328, "end_time": "2021-09-16T12:40:47.919863", "exception": false, "start_time": "2021-09-16T12:40:47.787535", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torchvision/utils.py:50: UserWarning: range will be deprecated, please use value_range instead.\n", " warnings.warn(warning)\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDI0MS4xIDE0Ni4yNzE4NzUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicjVJNb4JQELzvr5ijHvp4+/g+amxNvTS0JD00PRhEihEMmNS/3+UrEWkbD5B9w+4wM28ZB7IWjOwMjYM8FzDWsFbpd56kr+slkjNpwQsyDiuW6thX7HjK+Bz4rkB6fPwiKklYpX0thBmxVj6MrYKuQ9hsW4VXyLFH2A6VHqB2aoQI8Z4q3NAZKQ2YXXnXKd5RwlqYxhSLKRZTemKqkikHjTVm75YwKWA9M1YnRBShGti0OGoYtQp6TkHI85Q78tYBWtmDalpKGBeqqMnvQf4IEyrf9QM2LkKJrS2SgpYxrCeJViPet7HHO/rA7KXOs7zcHpEX2yyd4xPxhh5jiqhVQuw7NxJ65F8NbDzVfrxDQTxvL7jpntXb8rw/1YL43fysSHdTaQabbqfasMbhT9ZhetP0NtmR4rcdkb47lmvouhr9iy2iHyt1pwQKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagozMzEKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTUgPj4Kc3RyZWFtCnicNVBLrgMxCNvnFL5ApUASCOeZqnqb3n/7MKiLEdbgH/HrmDiGlyz4EvhWvGWs2DBTfMdSLaR2YOtAdeFcxTPkCo5eiE3stOBctrlJpK4gQyJKI9tyQ5dQtCk6JX9vmlu6KbcnTZpu08rA1MuQsyOIGEoGS1DTtWjCou2p+J3yjL86ixd+xw4rdNzh01MR9T3DZz6IS73G9qjZmUS6L8iQ05pLCU002dHvyBTOPDekkM4gQVJcgmtlkP3pl6MDEjAxtyxAdleinCVpx9K/M3jS5x9hXFSNCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA2NiA+PgpzdHJlYW0KeJwzMzRUMFDQNQISZoYmCuZGlgophlxAPoiVywUTywGzzEzMgCxjU1MklgGQNjI1g9MQGaABcAZEfwZXGgBSaxTACmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzMSA+PgpzdHJlYW0KeJw1TzmSBCEMy3mFPjBVGNtAv6entjbY+X+6kplOkPAhydMTHZl4mSMjsGbH21pkIGbgU0zFv/a0DxOq9+AeIpSLC2GGkXDWrONuno4X/3aVz1gH7zb4illeENjCTNZXFmcu2wVjaZzEOclujF0TsY11radTWEcwoQyEdLbDlCBzVKT0yY4y5ug4kSeei+/22yx2OX4O6ws2jSEV5/gqeoI2g6Lsee8CGnJB/13d+B5Fu+glIBsJFtZRYu6c5YRfvXZ0HrUoEnNCmkEuEyHN6SqmEJpQrLOjoFJRcKk+p+isn3/lX1wtCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicPVA7jkQhDOs5hS/wJPIjcB5Gqy1m79+uA5opUEx+tjMk0BGBRwwxlK/jJa2groG/i0LxbuLrg8Igq0NSIM56D4h07KY2kRM6HZwzP2E3Y47ARTEGnOl0pj0HJjn7wgqEcxtl7FZIJ4mqIo7qM44pnip7n3gWLO3INlsnkj3kIOFSUonJpZ+Uyj9typQKOmbRBCwSueBkE004y7tJUowZlDLqHqZ2In2sPMijOuhkTc6sI5nZ00/bmfgccLdf2mROlcd0Hsz4nLTOgzkVuvfjiTYHTY3a6Oz3E2kqL1K7HVqdfnUSld0Y5xgSl2d/Gd9k//kH/odaIgplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTM2ID4+CnN0cmVhbQp4nE2PQQ4DMQgD73mFn0AgQHjPVlUP2/9fS9h20wseyYBsUQaBJYd4hxvh0dsP30U2FWfjnF9SKWIhmE9wnzBTHI0pd/Jjj4BxlGosp2h4XkvOTcMXLXcTLaWtl5MZb7jul/dHlW2RDUXPLQtC12yS+TKBB3wYmEd142mlx932bK/2/ADObDRJCmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDEgPj4Kc3RyZWFtCnicRVJLbkQxCNu/U3CBSOGXkPO0qrqY3n9bm0zVzeAJYGx4y1OmZMqwuSUjJNeUT30iQ6ym/DRyJCKm+EkJBXaVj8drS6yN7JGoFJ/a8eOx9Eam2RVa9e7Rpc2iUc3KyDnIEKGeFbqye9QO2fB6XEi675TNIRzL/1CBLGXdcgolQVvQd+wR3w8droIrgmGway6D7WUy1P/6hxZc7333YscugBas577BDgCopxO0BcgZ2u42KWgAVbqLScKj8npudqJso1Xp+RwAMw4wcsCIJVsdvtHeAJZ9XehFjYr9K0BRWUD8yNV2wd4xyUhwFuYGjr1wPMWZcEs4xgJAir3iGHrwJdjmL1euiJrwCXW6ZC+8wp7a5udCkwh3rQAOXmTDraujqJbt6TyC9mdFckaM1Is4OiGSWtI5guLSoB5a41w3seJtI7G5V9/uH+GcL1z26xdL7ITECmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MiA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlxAvqmJuUIuF0gMxMoBswyAtCWcgohngJggbRDFIBZEsZmJGUQdnAGRy+BKAwAl2xbJCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0NyA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlyWEFYuF0wsB8wC0ZZwCiKewZUGALlnDScKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1OCA+PgpzdHJlYW0KeJxFkUtyBCAIRPeegiOA/OQ8k0plMbn/Ng3OZDZ2l6j9hEojphIs5xR5MH3J8s1ktul3OVY7GwUURSiYyVXosQKrO1PEmWuJautjZeS40zsGxRvOXTmpZHGjjHVUdSpwTM+V9VHd+XZZlH1HDmUK2KxzHGzgym3DGCdGm63uDveJIE8nU0fF7SDZ8AcnjX2VqytwnWz20UswDgT9QhOY5ItA6wyBxs1T9OQS7OPjdueBYG95EUjZEMiRIRgdgnadXP/i1vm9/3GGO8+1Ga4c7+J3mNZ2x19ikhVzAYvcKajnay5a1xk63pMzx+Sm+4bOuWCXu4NM7/k/1s/6/gMeKWb6CmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjMgPj4Kc3RyZWFtCnicRZA7EgMhDEN7TqEj+CMDPs9mMik2929j2GxSwNNYIIO7E4LU2oKJ6IKHtiXdBe+tBGdj/Ok2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlDcPVf9b9i3TmbiYHJyh0IzepT3Pk2O6K6usn+pMfcrNd+K+xVYWlZS8sJt527ZkAJ3FM52qs9Px8KOvYKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxOCA+PgpzdHJlYW0KeJw9ULmNBDEMy12FGljAeu2pZxaLS6b/9Ej59iLRFkVSKjWZkikvdZQlWVPeOnyWxA55huVuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rnKPn16LGz7D8UMUkZIO5jX/WP3ycw2vU48nkW5vvuJenKkOAxEckpq8I11YsS4SEWk1QU3PwFotgLu3Xv4btCO6DED2icRxmlKOob9rcKXPL+UnU9gKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM0ID4+CnN0cmVhbQp4nC1SS3LFIAzbcwpdoDP4B+Q86XS6eL3/tpKTRUYOYPQx5YaJSnxZILej1sS3jcxAheGvq8yFz0jbyDqIy5CLuJIthXtELOQxxDzEgu+r8R4e+azMybMHxi/Zdw8r9tSEZSHjxRnaYRXHYRXkWLB1Iap7eFOkw6kk2OOL/z7Fcy0ELXxG0IBf5J+vjuD5khZp95ht0656sEw7qqSwHGxPc14mX1pnuToezwfJ9q7YEVK7AhSFuTPOc+Eo01ZGtBZ2NkhqXGxvjv1YStCFblxGiiOQn6kiPKCkycwmCuKPnB5yKgNh6pqudHIbVXGnnsw1m4u3M0lm675IsZnCeV04s/4MU2a1eSfPcqLUqQjvsWdL0NA5rp69lllodJsTvKSEz8ZOT06+VzPrITkVCaliWlfBaRSZYgnbEl9TUVOaehn++/Lu8Tt+/gEsc3xzCmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxOCA+PgpzdHJlYW0KeJwzNrRQMIDDFEOuNAAd5gNSCmVuZHN0cmVhbQplbmRvYmoKMTYgMCBvYmoKPDwgL0Jhc2VGb250IC9EZWphVnVTYW5zIC9DaGFyUHJvY3MgMTcgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNzkgL08gODQgL1QgOTcgL2EgMTAwIC9kIC9lIC9mIC9nIDEwNSAvaSAxMDggL2wgL20gL24gL28gMTE0Ci9yIC9zIF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNSAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxNCAwIFIgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTQgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTcgMCBvYmoKPDwgL08gMTggMCBSIC9UIDE5IDAgUiAvYSAyMCAwIFIgL2QgMjEgMCBSIC9lIDIyIDAgUiAvZiAyMyAwIFIgL2cgMjQgMCBSCi9pIDI1IDAgUiAvbCAyNiAwIFIgL20gMjcgMCBSIC9uIDI4IDAgUiAvbyAyOSAwIFIgL3IgMzAgMCBSIC9zIDMxIDAgUgovc3BhY2UgMzIgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNiAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAxNjAgKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+PLy8vHx8fDw8O7u7uzs7Orq6unp6efn5+Xl5ePj4+Hh4eDg4N/f397e3t3d3dvb29ra2tjY2NTU1NHR0c/Pz87Ozs3NzcvLy8nJycbGxsLCwsDAwL+/v76+vru7u7q6urm5uba2trW1tbKysrGxsa2traqqqqenp6ampqOjo6KioqGhoZ+fn5ubm5qampmZmZeXl4+Pj46OjoyMjIuLi4aGhoWFhYODg4KCgoGBgX9/f35+fnt7e3l5eXd3d3Nzc3Jycm5ubm1tbWtra2pqamRkZGBgYFxcXFxcXFdXV1VVVVRUVFNTU09PT05OTk1NTUtLS0lJSUhISEdHR0ZGRkVFRURERENDQ0JCQkFBQT8/Pz4+Pj09PTw8PDs7Ozo6Ojk5OTg4ODc3NzY2NjU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLi0tLSwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx4eHh0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExISEhERERAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgMjI0IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCAxMTYgL0xlbmd0aCAzMyAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCAyMjQgPj4Kc3RyZWFtCnic7ZzpWxtVFIcRbVFLqdqKFVsVK1rU4lK1de2i2LrUFQRBilUUcUGUigsgFAgKVQQRKjWJ+VN9f8yJuSGE0gX7JDnvl5m5M8mdd57c7dw7KTta5JRd7RvYaFyw0HHBQqdkBFP/B0czmX0LwZnJyUltJqC3t1e7PT09n4Cd7wDb/RTi8bgdfQR9wO7fYElBZi54JXHBYhHs6uoyh3/ATv4E38Hs7OzPQMp5+BOyv+QvYDMAyWRyHHR5LBZrAV1Aigu64OUJsrsElqqCx61q9yuwVFSlMmmlM6ITbPcMsDkLOuSxpL+LzZfgghuACxaR4OqousBqARKJRDo1UkmNjIysuLy7uzu9u7i4qM1zwOY0uOAG4IJFJDgPTU1NqkOsgQ6qHWGCv8Pc3Ny4td7wI9ADsKMPQd9FLTUKuZm54JWkZAQzZUf3H1ygrql5zszMpAWsv0q50kYPYmxszD7BwCmuJpH271ewVPVrXXADcMFiEWT3N7BU6ofk1NSUbspSfoHcTw8NDWnzGbA5B1REupThZRf22j1tTyF11WtRF8z9tAteguDbsHfv3sfh4MGDH8MwXAlBGu+51HIXcvELYPdNeAIOHDhwD5w4ceIbUJ6ctGvsUytIBP3W1PLQUNEcFywJwVvhmixugNo1aAAFDNYUpPwp1MAmBrNRuKHMqKqquh/q6up09ADs37//bqivr38MOFG3Z8+ep6GiokIRcpVKG1IZiom7oAsWg+BJOH78+HvA5lG4GfC8BQLtTZs23QR29AysKdjX16fpotbWVqUqCN/f36/dR4C8XobGxsYHQZI7d+7UZtu2bemHUFNTUw3svQb6JG2gaha1qXRJNUJ0wZIQzEUFprOzU99wMgM/eQ1WKisrJZjONK8gu22QWo5ff2cnLUV02fRTKpphGh0dfQteiVB+jKEeBgRfhyDeHQTCXdAFi1MwL+9AeXn5HbAiDrGqoPEDKCqRfZ0mmWggtaqgB0gZywQpNOfLA9WT3Ldv3zTYiWAEpq91QRcsJUE6lnQhqyj08sx7WSC4aNNBqSgOmD1zNAULCwsKIiokiGc7WBxDPd3du3fzMMvpDyhFS4PY/AE6PHXqVJCZC7qgOHLkiMrEli1bVqxBW10wHo9b2dFUZ3CBIhOYqzTR/mksZUr2ILR5A7Zu3Xo7WPRb0W1bi2A0gQu6YKkIKjywefNmCdpKzQsKjo+Pa5FPIpFIVwwB3GoQHVTokCpHkUG6v6+CDQ1tkagC3Hapvut8NPerltgFXbBUBJ8H5O6DzFLqtQWzU5uB29Qiia8hN+YYTSdpTfZLUAENDQ02ZjyfWXAhV1s/q86sC7pgKlo/fSeQZ3ewcGwdghr/cBea9VSwg2KmciNJCyYODw8rWqndiYmJF2FwcDAdNrQyauMj64uunpkLumBRC74AagDr6+vXc3luJaNJJm58MPs6dUkRtGldrRdVt/TQoUOSuxcyl/Zb0D+VCWtYigu6YEtLy3VwI6xaCPILDgwMKCoRnGlra5NrEJjOdMEEBbPVyt+zYKmrLqdZkZkL5scF8+VZCIIzUF1drQpGs7Lr/FRuJROsxR6ytWiZCGL69TryugsYI2nuWIHE7JU/Nro6l/1MXNAFi1cwmUwqQ+Rug2Cd53oFsdL8j6VaqCF9EV1SWXV0dNjLrKna2tqaiPSrEZxUnbS0tBSMB8X7wLfL1QVLWJC+oy35eRcu4oNpQW7K1ngqTB1MbtprncFdax1aNG9VZv8g8DlY4eTD9sK2UHm2PwvQg3BBFyxOQd3N9u3bZXfs2LGLkAsFGeudzZ4OCmhubg66pFohityTQDtJGzkyERE8BLtc791ptsrCjy7ogsUpaIF6kW9l9oUFo78rEIr4qaG3Jl3gYYPFh8AGgTrBYzmTWZpt/2Oi2V35xmIx9R16rSZKXXJf1AULWVATOtfDZQtSZixWqDULKjdRcHvK7vx7OHz4cDqOvWPHDp2wcLbaOCt/NHqafg0WqeVm5oIuWESCjWByGghmRnQXKzg/Px/8W5Uqhqy/xVlubMlLcgpMVlZW2gm9FqFqhSpFG44+AJ2h9gn6tFpl6oIlLLhr1668q5bXJdje3q5DKzpq3FDW+8WaC41e+dQ46Vp4CrK/RB2zoNlM/ffastByVZuTckEXLD7By+ZodmaZydkUYsE/bAY9SsHYx/b0Dri6rpYyPT2tR6L13zmzxC64EbigHW2EYNHigoWOCxY6Lljo/AuybeG+CmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKMTY4MgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMzQgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDkxNjE0NDA0NyswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjQuMykgPj4KZW5kb2JqCnhyZWYKMCAzNQowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwOTIwMSAwMDAwMCBuIAowMDAwMDA2NTQ4IDAwMDAwIG4gCjAwMDAwMDY1ODAgMDAwMDAgbiAKMDAwMDAwNjY3OSAwMDAwMCBuIAowMDAwMDA2NzAwIDAwMDAwIG4gCjAwMDAwMDY3MjEgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk3IDAwMDAwIG4gCjAwMDAwMDA4MjMgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwODAzIDAwMDAwIG4gCjAwMDAwMDY3NTMgMDAwMDAgbiAKMDAwMDAwNTMxOSAwMDAwMCBuIAowMDAwMDA1MTE5IDAwMDAwIG4gCjAwMDAwMDQ3NDkgMDAwMDAgbiAKMDAwMDAwNjM3MiAwMDAwMCBuIAowMDAwMDAwODQzIDAwMDAwIG4gCjAwMDAwMDExMzEgMDAwMDAgbiAKMDAwMDAwMTI2OSAwMDAwMCBuIAowMDAwMDAxNjQ5IDAwMDAwIG4gCjAwMDAwMDE5NTMgMDAwMDAgbiAKMDAwMDAwMjI3NSAwMDAwMCBuIAowMDAwMDAyNDg0IDAwMDAwIG4gCjAwMDAwMDI4OTggMDAwMDAgbiAKMDAwMDAwMzA0MiAwMDAwMCBuIAowMDAwMDAzMTYxIDAwMDAwIG4gCjAwMDAwMDM0OTIgMDAwMDAgbiAKMDAwMDAwMzcyOCAwMDAwMCBuIAowMDAwMDA0MDE5IDAwMDAwIG4gCjAwMDAwMDQyNTIgMDAwMDAgbiAKMDAwMDAwNDY1OSAwMDAwMCBuIAowMDAwMDA5MTgwIDAwMDAwIG4gCjAwMDAwMDkyNjEgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAzNCAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMzUgPj4Kc3RhcnR4cmVmCjk0MTgKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-09-16T14:40:47.862252\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.4.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Score original image: 0.03\n", "Score transformed image: -0.07\n"]}], "source": ["img_noisy = exmp_img + torch.randn_like(exmp_img) * 0.3\n", "img_noisy.clamp_(min=-1.0, max=1.0)\n", "compare_images(exmp_img, img_noisy)"]}, {"cell_type": "markdown", "id": "2544f964", "metadata": {"papermill": {"duration": 0.03459, "end_time": "2021-09-16T12:40:47.989386", "exception": false, "start_time": "2021-09-16T12:40:47.954796", "status": "completed"}, "tags": []}, "source": ["We can see that the score considerably drops.\n", "Hence, the model can detect random Gaussian noise on the image.\n", "This is also to expect as initially, the \"fake\" samples are pure noise images.\n", "\n", "Next, we flip an image and check how this influences the score:"]}, {"cell_type": "code", "execution_count": 21, "id": "cc6e4e18", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:48.060789Z", "iopub.status.busy": "2021-09-16T12:40:48.060321Z", "iopub.status.idle": "2021-09-16T12:40:48.147462Z", "shell.execute_reply": "2021-09-16T12:40:48.147064Z"}, "papermill": {"duration": 0.12397, "end_time": "2021-09-16T12:40:48.147562", "exception": false, "start_time": "2021-09-16T12:40:48.023592", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torchvision/utils.py:50: UserWarning: range will be deprecated, please use value_range instead.\n", " warnings.warn(warning)\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDI0MS4xIDE0Ni4yNzE4NzUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicjVJNb4JQELzvr5ijHvp4+/g+amxNvTS0JD00PRhEihEMmNS/3+UrEWkbD5B9w+4wM28ZB7IWjOwMjYM8FzDWsFbpd56kr+slkjNpwQsyDiuW6thX7HjK+Bz4rkB6fPwiKklYpX0thBmxVj6MrYKuQ9hsW4VXyLFH2A6VHqB2aoQI8Z4q3NAZKQ2YXXnXKd5RwlqYxhSLKRZTemKqkikHjTVm75YwKWA9M1YnRBShGti0OGoYtQp6TkHI85Q78tYBWtmDalpKGBeqqMnvQf4IEyrf9QM2LkKJrS2SgpYxrCeJViPet7HHO/rA7KXOs7zcHpEX2yyd4xPxhh5jiqhVQuw7NxJ65F8NbDzVfrxDQTxvL7jpntXb8rw/1YL43fysSHdTaQabbqfasMbhT9ZhetP0NtmR4rcdkb47lmvouhr9iy2iHyt1pwQKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagozMzEKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTUgPj4Kc3RyZWFtCnicNVBLrgMxCNvnFL5ApUASCOeZqnqb3n/7MKiLEdbgH/HrmDiGlyz4EvhWvGWs2DBTfMdSLaR2YOtAdeFcxTPkCo5eiE3stOBctrlJpK4gQyJKI9tyQ5dQtCk6JX9vmlu6KbcnTZpu08rA1MuQsyOIGEoGS1DTtWjCou2p+J3yjL86ixd+xw4rdNzh01MR9T3DZz6IS73G9qjZmUS6L8iQ05pLCU002dHvyBTOPDekkM4gQVJcgmtlkP3pl6MDEjAxtyxAdleinCVpx9K/M3jS5x9hXFSNCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA2NiA+PgpzdHJlYW0KeJwzMzRUMFDQNQISZoYmCuZGlgophlxAPoiVywUTywGzzEzMgCxjU1MklgGQNjI1g9MQGaABcAZEfwZXGgBSaxTACmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzMSA+PgpzdHJlYW0KeJw1TzmSBCEMy3mFPjBVGNtAv6entjbY+X+6kplOkPAhydMTHZl4mSMjsGbH21pkIGbgU0zFv/a0DxOq9+AeIpSLC2GGkXDWrONuno4X/3aVz1gH7zb4illeENjCTNZXFmcu2wVjaZzEOclujF0TsY11radTWEcwoQyEdLbDlCBzVKT0yY4y5ug4kSeei+/22yx2OX4O6ws2jSEV5/gqeoI2g6Lsee8CGnJB/13d+B5Fu+glIBsJFtZRYu6c5YRfvXZ0HrUoEnNCmkEuEyHN6SqmEJpQrLOjoFJRcKk+p+isn3/lX1wtCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicPVA7jkQhDOs5hS/wJPIjcB5Gqy1m79+uA5opUEx+tjMk0BGBRwwxlK/jJa2groG/i0LxbuLrg8Igq0NSIM56D4h07KY2kRM6HZwzP2E3Y47ARTEGnOl0pj0HJjn7wgqEcxtl7FZIJ4mqIo7qM44pnip7n3gWLO3INlsnkj3kIOFSUonJpZ+Uyj9typQKOmbRBCwSueBkE004y7tJUowZlDLqHqZ2In2sPMijOuhkTc6sI5nZ00/bmfgccLdf2mROlcd0Hsz4nLTOgzkVuvfjiTYHTY3a6Oz3E2kqL1K7HVqdfnUSld0Y5xgSl2d/Gd9k//kH/odaIgplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTM2ID4+CnN0cmVhbQp4nE2PQQ4DMQgD73mFn0AgQHjPVlUP2/9fS9h20wseyYBsUQaBJYd4hxvh0dsP30U2FWfjnF9SKWIhmE9wnzBTHI0pd/Jjj4BxlGosp2h4XkvOTcMXLXcTLaWtl5MZb7jul/dHlW2RDUXPLQtC12yS+TKBB3wYmEd142mlx932bK/2/ADObDRJCmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDEgPj4Kc3RyZWFtCnicRVJLbkQxCNu/U3CBSOGXkPO0qrqY3n9bm0zVzeAJYGx4y1OmZMqwuSUjJNeUT30iQ6ym/DRyJCKm+EkJBXaVj8drS6yN7JGoFJ/a8eOx9Eam2RVa9e7Rpc2iUc3KyDnIEKGeFbqye9QO2fB6XEi675TNIRzL/1CBLGXdcgolQVvQd+wR3w8droIrgmGway6D7WUy1P/6hxZc7333YscugBas577BDgCopxO0BcgZ2u42KWgAVbqLScKj8npudqJso1Xp+RwAMw4wcsCIJVsdvtHeAJZ9XehFjYr9K0BRWUD8yNV2wd4xyUhwFuYGjr1wPMWZcEs4xgJAir3iGHrwJdjmL1euiJrwCXW6ZC+8wp7a5udCkwh3rQAOXmTDraujqJbt6TyC9mdFckaM1Is4OiGSWtI5guLSoB5a41w3seJtI7G5V9/uH+GcL1z26xdL7ITECmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MiA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlxAvqmJuUIuF0gMxMoBswyAtCWcgohngJggbRDFIBZEsZmJGUQdnAGRy+BKAwAl2xbJCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0NyA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlyWEFYuF0wsB8wC0ZZwCiKewZUGALlnDScKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1OCA+PgpzdHJlYW0KeJxFkUtyBCAIRPeegiOA/OQ8k0plMbn/Ng3OZDZ2l6j9hEojphIs5xR5MH3J8s1ktul3OVY7GwUURSiYyVXosQKrO1PEmWuJautjZeS40zsGxRvOXTmpZHGjjHVUdSpwTM+V9VHd+XZZlH1HDmUK2KxzHGzgym3DGCdGm63uDveJIE8nU0fF7SDZ8AcnjX2VqytwnWz20UswDgT9QhOY5ItA6wyBxs1T9OQS7OPjdueBYG95EUjZEMiRIRgdgnadXP/i1vm9/3GGO8+1Ga4c7+J3mNZ2x19ikhVzAYvcKajnay5a1xk63pMzx+Sm+4bOuWCXu4NM7/k/1s/6/gMeKWb6CmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjMgPj4Kc3RyZWFtCnicRZA7EgMhDEN7TqEj+CMDPs9mMik2929j2GxSwNNYIIO7E4LU2oKJ6IKHtiXdBe+tBGdj/Ok2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlDcPVf9b9i3TmbiYHJyh0IzepT3Pk2O6K6usn+pMfcrNd+K+xVYWlZS8sJt527ZkAJ3FM52qs9Px8KOvYKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxOCA+PgpzdHJlYW0KeJw9ULmNBDEMy12FGljAeu2pZxaLS6b/9Ej59iLRFkVSKjWZkikvdZQlWVPeOnyWxA55huVuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rnKPn16LGz7D8UMUkZIO5jX/WP3ycw2vU48nkW5vvuJenKkOAxEckpq8I11YsS4SEWk1QU3PwFotgLu3Xv4btCO6DED2icRxmlKOob9rcKXPL+UnU9gKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM0ID4+CnN0cmVhbQp4nC1SS3LFIAzbcwpdoDP4B+Q86XS6eL3/tpKTRUYOYPQx5YaJSnxZILej1sS3jcxAheGvq8yFz0jbyDqIy5CLuJIthXtELOQxxDzEgu+r8R4e+azMybMHxi/Zdw8r9tSEZSHjxRnaYRXHYRXkWLB1Iap7eFOkw6kk2OOL/z7Fcy0ELXxG0IBf5J+vjuD5khZp95ht0656sEw7qqSwHGxPc14mX1pnuToezwfJ9q7YEVK7AhSFuTPOc+Eo01ZGtBZ2NkhqXGxvjv1YStCFblxGiiOQn6kiPKCkycwmCuKPnB5yKgNh6pqudHIbVXGnnsw1m4u3M0lm675IsZnCeV04s/4MU2a1eSfPcqLUqQjvsWdL0NA5rp69lllodJsTvKSEz8ZOT06+VzPrITkVCaliWlfBaRSZYgnbEl9TUVOaehn++/Lu8Tt+/gEsc3xzCmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxOCA+PgpzdHJlYW0KeJwzNrRQMIDDFEOuNAAd5gNSCmVuZHN0cmVhbQplbmRvYmoKMTYgMCBvYmoKPDwgL0Jhc2VGb250IC9EZWphVnVTYW5zIC9DaGFyUHJvY3MgMTcgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNzkgL08gODQgL1QgOTcgL2EgMTAwIC9kIC9lIC9mIC9nIDEwNSAvaSAxMDggL2wgL20gL24gL28gMTE0Ci9yIC9zIF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNSAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxNCAwIFIgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTQgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTcgMCBvYmoKPDwgL08gMTggMCBSIC9UIDE5IDAgUiAvYSAyMCAwIFIgL2QgMjEgMCBSIC9lIDIyIDAgUiAvZiAyMyAwIFIgL2cgMjQgMCBSCi9pIDI1IDAgUiAvbCAyNiAwIFIgL20gMjcgMCBSIC9uIDI4IDAgUiAvbyAyOSAwIFIgL3IgMzAgMCBSIC9zIDMxIDAgUgovc3BhY2UgMzIgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNiAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiA3MSAo/////v7+/f39+/v7+vr6+fn5+Pj48vLy8fHx8PDw7u7u7Ozs6enp5eXl4+Pj4eHh4ODg3t7e3d3d29vb0dHRz8/Pzc3Ny8vLxsbGu7u7ubm5tra2qqqqpqamo6Ojn5+fl5eXjIyMhYWFgYGBf39/fn5+eXl5c3NzcnJyampqVFRUU1NTTU1NS0tLSEhIQ0NDQkJCPT09PDw8Ozs7Ojo6OTk5ODg4MzMzKysrJycnJSUlIyMjIiIiHh4eFRUVFBQUEhISEREREBAQXHJcclxyCAgIBAQEAgICAAAAKV0KL0RlY29kZVBhcm1zIDw8IC9Db2xvcnMgMSAvQ29sdW1ucyAyMjQgL1ByZWRpY3RvciAxMCA+PgovRmlsdGVyIC9GbGF0ZURlY29kZSAvSGVpZ2h0IDExNiAvTGVuZ3RoIDMzIDAgUiAvU3VidHlwZSAvSW1hZ2UKL1R5cGUgL1hPYmplY3QgL1dpZHRoIDIyNCA+PgpzdHJlYW0KeJztnGl30kAYhaECKrggigu4RKmCCxYUQVCpG/T//yPvPcZTMJNkJh3EhPt86ekH5r5Pzmlne0PpuOCU9l3ArpFg3pFg3jkYwbN/wfEewiS4g0wJ+gzLieA30G63y+AFcAiToE8kaGY+n98G5d9IcDtMgj6RoJnZbBbKXQbvgEOYBH0iQTMHJPgMOIZJ0CcSNNPv92nXbDa/AMcwCfpEgmYKLfgGhP9hFotFhjAJ+kSCUXYs+Ap0Op0nIAiC9+ATcIkxZdoKfgbhRvAeWK/XGcIkWGzBW6C8xRVwP4GHAGvHmQ/BR4CZrVbrK3Cxk6AECyJ4AgaDwVuAH4/BDYDMm2BDu1KpXAfhb0/BhQWn0+lVcAkMh0NHOQkejGCU72A0Gv0AJ+dMJpNT0Gg0KPgSJGVahXW7XY71HNjWZgyT4F9IMO+CsbwGR0dHdwCvY5MyU8M+gFqtdhf8BNkKkqAjEjzLs+ByubwGSqUSPdMyk8NWq9UDUHY7qI8Nk6AVhRfs9XqctOr1+keQlpkcNh6POVa1WsXsOslekwRdkOBWZs4EWQaqYVGozSYzPoyPB0+JY+GZZa9pI0yCqUgwkpkzQU7HKIiTM+Zom8z4MC4TsFrgogFrh+w1bYRJMJVCC3Izw00N9jbc4lhmmsO4yeJmC3uuxAWtLRK0QYLGzBwJ8lCPk1a323XINIfxsJFjNRoNHkFidt04l+Qx5Wg04qGlbW0StKHYgsPhkHc/vAOaTqcOmeYw3kmF11O8qapUKhvXV+FtFu+2eMd1fuNFewlK8AAF2RTQarWYzVYBy08lCrJ1gS0MSS0ObIEob8E2CQlK8NAE1+s1m+MwPFvl2DLnQ9AGtgKyJTAIAjYIdjodtgsmhknQQOEFF4tF+BfAhmOHD15Y0AUJxiPBlMz/WpAv1zSbTdr1+/1smRL0ggTNFF4wPKgncZ3ZqZkS9IIEo/AVfb6qL8GUTAl6QYJR+FU1oRw3gvP5PFumBL0gwSh/BNvtdmzXsk2mBL0gwSiFF/SVKUGfYRLcQeZeBAuLBPOOBPOOBPPOLyx+TD4KZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago3OTQKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjM0IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTA5MTYxNDQwNDgrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My40LjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My40LjMpID4+CmVuZG9iagp4cmVmCjAgMzUKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDgwMzIgMDAwMDAgbiAKMDAwMDAwNjU0OCAwMDAwMCBuIAowMDAwMDA2NTgwIDAwMDAwIG4gCjAwMDAwMDY2NzkgMDAwMDAgbiAKMDAwMDAwNjcwMCAwMDAwMCBuIAowMDAwMDA2NzIxIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM5NyAwMDAwMCBuIAowMDAwMDAwODIzIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDgwMyAwMDAwMCBuIAowMDAwMDA2NzUzIDAwMDAwIG4gCjAwMDAwMDUzMTkgMDAwMDAgbiAKMDAwMDAwNTExOSAwMDAwMCBuIAowMDAwMDA0NzQ5IDAwMDAwIG4gCjAwMDAwMDYzNzIgMDAwMDAgbiAKMDAwMDAwMDg0MyAwMDAwMCBuIAowMDAwMDAxMTMxIDAwMDAwIG4gCjAwMDAwMDEyNjkgMDAwMDAgbiAKMDAwMDAwMTY0OSAwMDAwMCBuIAowMDAwMDAxOTUzIDAwMDAwIG4gCjAwMDAwMDIyNzUgMDAwMDAgbiAKMDAwMDAwMjQ4NCAwMDAwMCBuIAowMDAwMDAyODk4IDAwMDAwIG4gCjAwMDAwMDMwNDIgMDAwMDAgbiAKMDAwMDAwMzE2MSAwMDAwMCBuIAowMDAwMDAzNDkyIDAwMDAwIG4gCjAwMDAwMDM3MjggMDAwMDAgbiAKMDAwMDAwNDAxOSAwMDAwMCBuIAowMDAwMDA0MjUyIDAwMDAwIG4gCjAwMDAwMDQ2NTkgMDAwMDAgbiAKMDAwMDAwODAxMiAwMDAwMCBuIAowMDAwMDA4MDkyIDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMzQgMCBSIC9Sb290IDEgMCBSIC9TaXplIDM1ID4+CnN0YXJ0eHJlZgo4MjQ5CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-09-16T14:40:48.094457\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.4.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Score original image: 0.03\n", "Score transformed image: -0.00\n"]}], "source": ["img_flipped = exmp_img.flip(dims=(1, 2))\n", "compare_images(exmp_img, img_flipped)"]}, {"cell_type": "markdown", "id": "d6f1c7b3", "metadata": {"papermill": {"duration": 0.036128, "end_time": "2021-09-16T12:40:48.219851", "exception": false, "start_time": "2021-09-16T12:40:48.183723", "status": "completed"}, "tags": []}, "source": ["If the digit can only be read in this way, for example, the 7, then we can see that the score drops.\n", "However, the score only drops slightly.\n", "This is likely because of the small size of our model.\n", "Keep in mind that generative modeling is a much harder task than classification,\n", "as we do not only need to distinguish between classes but learn **all** details/characteristics of the digits.\n", "With a deeper model, this could eventually be captured better (but at the cost of greater training instability).\n", "\n", "Finally, we check what happens if we reduce the digit significantly in size:"]}, {"cell_type": "code", "execution_count": 22, "id": "799ea05d", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:40:48.296476Z", "iopub.status.busy": "2021-09-16T12:40:48.296010Z", "iopub.status.idle": "2021-09-16T12:40:48.386691Z", "shell.execute_reply": "2021-09-16T12:40:48.387068Z"}, "papermill": {"duration": 0.131568, "end_time": "2021-09-16T12:40:48.387188", "exception": false, "start_time": "2021-09-16T12:40:48.255620", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.9/dist-packages/torchvision/utils.py:50: UserWarning: range will be deprecated, please use value_range instead.\n", " warnings.warn(warning)\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDI0MS4xIDE0Ni4yNzE4NzUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicjVJNb4JQELzvr5ijHvp4+/g+amxNvTS0JD00PRhEihEMmNS/3+UrEWkbD5B9w+4wM28ZB7IWjOwMjYM8FzDWsFbpd56kr+slkjNpwQsyDiuW6thX7HjK+Bz4rkB6fPwiKklYpX0thBmxVj6MrYKuQ9hsW4VXyLFH2A6VHqB2aoQI8Z4q3NAZKQ2YXXnXKd5RwlqYxhSLKRZTemKqkikHjTVm75YwKWA9M1YnRBShGti0OGoYtQp6TkHI85Q78tYBWtmDalpKGBeqqMnvQf4IEyrf9QM2LkKJrS2SgpYxrCeJViPet7HHO/rA7KXOs7zcHpEX2yyd4xPxhh5jiqhVQuw7NxJ65F8NbDzVfrxDQTxvL7jpntXb8rw/1YL43fysSHdTaQabbqfasMbhT9ZhetP0NtmR4rcdkb47lmvouhr9iy2iHyt1pwQKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagozMzEKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTUgPj4Kc3RyZWFtCnicNVBLrgMxCNvnFL5ApUASCOeZqnqb3n/7MKiLEdbgH/HrmDiGlyz4EvhWvGWs2DBTfMdSLaR2YOtAdeFcxTPkCo5eiE3stOBctrlJpK4gQyJKI9tyQ5dQtCk6JX9vmlu6KbcnTZpu08rA1MuQsyOIGEoGS1DTtWjCou2p+J3yjL86ixd+xw4rdNzh01MR9T3DZz6IS73G9qjZmUS6L8iQ05pLCU002dHvyBTOPDekkM4gQVJcgmtlkP3pl6MDEjAxtyxAdleinCVpx9K/M3jS5x9hXFSNCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA2NiA+PgpzdHJlYW0KeJwzMzRUMFDQNQISZoYmCuZGlgophlxAPoiVywUTywGzzEzMgCxjU1MklgGQNjI1g9MQGaABcAZEfwZXGgBSaxTACmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzMSA+PgpzdHJlYW0KeJw1TzmSBCEMy3mFPjBVGNtAv6entjbY+X+6kplOkPAhydMTHZl4mSMjsGbH21pkIGbgU0zFv/a0DxOq9+AeIpSLC2GGkXDWrONuno4X/3aVz1gH7zb4illeENjCTNZXFmcu2wVjaZzEOclujF0TsY11radTWEcwoQyEdLbDlCBzVKT0yY4y5ug4kSeei+/22yx2OX4O6ws2jSEV5/gqeoI2g6Lsee8CGnJB/13d+B5Fu+glIBsJFtZRYu6c5YRfvXZ0HrUoEnNCmkEuEyHN6SqmEJpQrLOjoFJRcKk+p+isn3/lX1wtCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicPVA7jkQhDOs5hS/wJPIjcB5Gqy1m79+uA5opUEx+tjMk0BGBRwwxlK/jJa2groG/i0LxbuLrg8Igq0NSIM56D4h07KY2kRM6HZwzP2E3Y47ARTEGnOl0pj0HJjn7wgqEcxtl7FZIJ4mqIo7qM44pnip7n3gWLO3INlsnkj3kIOFSUonJpZ+Uyj9typQKOmbRBCwSueBkE004y7tJUowZlDLqHqZ2In2sPMijOuhkTc6sI5nZ00/bmfgccLdf2mROlcd0Hsz4nLTOgzkVuvfjiTYHTY3a6Oz3E2kqL1K7HVqdfnUSld0Y5xgSl2d/Gd9k//kH/odaIgplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTM2ID4+CnN0cmVhbQp4nE2PQQ4DMQgD73mFn0AgQHjPVlUP2/9fS9h20wseyYBsUQaBJYd4hxvh0dsP30U2FWfjnF9SKWIhmE9wnzBTHI0pd/Jjj4BxlGosp2h4XkvOTcMXLXcTLaWtl5MZb7jul/dHlW2RDUXPLQtC12yS+TKBB3wYmEd142mlx932bK/2/ADObDRJCmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDEgPj4Kc3RyZWFtCnicRVJLbkQxCNu/U3CBSOGXkPO0qrqY3n9bm0zVzeAJYGx4y1OmZMqwuSUjJNeUT30iQ6ym/DRyJCKm+EkJBXaVj8drS6yN7JGoFJ/a8eOx9Eam2RVa9e7Rpc2iUc3KyDnIEKGeFbqye9QO2fB6XEi675TNIRzL/1CBLGXdcgolQVvQd+wR3w8droIrgmGway6D7WUy1P/6hxZc7333YscugBas577BDgCopxO0BcgZ2u42KWgAVbqLScKj8npudqJso1Xp+RwAMw4wcsCIJVsdvtHeAJZ9XehFjYr9K0BRWUD8yNV2wd4xyUhwFuYGjr1wPMWZcEs4xgJAir3iGHrwJdjmL1euiJrwCXW6ZC+8wp7a5udCkwh3rQAOXmTDraujqJbt6TyC9mdFckaM1Is4OiGSWtI5guLSoB5a41w3seJtI7G5V9/uH+GcL1z26xdL7ITECmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MiA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlxAvqmJuUIuF0gMxMoBswyAtCWcgohngJggbRDFIBZEsZmJGUQdnAGRy+BKAwAl2xbJCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0NyA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlyWEFYuF0wsB8wC0ZZwCiKewZUGALlnDScKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1OCA+PgpzdHJlYW0KeJxFkUtyBCAIRPeegiOA/OQ8k0plMbn/Ng3OZDZ2l6j9hEojphIs5xR5MH3J8s1ktul3OVY7GwUURSiYyVXosQKrO1PEmWuJautjZeS40zsGxRvOXTmpZHGjjHVUdSpwTM+V9VHd+XZZlH1HDmUK2KxzHGzgym3DGCdGm63uDveJIE8nU0fF7SDZ8AcnjX2VqytwnWz20UswDgT9QhOY5ItA6wyBxs1T9OQS7OPjdueBYG95EUjZEMiRIRgdgnadXP/i1vm9/3GGO8+1Ga4c7+J3mNZ2x19ikhVzAYvcKajnay5a1xk63pMzx+Sm+4bOuWCXu4NM7/k/1s/6/gMeKWb6CmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjMgPj4Kc3RyZWFtCnicRZA7EgMhDEN7TqEj+CMDPs9mMik2929j2GxSwNNYIIO7E4LU2oKJ6IKHtiXdBe+tBGdj/Ok2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlDcPVf9b9i3TmbiYHJyh0IzepT3Pk2O6K6usn+pMfcrNd+K+xVYWlZS8sJt527ZkAJ3FM52qs9Px8KOvYKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxOCA+PgpzdHJlYW0KeJw9ULmNBDEMy12FGljAeu2pZxaLS6b/9Ej59iLRFkVSKjWZkikvdZQlWVPeOnyWxA55huVuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rnKPn16LGz7D8UMUkZIO5jX/WP3ycw2vU48nkW5vvuJenKkOAxEckpq8I11YsS4SEWk1QU3PwFotgLu3Xv4btCO6DED2icRxmlKOob9rcKXPL+UnU9gKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM0ID4+CnN0cmVhbQp4nC1SS3LFIAzbcwpdoDP4B+Q86XS6eL3/tpKTRUYOYPQx5YaJSnxZILej1sS3jcxAheGvq8yFz0jbyDqIy5CLuJIthXtELOQxxDzEgu+r8R4e+azMybMHxi/Zdw8r9tSEZSHjxRnaYRXHYRXkWLB1Iap7eFOkw6kk2OOL/z7Fcy0ELXxG0IBf5J+vjuD5khZp95ht0656sEw7qqSwHGxPc14mX1pnuToezwfJ9q7YEVK7AhSFuTPOc+Eo01ZGtBZ2NkhqXGxvjv1YStCFblxGiiOQn6kiPKCkycwmCuKPnB5yKgNh6pqudHIbVXGnnsw1m4u3M0lm675IsZnCeV04s/4MU2a1eSfPcqLUqQjvsWdL0NA5rp69lllodJsTvKSEz8ZOT06+VzPrITkVCaliWlfBaRSZYgnbEl9TUVOaehn++/Lu8Tt+/gEsc3xzCmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxOCA+PgpzdHJlYW0KeJwzNrRQMIDDFEOuNAAd5gNSCmVuZHN0cmVhbQplbmRvYmoKMTYgMCBvYmoKPDwgL0Jhc2VGb250IC9EZWphVnVTYW5zIC9DaGFyUHJvY3MgMTcgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNzkgL08gODQgL1QgOTcgL2EgMTAwIC9kIC9lIC9mIC9nIDEwNSAvaSAxMDggL2wgL20gL24gL28gMTE0Ci9yIC9zIF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNSAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxNCAwIFIgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTQgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTcgMCBvYmoKPDwgL08gMTggMCBSIC9UIDE5IDAgUiAvYSAyMCAwIFIgL2QgMjEgMCBSIC9lIDIyIDAgUiAvZiAyMyAwIFIgL2cgMjQgMCBSCi9pIDI1IDAgUiAvbCAyNiAwIFIgL20gMjcgMCBSIC9uIDI4IDAgUiAvbyAyOSAwIFIgL3IgMzAgMCBSIC9zIDMxIDAgUgovc3BhY2UgMzIgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNiAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiA3MSAo/////v7+/f39+/v7+vr6+fn5+Pj48vLy8fHx8PDw7u7u7Ozs6enp5eXl4+Pj4eHh4ODg3t7e3d3d29vb0dHRz8/Pzc3Ny8vLxsbGu7u7ubm5tra2qqqqpqamo6Ojn5+fl5eXjIyMhYWFgYGBf39/fn5+eXl5c3NzcnJyampqVFRUU1NTTU1NS0tLSEhIQ0NDQkJCPT09PDw8Ozs7Ojo6OTk5ODg4MzMzKysrJycnJSUlIyMjIiIiHh4eFRUVFBQUEhISEREREBAQXHJcclxyCAgIBAQEAgICAAAAKV0KL0RlY29kZVBhcm1zIDw8IC9Db2xvcnMgMSAvQ29sdW1ucyAyMjQgL1ByZWRpY3RvciAxMCA+PgovRmlsdGVyIC9GbGF0ZURlY29kZSAvSGVpZ2h0IDExNiAvTGVuZ3RoIDMzIDAgUiAvU3VidHlwZSAvSW1hZ2UKL1R5cGUgL1hPYmplY3QgL1dpZHRoIDIyNCA+PgpzdHJlYW0KeJzt2mtz0lAUheGIgAqoiOIFvFBpC16wQZEKKngp+P//kWuPdJo2OUnOsI8lh/V86fQDXXmZKYGE4NBzwXUfgGsMLDoGFt3eBP75Hw6vYYyBDjYZqDnGQAebDNQcY6CDTQZqjjHQwSYDNccY6GCTgZpjDHSwyUDNMQY62GSg5hgDHWwyUHNszwPfQqfTeQW9Xu8TfIVtNxmogoHiAdy45DY8TfEcTiFtk4EqGCi8DzyB4XD4AfDjJdwDdN6HSHa5XL4Lm98OIG2TgSoYmOwXhGH4G04uTCaTJdTrdQl8A2mbDFTBwGTeBxq9g1Kp9Ah+QtomA1Uw0JLfgYvFogFBEEhn1iYDVTDQRr/flxNgrVb7AlmbDFTBQBteB+J96KRSqUjgeDzOs8lAFQzMy/vA14C4Z7BarfJsMlAFA/M4g8dQrVY/Q85NBqpgYB7eBx6BnAC73a7FJgNVMDDTaDS6CXdgOp1abG4XGLkpG7/3Gh9joBkDYwoU+ANarZasvICcj9J+kQnD8Pz+1nK5lIOJjzEwGQMTFSVwvV4/Afy9h/ANcj5QOzCi0WhcuWvAQDPvA+fz+eZE9B4sHugoUK7rXf7/i4wxMAEDTYoQ+B2azabUDQYDyyNxFCgXTeIXTBiYjIFpihC4uVAvTN/MNnMQiA+lI/lEahxj4BVeB36EW7AzgXhTLAeS/G6YgXEMzLLrgcewiZMPgrPZzPKAtANxQh4Yz8YMjNubwHa7bfzWcirNQHlB+PdCcJo6xsAIBmbZ9cCtaQbKs41nOnOMgZoYaME20FsMLDoGFh0Di+4vuN8LXwplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjYxMAplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMzQgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDkxNjE0NDA0OCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjQuMykgPj4KZW5kb2JqCnhyZWYKMCAzNQowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNzg0OCAwMDAwMCBuIAowMDAwMDA2NTQ4IDAwMDAwIG4gCjAwMDAwMDY1ODAgMDAwMDAgbiAKMDAwMDAwNjY3OSAwMDAwMCBuIAowMDAwMDA2NzAwIDAwMDAwIG4gCjAwMDAwMDY3MjEgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk3IDAwMDAwIG4gCjAwMDAwMDA4MjMgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwODAzIDAwMDAwIG4gCjAwMDAwMDY3NTMgMDAwMDAgbiAKMDAwMDAwNTMxOSAwMDAwMCBuIAowMDAwMDA1MTE5IDAwMDAwIG4gCjAwMDAwMDQ3NDkgMDAwMDAgbiAKMDAwMDAwNjM3MiAwMDAwMCBuIAowMDAwMDAwODQzIDAwMDAwIG4gCjAwMDAwMDExMzEgMDAwMDAgbiAKMDAwMDAwMTI2OSAwMDAwMCBuIAowMDAwMDAxNjQ5IDAwMDAwIG4gCjAwMDAwMDE5NTMgMDAwMDAgbiAKMDAwMDAwMjI3NSAwMDAwMCBuIAowMDAwMDAyNDg0IDAwMDAwIG4gCjAwMDAwMDI4OTggMDAwMDAgbiAKMDAwMDAwMzA0MiAwMDAwMCBuIAowMDAwMDAzMTYxIDAwMDAwIG4gCjAwMDAwMDM0OTIgMDAwMDAgbiAKMDAwMDAwMzcyOCAwMDAwMCBuIAowMDAwMDA0MDE5IDAwMDAwIG4gCjAwMDAwMDQyNTIgMDAwMDAgbiAKMDAwMDAwNDY1OSAwMDAwMCBuIAowMDAwMDA3ODI4IDAwMDAwIG4gCjAwMDAwMDc5MDggMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAzNCAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMzUgPj4Kc3RhcnR4cmVmCjgwNjUKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-09-16T14:40:48.333827\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.4.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Score original image: 0.03\n", "Score transformed image: -0.02\n"]}], "source": ["img_tiny = torch.zeros_like(exmp_img) - 1\n", "img_tiny[:, exmp_img.shape[1] // 2 :, exmp_img.shape[2] // 2 :] = exmp_img[:, ::2, ::2]\n", "compare_images(exmp_img, img_tiny)"]}, {"cell_type": "markdown", "id": "abbe7583", "metadata": {"papermill": {"duration": 0.037748, "end_time": "2021-09-16T12:40:48.462802", "exception": false, "start_time": "2021-09-16T12:40:48.425054", "status": "completed"}, "tags": []}, "source": ["The score again drops but not by a large margin, although digits in the MNIST dataset usually are much larger.\n", "\n", "Overall, we can conclude that our model is good for detecting Gaussian noise and smaller transformations to existing digits.\n", "Nonetheless, to obtain a very good out-of-distribution model, we would need to train deeper models and for more iterations."]}, {"cell_type": "markdown", "id": "714313c8", "metadata": {"papermill": {"duration": 0.037582, "end_time": "2021-09-16T12:40:48.538109", "exception": false, "start_time": "2021-09-16T12:40:48.500527", "status": "completed"}, "tags": []}, "source": ["### Instability\n", "\n", "Finally, we should discuss the possible instabilities of energy-based models,\n", "in particular for the example of image generation that we have implemented in this notebook.\n", "In the process of hyperparameter search for this notebook, there have been several models that diverged.\n", "Divergence in energy-based models means that the models assign a high probability to examples of the training set which is a good thing.\n", "However, at the same time, the sampling algorithm fails and only generates noise images that obtain minimal probability scores.\n", "This happens because the model has created many local maxima in which the generated noise images fall.\n", "The energy surface over which we calculate the gradients to reach data points with high probability has \"diverged\" and is not useful for our MCMC sampling.\n", "\n", "Besides finding the optimal hyperparameters, a common trick in energy-based models is to reload stable checkpoints.\n", "If we detect that the model is diverging, we stop the training, load the model from one epoch ago where it did not diverge yet.\n", "Afterward, we continue training and hope that with a different seed the model is not diverging again.\n", "Nevertheless, this should be considered as the \"last hope\" for stabilizing the models,\n", "and careful hyperparameter tuning is the better way to do so.\n", "Sensitive hyperparameters include `step_size`, `steps` and the noise standard deviation in the sampler,\n", "and the learning rate and feature dimensionality in the CNN model."]}, {"cell_type": "markdown", "id": "a34043a7", "metadata": {"papermill": {"duration": 0.037416, "end_time": "2021-09-16T12:40:48.621320", "exception": false, "start_time": "2021-09-16T12:40:48.583904", "status": "completed"}, "tags": []}, "source": ["## Conclusion\n", "\n", "In this tutorial, we have discussed energy-based models for generative modeling.\n", "The concept relies on the idea that any strictly positive function can be turned into a probability\n", "distribution by normalizing over the whole dataset.\n", "As this is not reasonable to calculate for high dimensional data like images,\n", "we train the model using contrastive divergence and sampling via MCMC.\n", "While the idea allows us to turn any neural network into an energy-based model,\n", "we have seen that there are multiple training tricks needed to stabilize the training.\n", "Furthermore, the training time of these models is relatively long as, during every training iteration,\n", "we need to sample new \"fake\" images, even with a sampling buffer.\n", "In the next lectures and assignment, we will see different generative models (e.g. VAE, GAN, NF)\n", "that allow us to do generative modeling more stably, but with the cost of more parameters."]}, {"cell_type": "markdown", "id": "6ca06d61", "metadata": {"papermill": {"duration": 0.038052, "end_time": "2021-09-16T12:40:48.697258", "exception": false, "start_time": "2021-09-16T12:40:48.659206", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "![Pytorch Lightning](data:image/png;base64,H4sIAAAAAAACA9ycyZKjyNqm93UV0bmlSeaprPJYCyQ0IAkECIQ2xxicScyj4MZ631fWKCKzKior8++qJvQvTphFhAbnlfujz7/BAf/ll8fPb/9jKQu6paxewiZN/vXLb49/L4mdBV8+gezTi2c3NuzmSV7Bae6BL5/stsm/vpxEQdjATQjS6fXXJ1/f8Ozq9u31x+NP//rl5eW3ENje48H0MAWN/eKGdlWD5suntvFh9q1NEmW3lwok03FZDRcV8EHjhp9ewunRl09h0xT1rwgSRE3YOp/f/tn1JFJ/dvP0n0jY3dTRqv6q0dagcvOsAVnzT4XeBGA3yVvvc018tlN7zDO7/8c9evQBjlI7AD/pFfK92qQ0vZ0Bt/m7gF7cKq/rvIqCKPvbYv8vVNPPN6V38pOlZHk2pHlbf3pJgRfZ0ytJ8uklmo4NqqgZvnyqQ5vCcNiXx8NWJdSTH906wtnTZd0KCI4F29JGk8vGjWQNkitdkNex0KZMhvSieSwLYWgWWMmZLbRsSkjZ3zVA46q+XSBpdzx9+fLpbWR1MySgDgH4u5iQt4eIX9kp6PPqVsOMC1zXxTHM8zAS81mbdHyGdh1g2w4OUOezW0/DRL6a9wwWMUGcj97O3TW3nU9owJeyrawf2up2zA/4YnfQeNs7G2YjhgpBu/uDn9BSs6Lp5RpSBSjz2Ei/6xy7rTdR1soXY8MevH4+CweEdhflVQ2zPuN5JEF5OOXitMehHOMzJAcY1+ZcluD+hOK7P3PIBJHHMAsTv0c34QRWGdJsq7TeC8nWxb1LpXgLhowcvVzjhwCctqeVme2tzVG7onVFnE5kcr0dUdwduCSPwkAfh2N3uX8Ama8OgMVxhnGAi1IO7dI4h+IkBjCc8YGH4w5n/47lMVlqt4qK5qcgvMlDVJO3ePz7EYoj8DOLlRlhzRy7QD/J5yWn007vHO8BNx4tXTQx/RZltLppz/q16xiNNCoO0xeNEfKWO/hifR7posmKw11VmoVOss0rimYoHj6+KJLItZsoz5B4mv5v3f30Ulfu38YCsi6q8iyd/ARMUIABDE18jutP//oNeZP7Oldmolg6SVlQq/rg7YyD194JbGfv617Y14BIr9XF1bt1z09tTbDDufWSQBrF84f7itMXldcBVb2G2UndQNxVdXyeWt4QjT59LAo3bLMb/M6boJ7D0bbNPIGHrZ16NPUGIk63NX4vlyil7RPnDLVXM2zoneS3F1xJJZe8mGlP7NpBI52twSuF44pLtBCXS5fL5aqk4w5V8bO7btxN8AweHci8KaugOZwkUA//C4sPmCc4F8ThGu3jkJST7rjyd+VtWbirTOnkLC+I43q3Gpv2ZLNHbx+YeOKdnJRaczWISrpQgtBIvcWBuJADvz6wq71u3Cvtg2H84Vg9x2NxgiV/RGI+C/68dKrL1ZG4U16dKx+V8pVzXexCPRshvjykGOqUV0Gie2rTD4xly+qJFMnlJdzfFobe2bvwGN7v4BKkwW6voDrP6PLf8BmvCeGUOrYJgCPvy6fPX79+2+vszAXeY7Rvjf65CX3TgFGKnOaU7T1hQlGNKWkVk4aLUQtJ7U5ZqhhIfuOt71K/kTJ5bfXUMrnHm/B2Ms6qJZ4J/kZtD/L6ILWLaKcnAiFfrANxH9fuEI/tSjz9DQfzU27ZlCM2AM4zeGqaJ8ksgN+LwYD2qCm7sZ9AskYVZXJDFxkTzCW6VBQDVTjCdiplYSWIKOvbzk9DSJCRi5ArxMpTOG7ICHHfHRtyf9gyu9Tk/XVkcztRuRZVfmYX1QwLdKd6Jo2qKq/mIPxDBXYIknB9/xnsnBCMLc0PA8Nsh9Aq8hNKM0TAXvOy4nmUIBzI3g3cyEQ4CV0Y7YDcepKEIPm01lRzGEvkdrrR7A3KVG5TnV31wp70v+HJfs7uW0VYz4P3uwxMA4xxCMd5Aj0G21xbTKd2ZqkeRa6vUvx4uS5Ky+jG+5VddnlFnakE2bcr3lkL/IVitGygUTWGyvP1uuhW/FpMjvWIMdxyleEbsQ+MxRx62VQCNtE8dG8aMPApzOOewk3BuTOaHNPwHBMUYVSCWdDu6IXH4tCIDHo3JJ9cT2gMScjLdXMv0HroFwcWIWkoEtCbJ0Zqd8CIFd8ts5LOB1Pto1kzNmuqyGkfB9RwXUTeVDIHlV2EM0n+RBUmfNqnyB8kJ/PZdkWD616Do40sqU2lmo1s2GqE93y+sP3kuD6r9L09BWcEtXr/3jn2WKICHoZS30YC69ALFiOCyzqFWD9Gj8Iqrx1rRlxx82KYR7EYYAdwDkOTz4jDduUnB9E1RmNh2QqeMSkT3BJFCM3aqIVloyp2hwsO7ZR7sPPZInDy2NwRqxpIl+1e2Jb+PUE62q54r71y2j4tDqvrDFv0QGIPU+qR5LYXZQEMEvCodeYg/IkkTNsOAygCfQJVOkZyTYRujgkNoDkq3ZZmSWNctFwxtgFtxHgIES2294RWDVUyztpoc/XBSUpqIru7OqIRGyjfNozYRMEKsUEeQ8GMuOJVdjD9yYtZHL+JwMAmfMCS7BPIGZrmQdYY4dFBg4rIsehwiZjeUNwXKy93KBFXD9K9ldzFRsNXhWjtFKBfcuPeoMdLgBKNH+PezcbTVbPRoC4uLM1ezrBH4EUNHOb5Da6BW4HmIwzyZ5owReEk5/ncE7iuqvIqJpNFjVtnn183Xe+X3DXOqTGHLGxBYmgGlOW6kyDT0A0FQ6Oj7uelJ8va9agNWNDspa253G4j2SDQpKLH806YEasfDGYzxHDH5jiMfoZfjMrDdr2+rnN2cTrEbr5NJeWg11fDUBM6bATNYvhCTYu1skYWImSWkILIMeI1oXmStzfOFY4HZWXXjIyPC5XOywxt5vBK8ziCi8i9TVH0I2zwB3qTR8Rt1sWf4RHLU7lOOAOTk7B0q4uoiqRzb04dxVR+6KTKQuba/eLSWYudZp2IqGNXTrDWK7zsJ69oEqlGWsgaNR2CyMotr6yzLXmeEZd9O0qcfBbCrxKwzaE2S3PPWHaqB1Tix4TYVC3oobzaYciKdIf2HN0bA201lNC0TWGpY7gAhQniNhiUmlj0m+IErVJqYx98DzsoI3IgofvKFp2TSwUzrNAHdtNWAHbtJMnbD3GGP5Gcqj7cQ230GVRDdq+We9/rPAlHbpoUsztroOOB9xJ96O97Mil7g5YN2SM0QVGD845EKkLaiKer0lo9umYWphnuLGSbR4217xamCKg5thglzeMkVVa081C+04FZxsU9zH5GjI74273ak5VPSpkEZC61jXsX3kRO93pFOi9RrNTw7YiE2g2J9zjXI41zrHhkLWJUYYbLprGvnjey9AKxw2t143eMvpxhlcEUQitQ5B9hjt9rwSyLcQRmu0/gyIuYf3XDpj0JK0zPi3Spaq686xRX2hyYmPe3w96uR0kO3dWOF2nfE+zbDU2j0ZExahAO0eUGkcZ2qR/GYJ1ytIzshBl2GGX142xzDZctqGZVLX9WglGS8iifpp5xoiLiyMEV66PT9LRFjXjtbijkUm2QlBV2ZUst1MU1lqsTHflMq+3D+5E9nPl86y2RqtZD7M7wPdL640nId4V19cbVjp3FsIua1+azVnDeycAowH2GdZ+RFXKccYw2K2tvIt41cc3t9WofFxdraY7a3T22Q7UgTrVfF8iFCLc3NhfEIYgSos28usZodc2XkJaqEpax1C6wqTTmr9iMmRy3aQE3+RxyXyVgn/Eo0mOfUqM0nt0fEFmrW8ijO5y6WvTZPdnIer1faI220uK69Fb9HQIDy66P1vp2wskc2oliTiYWxPT7g+oztlDt77cFpZUtnx9n2Fxit5kbvi6ZfoQL/IEcTNEeQ9nuMzJDVaXON0hacQdEqMu9mwYqEEKNuS7jlA3urpwtLdO8HTyg11VhIvdlstucVsHZb1cCvdld/EE5uiaU8/qmPGStO/HW59CMsikbCe1qHsTfVWCSwgBFcs84TVsYtb/wTmEa3ZBlMV7rUsdzRQrprbxsOfVmub3e+inEblFDZ0ZcNZo4XlfQ+TiW2MI8bPTqpJ1ZbMkOtKx7uj+ihTJj/qagqSJ3KmsT4D4O+ghz/JkmbFOUg/ko9oyV7fCKElgWRCJB9+vad/NhEEy97K17fIFKoO8bwWCEHbSVoX68jUKKqblKmKPpjr6Z9vtjWUGblrid5ODG2+uMzBpzhk1meRP5Xw+YTKtuYD9321kR5ieSMAAYzZCe/wSq66yJ177cHy718VGI3NNRJ1DaE2hiL6GsXFKbympHHlRbE5cRHhnZcuPEy20qnnnyOrbsrZMM7H7m+PTMCPZ+D45gxqpYUYE0alO4arMMVLNoficFYzbj0AT2jJjdX5IOOR1Iub5Yw7nVfDG+E5dSSzt/XzZxx9/pXkzWx3QzevTyllbY5qZfa6wsBEsX4mjosTGlZNUmZTIwF5ILuZIyY4VsGnoNMhfA3y5om4fxT1qwizEc7fvPWPlObVSWByKLURdzS/6O8jc33Yr1vnS1K3TYtrfTfViJU5w39fFarKHTIZBzU1lsriNJHGh/iBfnNRZvaBeh90oUSYowzOKYTwUcgIspef4Iv/kjvSnhsz0UEM84qxUGuSbt9WTPbomtWSH6Xj4KZ+hI3jyh8aqNUFWtRx6lMFCTfaX36OWshCVvL73rORK1cXlSeyQTT9itxw2gp2jAQcmMWPR1/B+BkKVRG8PxZ+SSquwUVd+JKh72ayVfQtZKWPDn1Cx753yXiMO5d1RLbp2Oijjs6sZrG4yrUlyyyvVK7/B7appXh0MiSLTMgSC8yoeCGZGmaJN6mn5tEz6unXk7XfcRxvhzWZgkAQ04+xkR5yiMuqrp3V2IWhe9VyDARgznq6QhfI2DzCaz1mfXWy2XxcWJETSqDDek9YpW48kh+IclsHdxT9zO/qrOtM1JO+WXcUbEqYDtpWAqT9yPgPpXNZggcdcjyKdU2uPoqaGeD1qr3jc3K9G3JunvLwQyWLSRGLcAAhU+SEN21pxoX+pqHEFNgnNnMfJdUTOcm3lsyjOkjDvKOS1Iww4Os1j6X5PCeVeavNeBUZ9wPY98xhkF3yIsixVp3TQ3mGllVFuL1NlkN/LIN1bskuesWVGLSORW/Ha7Ellvx4B0rB256WQ26RSBiqsjI1mcgB5jeSHeDHsxI95MGUveTDVKEhVOblfe7FPVPxSEGY/1PPYpF6lumWjLofgSW9A126bCEg8dkczRUDKxpCqOvrrDN0zb0753MFSE3XlNhnEOsUaWy61OJsVFPuv59aKnjjUFnoU15aT2LIusizyrow7AbeaB6lEDZnY3j+kPJWGccFiW9Z+xOq5NYYTWcHZXQ4g11p0VpAW0XXkCWjYHl/fp2gov+sqNMVtUhL0i63ImS+UoRsdhg+dHY+lBLXWKWFBa26O2bXF2L86ISG8p9RQz8raYlVz+SQgmOdp1GPCMDF0zPAmVaE7JTqQohF6R0BQqq1BXJJZ5KT3vfryt1lJ48unmBsKyk27iKqD2J+0aCIbU4I3vckJm7pJUEnpH16NiFc+wyxpUEajhxnbmpUPvdSZ+FENOyeQzViUXpX48rdBaOHJIW+Do2Pmee2x2tGgZbblWdDoufINVO0pTbe1SMZt7tRbQy+riXlG7SwS/tXZHlI8vEnPVOdQy0N0sfh14dPBxjsptE/sRKD4gfP8XsrBPojZHEM+I46GvgzE6rsIIq7YlQfKurEku5ldouyrPoeQYUrhDRIfsDt3ZI8OgVXTWPuqutO4cNK36tO54pjAMkO6OrLpV3Tw0ZsShaQxvFvXtku1ZTL8Xg1nKJz3ggieQbEDkVxsoofJtVmcx12bbcap/Lv2aH8+XeIFVtXA1kvNuu2sM3GnHKzk22kjYKh2up/juNAdlHO8ANKpDhl6b7v0rNKPiqRvbvQEPtqfEcP4K5l/VYIcCLOc/5VwY2maM5Wvk8sAzVy5VefR+2+/jo3hJDqaZefGIjP3V2J33dmny3R7BRstU9ukqk0k5xynpmq0J3+Uvd2TJqAsu2Y3dckbUeR19/fVc6uOi0CiAuwj0s4n+QBP2cMD5Dv2Mlcxdcdyt2eJ+Pm9wYkzjWMlXF6jzdisnqrQM3V8lJ7+emtsJNAFtiUTAa4YULQX84jQub2PMSbph10OJImvG2cQr3HfbOb70jcFUWQdTblN/FNM/6cE4zREkhz1jlSOTbOLsLXCZMQC5y3AysHkcHc7+rtOYPsk89sQKUGQu2QhvLvvAlkC6NG97AO7g5FU1ZGccKwoJ5cvRgO6gaMygZoadNmEFwOe3N+ZwfK8Dc65NOwz1DHtky0aXEyPi013RJsWOEBCIWTgEoTEHhWoWq62/ue6ZHZlCRBQ0xnqxIitmCAas4tUYcd01u22xQpC2Urfw15YxakUyh18065rI6XDYx22HIohn3J9gSqxx6YnEP58QQzXRMQkkJXQP53KDtnrGI9IIlLPiXUMhRTbF3b9cNpK+7mJKQGVwZ6DR2R7wlTCe906jCzrZBsQwY/Y2uV3PCiqvAjDF2j5GUc9Y8TloqsZqVpCJ3UnbSrFSskRTk+OpbDTGkq9TRrgkLz6qgQtPbpRmBDVqgeVCPAHhVtxbtoAEV8bPzUBWi0GxrfEkrOYQ66fwXude1KazuP0hAxMYTlK4/4zZ6XeuLFt2IdyV21I9mSNunsaq3NOVGl5YdtiYIsv4DnfJBlwsyaSKUPloG+KeuSw1PdKHlpT1AmLNfPAPRhGsz7ZtzKDXZh2IZt2K9aYAM8BnUEA8o9pr/GXetX5cFsuDcknLc93a9vEoy4h3iVhbFfST1gq7Vidxi79DZnvrAJlANo3yIdNzVnZgz+NZrBbKHrEgKI4Pi+WciPC64cEUFpu2huvWSeddZ/tXtSkL9FHAOs/IqF2pRaALE+iQaWzI+iI1aECsGcbxT74b9G2lys4B8hr2MiRKyoGVuYUQ25RCw3Zoo7barmu3FK9XkXQJL/6un6pDbwbLHjhBAvd2NStI/KECM6jtOP4PrvD+gNtyN1vqKrAUdL8BEB6DRq6ZPe0q1EYkXCyClFUodXRTHblDahNOes9psUgGylGCbBky6WINePsuCy1vHQ66xzNx50V/J4X+J0Ael9bV0VTpPhYSMJcFFId+n6bNLiXANQVcOmp8jq7qgTxuQc9qUm0BRhukxmR3bR+NWxJQcbPQ+kL3T0hm1ZsrD6G6rB01x4gZs11qneUuoWUMdtap+eDbk73I92vYwwBN08z3AH75tpVLZj+2enmktMVU1356+bpByJdPfeQ14RcPdJEL4Ncnn74ZURM1CfhXMUyM3fBtL5ksygIkyYP8c5EFL3bzkk5BGFQv/+d/vyiD/mi3/73ZXw58tFpHzaZ1fkPetN/tNfPWQQ+89XwC8q6PegheXlV68Pj77aNe+moCOH26n1cv4fQOPD2ZHqePW45fFtuXxylge2r4+UVz7QS8DHlbvTzuZUzq//mS5c1LMwk7eZSAqkjsBnx+gV8+brif3u818raDxWtvfvTl5wXI3t59RwC6p8m3rS7etfj8+vIrwC+f3nh+v++Mn7SRB0fug+IPt8p42M9ro0ebx+B+KPj6xUz10IT1MSl859epz/+OvHdfDUaiGMmyjyqJoCmcfXfg2zf6GOXkCZs2m6bp9OTdsdOzh4vESIYhGJpDqT/2Z3kn0PRRMzH/9XXjnV+nmfFO4duofvcFw3+9Pw/GsDROMziKMBzmcSSLPm78ADCGAQdmKZuBOZZhGQfFMAb1Ht35UU+mTwLvOvG/3j7rZ61du3oPrG7T1K6Gfyd2FYB/v/b2Z0e+fiPvDv1A0/zJJ/5HTb8/G9MfZpwHv37F/t9uR3/txK920vxHoP7BAB/z5N8PA3s3wK/u5ccHPPziu7a5E79udvXjtv8Nk+NPH9hWyQ8s5p0//f9W/o+bda/7jU05xJ+D0lu+8nf3bnrLQqbf/wsAAP//AGQCm/08bWV0YSBuYW1lPSJyZXF1ZXN0LWlkIiBjb250ZW50PSIwODAwOjZBMDI6Nzc5M0VBOkIyMjY3Qzo2MTQzMzk1RSIgZGF0YS1wamF4LXRyYW5zaWVudD0idHJ1ZSIvPjxtZXRhIG5hbWU9Imh0bWwtc2FmZS1ub25jZSIgY29udGVudD0iYWEwMDYyZTdjYjAwZGUzZTllNDBmODY5YzBhNzAyZDU5YThkNmEwZDM5YWY5MWNjMmEzZWQ4MjljNTQ5ZmM4MCIgZGF0YS1wamF4LXRyYW5zaWVudD0idHJ1ZSIvPjxtZXRhIG5hbWU9InZpc2l0b3ItcGF5bG9hZCIgY29udGVudD0iZXlKeVpXWmxjbkpsY2lJNklpSXNJbkpsY1hWbGMzUmZhV1FpT2lJd09EQXdPalpCTURJNk56YzVNMFZCT2tJeU1qWTNRem8yTVRRek16azFSU0lzSW5acGMybDBiM0pmYVdRaU9pSTNNakkwTURnMk5UYzJNek0zTkRZM056UXlJaXdpY21WbmFXOXVYMlZrWjJVaU9pSnBZV1FpTENKeVpXZHBiMjVmY21WdVpHVnlJam9pYVdGa0luMD0iIGRhdGEtcGpheC10cmFuc2llbnQ9InRydWUiLz48bWV0YSBuYW1lPSJ2aXNpdG9yLWhtYWMiIGNvbnRlbnQ9IjA4MGQ5MmVhOGZhNjM0NGZlN2RhYTBjYTEzYzE5NDAxY2YwYzVmNmFmZmM1OWZlODQ2MThhYmIzNjhmMGMyMGYiIGRhdGEtcGpheC10cmFuc2llbnQ9InRydWUiLz7sPWl32ziS3/MruOrt3t3XocT7SGLPcyfO0XGOiZ1Od8/M44IkJDGmCDYP28q8/u9bBZAiRcmmZMk5Zmc6Y5IggCoU6iYA3bsnwf8ezWhBpITM6MFgyi5oFpAslPPS/0iDQi7IZCAFLCloUhwMMpqyPCpYNn+g2o6lWbamDKSQFEROP5IruchIkkdQ9fDevXvLXU+iYlr68jmd+4wDmLKsCMoiX9v9/ZyVWUDlgIV0LYCDQZGV8Gp0iIC6wHIaA/I0lOMoOR9IFyQuqejeEx1fh3SXIBPGJjGVASkqA2micRSQImJJC+lAPS+fyC+15y9+eWrpZjxPg/zDW/bEOI9iZsofj7zLn57Oz4zy1W+Dwy5R+rt/eWZOcmeqXF4QMjl5efTLhz8c36evL5Pff3+pZuqvf53nv+pXZXby/jbd//5p+sv8+Omlb1/qVJHLN2dxMXN+zoNz7al5WvwSTZ8o9OpSG+dHt+n+2a+5+ZK9f3/++jEjR78n9uXb13KhqG/nl6n7Ss+PPyaFp3u/f3gbQPf32r2zoGDxvIiCXJ6yvGhTnMU4vSwbCqYiaToM2AyZ4ZoOoIYcha0uRMMbWtALqCeXWdxqNC2KNH8wGl0Df1SxOL0qaJaQeORn7DKnmcf7ugHWbaDUr2sJaHdNALjoOWYrMzL6IS4eloCWjJV/mBQPeQnKRlPix8wfgYRebiR7XfgsLaJZ9InGcxlbj6OYthD45w9/lKx4CAyTA2bi4YEkroa43K8eMxgjAxVRV/rbP+o3xTyl4VEZAjoBXfOeJCyZIw4v3tYvEef6dZox1G4vwmXwqmXrtm0pqq0sI3JBsoj48TpIY0qKMqNPYzJZ85ZepSAYMxh487IiQF5Aw3wZgXdlkkTJZBk4qYb5IlwDgGOGU7zS/w04R51xa4qhO5YO2ktbBg3aerkmziLMinj48750K1iO4Zq27to9sIoMSIukq6Dd1KXtapptaEZPlxOQyGLqTaY08FjCDRGQ2xOzhDRebh+TOc26TALQLNsxHNM1l2uDfIxBBR7FtdR1ZwSGEhXzNf2tJwhNwjfjdySZ0LqBqihKQ/Rru1s/l6vdmfvtztqsu00H6+y3O6Sd0vDQmIEXEP6yIj3//LOB+VVJqKk4mq6Zqv4ZJNTUdRAmxTF7YG0hoaaqmbrquOpmEpqWcexl9I+S5oVHAk4+D7T2LC02ElFTcx1N05QO095aRK+hyG1l6prJ/BfkWkM3bU03+jhpL1yrKo5t2eYe7YqpOopua2Yf+hXXRnle0tuxq2ZqhtaVjtuz63pS3Jpd18/iLuz6jy4XrvAZsEQYLTUT17/9XVxZ9vcanX9WdzNSBNO/19WrG3oFE7KoWt2go9qt+J/gt3phOZvNPVIUWeSXBe22Q7+z2y4o84LNrm/Do88bgLXf//mPZRJ3WbKNY0W45QbchV9q8qbxxp/RhGYE4mKpdp0lmCLpJxKcX4IrlEuPgWNhqvwoBj7oigcweZmukcQ8PH/ZFYgPZ4FFLpNn5W9Pwmeu8/i3dz5JutxzEWUsQUFcaQ6iE5ZBw/ELZqlpvMItXUKpFuh/wwbp7ZNdjIW8unkjCysd2qqhQ2igmX1OZp4SmJ/e/hzVNi1L0RW3pz+WTUgSfeIi5KVxTcebunZUXVFUsxvGrHQd5V7MJhMaelF/r65i6w5oRK0P4QllfZ1pEGOZhqXpZp9DUDkBNHxcZhmw7Apb+qx4GsUQci8M3ANpTOJ8EeyRIGBlsnmwx8P1FQZrIrm19nKVXVwVplg1jL4BTudhxrwgjoLzYUjyKY9KhgUlwRQYs2As9tmVFxSkS9NtMXJUzVBh1EqfC5mX/iwqhm2+yzH1kMa0oF4eTRKvTHfHxjAdV9OdPnOdMG9GQeyDIZjD4Bx4FVMCUUg9NvaaVMPu+NiObZqm5fTgI2YKiNOOIGGugvM4yoshCUMPkym74+MopqlrvfxTzVaTO/WiWcqyAqYMHR26B0QsFb2gzQgzpXE6jCnJEm/GMvCGfJgur4Udx2qh2XfCyzF1RTX6lFyBMQQX6mHIvIQVHqh8UHhe18rsgouruhAlG322QdCIgl8495CFeEoaZHvIk2tjEPiGUntAyrRUzXY2m7g1SAlG2idKmqq4hmm4fSakR8gi8B1AFc3ozKdZvgesVMUCUbs1oYSk7ZdQ4HEDh+9IqDINETHgdMzC7o6Vhj6Q3ZsY6MEqjPIZxEq7o6Mbjq5pjrW9QeNR2gW/XZi23fExNR1N2uYGtsUy+1LXmglmTDX1PrvaQkLIkcelqiLJvnBxIbTVrD4VvW6CKp6tWHhnVEzLAGfb0vpY9yKil8MgJnmeMYjqxJdEr43ZHlABiwoecB/bCilq4YKeF9gtXr47FjbMDkQzfXPTIUgTYO4E29AUCCY20yMN8ErP7nc2HEzvGeq2uNwJZziuYelAm9vRZVGwOyKuqms6ODS3Q2QcZeBz7RUdC3Sr1ucLd9GZgFeF+eOA7sHagOFTddXYVoNs5++Km5UvWjeg5eiqBaFCr9EpU6BGiOlJHg6vzSRsSxNXsQz4Z24oPKjAdo8cXcUG66ZbvZGjAIrTsRi0H8Ux+iIpmeysxQAPWzdNpzdCq2J74fR4gghL2eGdcHCBK119M9cV0+zg+Hg5WeTUd4ANNgyCHqU3y1TDnqUkW+a/PeBgqpar6OpmvNCVgT1kVVyQUdXV1d6sWOUTpzRZgM8vowKzPDvjYII11RWnVzXVEgEeZzMP+4kNXPAkXBvCqA1VAR96MxG4OuRqd3GwdUV1Hac3iVM5nFM2o6gKvD1pJnAlHF2xtM24cd/QYeCmrSrKhlZSmGronmZpFuXUK7KIrHyC2xILTQEn29Qdzd3QdVj16LyCkl1dBkADDJOpaxtmGqIkLYsVWnhjlu0BEwukU1F6066Nio6SknogINM9kcICt9KGkPDWMzLO6K4KAtBwLEu3LHMzU1WZyZocQ/xMs4fYA9AAX9Jw3A2zYl00KupUemtnZFSQFt22tvKfGmTojES7C6yqqhiPbcgeXRxS8HMvWbby/Wx7NFBmDUCmBw3uXoOQBnvy4wAymE8LM5ObGY19swBIBDCl27f0roJeuzApmSOQ/ago1UYa6Op2FNhj6KspmCYyTMvezGxkFNxJ6D2kIXckMS0iOBP97KAEP3vOysxLKA13dTIRN9swwKxtmFLja6Hn4FhFM5LNPf6BeT+iqukGRqG9+bReIhWAV2Pt9oAXfs22nZ0nby/2TtNdx7A1tXcxax82C1u8F/unmaBibIgVetDqIrQzYFzBCf65vtns1Ko1IHFQxgQXoKtVhnh3TEzLcl2rN1dxPSbmnj76iFW8gIm+mbe8Dhdlf7jYuJfG0DfjWOGbhmSeexnqlWZp376zSxwz3bStDTUO6l7cQuRVq+L3QBhXc2x3w5BmzSRVi7P2NlGOqiq2siED36BaSJrGc69e5eHThI6jYh/46Yrj6r3OxLX02lK+bsFQjqE5urvhl/LW58OC5Lt+9BDLxV1Lczbjp975y4syRCdsu/m7DdEsW9VtbbMvRh2ieT64a2GQlTP/zvDDPKTa7z8uZ+EKtrUPchvUcKciaLDNVCuJzxGv8u6m0jVUw1A2TF7XSePbZWlug51rQ1WzdyKrT+ahR+KYc9mdEcxUwOOFQLlXYSxQ6vD/3WEGHG8plr2pn0lCL2TBdth09k3ciA04ecDofdjUy9NpgivrN4zbb0Mcx3UUt3dtxjKfC9fGJ0myqbfZgtdeqn8TZrppgPO34cfEdWv6F+hu8xHnFoiatqLgKubbI5rToky9S5adj2N2eWes55rgFGkbpr2X9vbcBsEbMAE32lIUXd3Mu8jBS809nvvdHbKh6aqiK5tl8kKKORTQ5gKFpNxLeGVZmusqSv9umyrnfT0OCyAZxa9F3X26i3Vpf/JdwJL06D9kWTpjUprxhZYS5uWkcUzyKWjh+1IxpVKzVFf6+VTiiRmpYJJPpZiBNxBKUYL1xBb8KSjLQ6kgE3g/ZhnlPTx580rK0BvLckmWOdg8yKK0kIKM5TnLokmU4PZn3PvLynwghXRMs4MBvwwAQEEnWVTMDwb5lAB7yzYxsl989uPo5On79ER7Nc0/Pvvt0/vzX1j44VzV/8jJ89+OkqcfP72JX7OXz85+zt+w55QVx6Pgr1f+rz+RJ5f5h1+jy9J2Ez26SH4/vWLp5cHBQMKNIoAKeInVpvjRR3JBBLoDKc+CZqO32NBd7/LOQSL4auuRuB21dlPTkFDNN83hx3xw+GgkejsUG7DF2QXd4xzyAu9WNr7z7fKHK6cd8G3hMQMqtloMxFkO3brInPyIhT4gq4cqVJZHroLE9gkQP707ev34uff23Zuz48dnL9689t69Pzn2Phz/9PzNm5f3Xx29e3l89vbk6PGx9/b49ZMXr595L16fnh2dnBxh7dNBazM80lcGNRNdHAyuxBb2atN5C6DiQDTi+4GvanhjqYEb6qaqmGEYjlXXdyHgUwNNH49NzSCq4YahaxPbIgoEpo6htg5FWAMvyNM1MG1KLDClYagSh8AlMFxqjxWTgm84JiHofV8lZmhYjjGmTqhRywdrhSs9wrGtm34fzHwNTG1smJrt24Zt2sR1QzM0LUMlJHQU8K1MxQwoOFqBQQwH7btt+MQc65YPow+d0HVuhvlxHUjf11RLd2zbGAcaDcaur/lAQYhiAssIA6pr9pjothM4Ci51Jw5VxyoxTAJuFVBHDFNwz8rBE7JYmryW50Zv52csC6Yn0WRaYG5klM4LLJDjukSC2lJH/jZriWI6WDlxoXWYQ4gKGykhVxuLWjiaju5YrqnecBZEp3lXGLsIbtRTe2tAG53WES7b9ZJcshtwWiXatv2npQ9qswWiPm9iq6Hm+G3kvNUL3xW09Vhpgd6JlzFW7It87S5vQ8t7tVLF423AJMYHgwCtHpiaeCBNMzpesS6bcbc4+GNGcog2RxCr5CNxXM7Iw13FUTCKZmDY8xGwJRumiMxGZ/9UB6HI2Ml1x52QNBq2cMWvJxekoPUhKiPedlXy6q5plrHstn2Lxtd33jLB4DpB17uBa7rzRHfLCDSTChNxLkcBatW1k7riMqQRBE6hjAwIXscwv+CHN8UMfKDvFP4/occbECTmh9UUVBJw+LrFgwEodH6sz5hciHLhz/DpH/GJ3wyhqn1e33Cm6aCwOWAYz49Xs+tYvB840qNzyhD4lTM8Zgpo1JrJ71SqgXmCyu26vJacB9ikVZlLDwgC6ppRxfztOUyiMcXziwTSo7oAnDg+bnRd31SuK+h8GZxyzClCHJx3ka0dp3rhkJC5mPuXckySSYnS2fHdAJsRd6f5rc/CeU1rsfVTZmUh0eRCbrZDcfcdNFaeAvWiCyqeUTeA61rMY0AFP/nLlxlJH0iYXjyXseBh5VNW2imMLmpQXPcJNYjoQpcw0YgVSBf2AoGOVDWGhqSi1Xcg9Fkhs7FcjWjBJumVrEnpXDYEe8v+RI6SMbgFCfohtCrFJWXy5RT4ScLDjGSAP2ZBmSPw/DxK5YItej48hQIMSKqCRyOywCdPSSI0HS4+kDHTxJKWYaqRytgEiJYLfcijmgxBtR5ln2TS26qetKDKOLqCAOgyCoupPIbwuPbNOeCK4j4JznELdhIKdn0gfWe7vjMeP+TtHkjK9w8HG2BZQ5eBLDNpHc6IJI8vADpIgLgKUlQUWUQEi5Lq4ZGY0xrWczHDLA6larJbPAeUCYG3ozjnk0CiBBo+ESXSKsOMDZxwbSBlTJADc0RCl7QYbdGTfBVLoRxP5HFMryT8wwcMsHguVUpb/F2Tu9VP2Gr3scyLaDyXfbDblCarnS04l/Nu1cMsk40bjPFAwmMRQGp91BLPK4muJnBCZB6jHwz++4QTTAKC/Y8kyHlfmjBk1FoL3Oeq+wHaZC6FM9REh615eQR6D+iPaupgoGsV5GkUgp6pmQNZ5id2BQGRpEiqBf+gTLj04OkMwWXlXCba9zMZWCBESqquMuIki9GvyGZN/ZQUU2kcxbGclTjDmElgIXhc4cHglSMpj/WhCRf4r7pxAn5rSNpQcyVraOqSOTRsyR6a7tAYKnA15aEqLjq0gAdXHiqqPHQ0flWHhitrcDfUbbiauiwKLKjmGlAFrhoWOuIZmgEtoECDAhV6teBqir6g9dDS8VZSh4qDKKrQGNoMbQ1vsdyx4R/gq0NdCzDE1rwzeGEiPlCkDm0o0GSogiBdcaMPXRNHABV1rAhDBGw0joXC6w91CzvQ8BHKNU6qoQUwADRAQUywRxUR06G+BpjgXxgFTjq0hwFLmoQvVCQG9MaJiwOvr0PDgJfwH/DI0EVgEgID7IemhfVU7JHXV00+V3xIMGp9aJs4FlPCscCUDTUTZm9o8zmEqRBYIFE4DV1BTICL49AAJMwuzLIJc3nkwCtdEn9hnAqgA+wgG0MDCWc6Mv/vE6ox5Cuuxi4mLTlF9d6WkSXZB7UB7jSYEVCMwNdJgA5S+0EGwzShRa2yIZpMQTUdLivDdvcj6L8F8DpVc41W6djFkfgI+Rd4wCXsB6fw+GOZ/kBm6UMsi1lwIPTtj0Lf/gjqY/EWdcbB99rT7/XHi3MAv9ePRcniHEBRgjYfLmg9eXsRDlSd85hm0EFSaoYVJeAGcb8hOJcaoo7NVfOcMLnExB42kHxQYmhDhGXnD1A3A5com0vcAoIFAfNf+QCqBKpWxz/5TDZX8eHaip+yUCvVOr+KqUoPHcwqu1l9ICnBKQQHLKi2Qja77e6LS0rmaCirRlVn9RE9fGP5YlVi1SW6tJUFXOppKTKvWiRg/+83mKzid/ri2Wvv/duljkTuExBIJh4EJUv1P0skuITN0pEqfEB//jlYmQd5OiMBRCCEarZGNd8nNtXUUDOoGRg2pb5r2cFYoVbojDXFpoFvmdQhvqlS23RMkxiWqQRhd8K7UiNJKB4/JH6ePizTrkh11YBfFgXYrbZtPgMRioFFIYiYVKdp8tf0CvwiYMVFYkGEKqKHTYxkywGq1IlfJPzI2pa0zApZJBnvdXwtbtXXWPLa0GvGFlYdK29v1QtQRxS9xfx2Rh2Vu20ewf/hH6pxVcUbbYr2hCyKVRXtgvlc7VbmzT/NFGm1D3vjPmzs4xVvpGqtBgpv0O1GUeS6m1XjwvOVj0aCAw47ur9rBtomQDh2r2hSSs2t3HaWO6FCwVJZkTKcabj6DADO4EbMvbBHdQNgpIUP3XGIex1bUQJqiDXB2arlaqn2dq80CStVLgsEm1gtp3i6F6rzVNbb3vPXIX5r5G1radPAczL2Km1XFf34w4KCGwmaiS4o/iFLcqJYknICjqKqgHcbg0umyfinLTW8Ev45UXV+q/F6Uree3KoIHWLdWF5TUxE1ZVET4UKXHDVw4G2nXVHhlTaTshX/CthlEYihiICzoOMf4FRFmvmyiX/wYTkOewZWj8TL0VMZLzmGnOXxGAuZB+ac8VddtThqGrUcoEb2uMhg6mPxuCR4y4JTPUGlhaiDu8P/QJku0XBCeXtUD+tiZwTRAOvK/XUeqLQyLpT/KkJfUV21NEm1VOFK0JjMF88QcDcuc5Pl6PSflzNcNr+m//pNq4gbS5hWBR1BvaaTgv4kH+4S8dv+6FrYkvRhOpeeRcXz0v/L2vdcDaCEp1cDaV5dl62sITkD6WoWP8hTAq7yIMVhZxcg1yidBwPOL/XouCwHU3oBl5BdQqB8XkyaCSR+zuISfMd12vyaIdS6gBvY+2psDbX7KL331UaS1rZaDpCWXgjCr33ZNgcZS8Ug0JAtTEftsot5EtLoywaofhiMsYZb24NdUCCmY2zJLyg7xjXDb4Kk1uflKj83h4AhnmK2CTR4Dpa1jIuMLJjkBJhJlqttG8shydSUfipnKec3WeY/k7BNquZpjcphfVfl9irMmr7z+cxnMRCPkULm9l3iTlXCshmJ227WIiJKM7ShP2Qkyx5WybpW8rJDnUaZdVQYhmUwLXrrc//aDhrFtqR0gLcWhJ8xXxz5vg3ZG5dgifDjNYQ/fMUB3D0JH43i6Doh25QYNReOqpVZd0iWIwHhG6ML/sQHV5V3SZrHCyDfGHUA5/Pqo85d0eZtBeIbowwMscTFXHdImdMKxDdGGZQoGRfu0cvRHcuUJMB8YwTiS2Xzu6TNCw7hWyMLXx9J7tpMvWiB+VwUWu/SPBqV8eE1r653lCBwVFoRWcFSKeU+qX69R77BNPCPkbgKZFPKL1H67nzX0xqvQxEYLQq+LHdvQlKx+Y8vPGJZtIUR/UykfVzhJ9X4HXZLviCJhXCsK1+kMzvFIsxfSYWsgdCavK0SHevyEYvZxr3jixnu5giWpo3nCq5PDWwxg2cI8xD/rgm5PtvIm91hn3v8xw3kw+b+Wlr8O0v2L5ElO75KQcHQf2fImhd1hmwfKTLwJjTMxXyGZNmNXs5u/gwVTPIVZN8qdgUNJW6qDO+X9F9u8jynRrOYbRVUGxV+P2MJQ0cbJky7zi09wfP4JZKEfHWj+AmVR6Op8fm5AtCKgjuMLrax3QKVQ3H90rHaRv6s+M3Pu43PtvFfW/gcth6+/sigwH2EfDfFV0DGsxqZw/ruqydgvaYoJv5waVHrV0BOruyQiIBcpfrw6YR8UY2/FVnxlFix3mo4KaPwDr+lbEHXN4CUJLCSOFYg9Ktl/4+s6mOWJKDxJDypU2LFlGb5bY2qsotRXbPADw+bmN3lJ7izKZXeAZBXx7idAX9M+ZsRrhaZxE99fRXSdSxQORTXb5CYszK5068xW/klFTb484blDCV1qeCboS4Ny2oVcmtL/ddA4SqCOq7xWySHFyXfDI1xj1l+LX15GP5liXyKCIpNY5hsXCpNRemXN7rrym+TJ/4c6VLcjESLNCbBZ8+XvmqBPmw9/Dtj+q+dMX0rTlr8djOm30jK1KhSpne8tLA6OLMxFf6Xym1WjDU4fIs/p/IZVhV+9oxhRevvqmUKsvgdmShvHRrwhX1N/Fkbif+cDfc0m8dvxgdqPiC2sznVrwV9JWTmuEin+MtFPPRuHr+89/N1LCHZPGro82p3Wv+wTdDbxA9fReBwV2selvt9NEpIs1llZefxdQcWtHew8HFWL/g9lKL9Wt65srbnWZTIwlfiG2H0xUaY7u7le/faHVRbfnNKsmDaMsP1gS3Vi2t8xJzGYxCAjBZ141koCnknNR5Q2Hig8KBLecBSGta9C0hLRQD/I/JIwQb3uodBrCBTb5bC0wr/a/C/eIogv380QiKC3kbmuJqlh/wNP6iHIc/gj2wcPsK/rc1jrWHL+Ko+mUIULe8rOsUtkUIwOPqy2Kz2brH/d+llFC6dLtW8qUeNBw+BadxgG2+NDO8DvCicwjU9sWySr3a31LhM1jVcDJZ/+dkOJ/xdRlB7wRQiWVocDN6fPZXBz53RYsqAAhPasPMjTsea+EhtfmYIEFzix3fK+Uxa4tDFqTK4TxImK5hmbEZjfu4Jb9AcObK2Xc1Uaxj61keDcMDVNkXkt/bO5S1GJrqp8AMTQOOwJQVNwTJ/1sfetAt5zQhQxd9rx6OF2hiJ7dqsOKfzg/z+qPVGHEb0x0pl/KF1WfwsKp44BUpuCeV2fR70Tlkc4vGcp4Ifut0tGG5N5So3stLm+hbtqqhzApJGBYmjT3hg23jcfi3EGIymz3x2tdSQbwAlecrSMhVmfV2N7tbU7nsOvzquWvSyUqXiAdQ01bxCyAdGOl+pWWuY9USsW+cl2OCcf52VMVwDScVfR02nf8SjZ7Q4Fa9p+HqxzfYJVk+q1cCtbvOUxjH/Le91w1sa2RJdG0H4PwAAAP//ALcASP88aW5wdXQgdHlwZT0iaGlkZGVuIiBkYXRhLWNzcmY9InRydWUiIGNsYXNzPSJqcy1kYXRhLWp1bXAtdG8tc3VnZ2VzdGlvbnMtcGF0aC1jc3JmIiB2YWx1ZT0iTFc4d0JpeENFcGVzZHBkcS9JM2VSZzZaU3QvQUZhbWpEa1R1MlpTd0RIcHdJUW0xQlY0bzU1aklMMGcwTDkrbVRjMTJaeVFXMWRYMlZxRlRubnhtSEE9PSIgLz7sfdt23EaS4Du/Ird82tN92kDhfqFF9sryRZ6lZJ+WrNntF27iUiyYQKEaQBVJ+czr/NH80H7JRmQmgEwARVZJlCW5S0dEAXm/REZGREZGnBDx70m2Wm8a0tyt07PZMkuSdDUjcU7r+mz2a63VWZNqdUqreKlhGm2RpXkyIytaQHoMmZHzEyL9e1Jvr8htka+ggGXTrE/n85ubG/3G1svqam4ZhjGHFDNykyXN8mxmWTOyTLOrZQPvxozQKqMabweUX23SrjVFpZmQlCZp1bboOr3Taohdzs6frGmzJIssz89mq3IF2eqmKq+hkV+Efvg0fDYj5ZrGWXN3NtOdGUnOZi9s3dXdpWnFpu4Tg9jE1G142lvTjg348DUWoMHfUoNkGkun2RgMT3urYUIWykIw6VtoyhzborSoawNWa5p6QLyLgJiubi41PbwwDQxZmlJmHKTzkxNpaJ8k2bYdi2/KW7IuYXKycqXRqC7zTZOScptWi7y8EeNHEg1Hgvy6KdZaU2r15uoqrTFLTWBqJ4K1uFw1NFul1Uyd1JMnm7ytuy11uoQmLdY5BaBRizo5eZJnfQmLPL0l+IAyIOMCprGhVcODAOQKaEq6atKKrDWDLFyAt212RVl3MRprnwgaN2hGqjIHICjX7Aub8oSShkbZKklvz2aa2cHXqtQ2EFjl0GgiN5FumnKiZVO9Z1M+2ZAuSmp2uYZJWmu4Aqp0cTYD8KANZbVofEFKHeETIgNBW0kZNxmMtlxvG8RaXS+rbHUN4wgryCJNetu0XeBT2c01W7rtYjS9bo3iq6izLVj8alW6Lge1jJvBU4nKSJM1OCN/TxkAl9WdWPQ5jdJcDd9m6Q2A+tnMgOVpegQbAjBew3hAq3Sznd2suJIxgFZtMDTdpqsySfiis4ilu0/hD3+hNMN08HcZ6L5L4Q/+s1D+ujUhlRys8felpoSbJq5+QCGYRbOWWkBNYmIGAzKYDuIQuRBTN3wLAg1siRNiW/CJcRAMJW218G0B2MDVzDchFBdruu16gHN0L3QgswMoULeM4A02Q1RlQrnL4O0Ll0CjLXdrw4PCH/xnLdEd3cpN3XGx+sCXohC7GReB7iEm8tU8mm5ttWFREOji3xJiRhHwf4S/3hmu1lX5axo3D4NWm3AAXT/zYBW0usBHhCuTTe5T9mOKiTbY5sGB6JmBg2uZnu4HDquLpQRkj/CoZsNYmCz3DYYM4xw21s8x4C3UyqOUyRRzw0FXCUewGAaLKXs+LKad4zeiJtYCW1oLhlgLW19ZIga2ydhqg1BMjH9vCy3AWtTVxvO4g7UmSoLgty+C6brtyboh9KkUCnkfDyI5yfEwQIp0A3h8xUJVcGzDHhMacSB86gBaCQl/IpaCoQ/DAMBUjYDxZ+lhbvTAgmDfoR7xWBbAFR4iDC+3dcMh+JDxHosiLF7DOPaYHOwnc9iyzjkh8wS60I4v3cJeV/E9aeegSvunSN+OK80b3DDl8Xyd0gLIvirGiJa8DCTyMpi17ZjYRaWqkLjd0QoW1VMFQ4qLba55umgIUBM3FV2TuAaaqNqsYqCJlA8NenOVNrOJUZJaF5UVkCQAAEiaJID91aGKy7ystOgKyK6qyWh1R9a3kIgHs8Z0EUWOuT0yJt8imly1RL5EC6zpSmIEphLzGpJ0QTd5j4HlKclgSJZZTap+U+8Jyx9HkW3dc6z88JZc5WVE810NoXlOygX5IWuebyKpGU8hnAfeU/1utiTRshWSjFqUl/E1H+atRvPsaqUVkCEH+ur//dd/y4WOJnt36R9s/hONQfNWW5Txph6DA351g/Tv8EGacmpC3r37+ELPT57M8wzG4sl8k+PPfXwGrKkqrQHWdrIZDzAZguwd0/LAYdzH1jAiXTRe7rw0znUKLUkoAvjLkoisacK4BBg6IhrejwPvt9TtLOmRkUjdYv48q5uovO2gApmigmNKiZ3ohmKSNWlHTko1zcGNEn4MBi4G3igZ7qWfNyvHunSpoNkjN3fk5o7c3JGbO3JzR27uyM0dubkjN3fk5v4Q3NzvzS+UNys8FfzjcQ2sY5dH3uHIOxx5hyPvcOQdjrzDkXc48g6fMe9QVld0lb2lkk6LzD3I0Uf+4cg//E78A4eVPxTnwLt05BmOPMORZzjyDEee4cgzHHmGI8/wWfIMx/OGI79wP7/Q6lGR7l+XjH+wicTki7IqzrtoeSyVfF2IvJy6GzdVmgMZvk0R/9ikiDQHH/kVjKnaI+UaDXAPnFif5+VVtvpblTabanXZlGd4T6r+k/30T9b38P8qa5abSI/LAj5+vntdAoheIBJaZasrCFrfNRik5VIY1BbBT0FrmDJ4Scq4hp+63FRxCi+XwC7BDgNvWUEBHcELNKLU16urmXLTR3T1Obtn9SJdbaCa1fUAcGR+R83OuJDlXVKVWpxn8fXZ7Lcv/7kpm69xo2kukTXh36f8B/DgEiKymLE1OsvDY77iP2t6l5c0EZlEYTC0LP1ltrpcQ3eUIvHKmrgnRgrogFJcjy0us7bQ1SbPv+qbM27kqx9/eHn5y89KQWWVwRxCK1ZXl5sqV9Kz6Tydz/uJnA+ncT6axDlO4ZxP4Bynb84nby6mbs4nbt5Om9KaTZ1Waof+8z9no8nQlgWF7S22g4Vj+TE+TBoaaRrakbeIUurB0zYSxw9jNwqpH/lpFPiJFSRJEoa+b4deHHgTM35F2+n+80UJ211Cyk3zF8Kh6CvC4iDwFaxrkq2+YjvdaY2rPFspa4S0aZQlSfsk4xUrLywscrOGlbW4jBt6hmX9dbP+khbrrzEM4OaMQ8Zfc9bMv0Izu1iEpDNcgvYzHE+2W//J/o6HIOBIIWK91cvyhuXncyUKZxyrFHyJ32f7rGR1aPdcjQMkqggjxA4gMD376JG62BcQ61tkfaeZw+qPq/lfbTW/w5yvaH4HPap1luDh6Uby9QqmTJ0UXPWwUuXcNEaoUJKxFqE6dc3TE9jOCY1jAGS1ZrbfK1nbFX46/zLnQ8yW85dXzdcspFvgLIRNH67urwUuORUtbPHIqYDJvEN3fxSAMjwrNBybxmkQWKnvBbHpW77lGjQK7TheWKbl2YvESGPDiI3YWThRmlIaGIGZGm6QqAB1PgAvMYwq/u4RvES2SYSeoNX4kLekb7cRMBINteeZSFgrF0x1HWCxo4NxIpFiZdQqspk8K/vHysIS2HCsf6W3iL5z6D7XyK+1Bd4wV9ThWa3tFQSJueAp24gJoStLQHiyBUwDIb/9xuNeImsIs3J+ouZgyLm9Y/8k2jRNuVJKg7kra8aRSZ8zcaGfp1eZmW+zusjqmnNPRVrXADeqEHeCoZClHQ9LHCR5CBtVzAEDWKyBd1g1AyZlKCq5bcftXlmFrfsWwYcqEDI8lEkSTw+d3NYtC4WQlix66CQPFyG+BSwVGabqBRRQWCjkFKN0XCbLBR0XWCUJWJNQFhLIyQyWpBdwnCgSDj5J/KudBgRQAA0xPQAYErgrrAt+twDX8zO8rBNm3iHON0mqLSp6BftnI0HrqmyyhdirgZpI84U2TC3mL6I18MooGpnAXXIx9TxKGzpnhWFvh+Wdt2sOO3giUTh0vc7bphSwzAjHIqx2qKTIGm2J4pKYVkmtpSsa5WnSJ0myGlY2QuC9yQDsN6lGV4m2rqYTSoQmwyp4aoPKV+xNGMngRi3qeJkWlBm1eFUumhtapa8YRn5WJj1wt7DM+iTwCdtpGKLpcYqEffoLOm1jTnrQ6JDWiYL5ekKoz65xdNk1ZXnL9j8RDGw/WTfAvcI6Txku0uq4StMVcrLubIwNO5pbxk7irAuy2Iio2KNIgB2GX2CH3dkOTrqXgRXZSmPoAghZ/rvIGsZXD/iCJ0uzw4nyERuTk42P2G4iLapSeq3dANFLFjYXrq2A+6d5d5Xro+M6drTFiXMh/hW3yJhkcy9EeDya+l2PpnrUrQiykNUoxc3CGjFp3VRpg/J5hMp1Va7bNO2k0jYrkIdkgYZKcHMWaQRt1mKoVhlU1p0YJoFigAGGJPWYmuxStcfFoySz82GIIMqkC4tyf4vbTpoo9ZZMX4icy4U0Vbm6kkYFSW7Zxo81MYjdkHVY6Wz2xS5MuqOHY5J6dj4K6jvNmjnq9gWSUIQ9NWmxqiJLAoO5iWA7a7sNlKsp2/KZkDr2114RRyOK1jj7U4+YfSGALVq2Hy0d3SFaWNMkgT6cEmt9S4yvOe59gvJRUUkHck1Z5k22XgPj0r9qNYmaFf5pdaGSjP+n3JBiUzckShnnBflQNg+oa0lXQJ7IJACp0wbZnlpA9KpclHle3oz5jY8gTVAbuolgy8uYegeTLZAbCtD2/hKGi59+uPzx5R+FH0yi1HRdmvgpcH2RGxuRm7qOaUZOmvh2HFihnbimY6e2bSyMKIn9xKG+ufBd07WoZ892Sb33lG5/Ors1rIVc2ZNx/w2gFgqbsDhJDgPcjnw31g3T13TThocR+mKDsftdJsbIgLAdxtJN02bR8PG0LQxLnjInNkUFQFqgAqBYYnMqwEAqwN1awJX4sUF02/GgQsPSvQBrgk039HNsqWEj4eB6FBoc4J/Y9wzDhq9cx04ZhodlGJCd/znij+Xy8Y/vrhCCMZjLx5xL2PJDj39jkSam9/CvS6/xotRQB0M9Vjm++Wo1rHEato53AGkX16GKyoPJOqlBL9+o48JG6u0Lm7gUBojTGfDoR8pwoWT4C2E0HJgZO8z7YULFCtdkZfCRgonTnQCIHPu5hWF0GA9NtDxsou3lUmufSkSGTXys/M0kmUHIS5nBOum2KKEoJyF4Fcn3yJz93AB20ZigjJAd+H9PpI9iFkLlU9lPENOzRnLO+kGEbvqBZ3m+ZfwLYfWAJot0EfteEBrUiOzISNLYT2NqwV8cLyDQTTzPts0oMtw4Dt3Qju1oYflJahtO9EhY/ZPA6wxWWhqOEa9RCYBTIBNm7sWEBYQxFBK35fmAF8wAlnwQoNwoMF0C/ApEmIoECRKhoMnyQ64K4wGaCYG3MlHRxgxH0qYA+LrQROORgFEcHzP5HpYQBuOkwKuFue5bGhblXEAjUDRm+86wBYikghzbBwyZaV74um35uucFkkKUyTr5tjCQuXPcCw9SAgYdspOu50CJrDPQDwdZQcuBhxmotaJSmwdbA+SwUGxmBJjK90INWTpPHUxUP8ohFg1tQiTksTWWZVikxsrEKjWskqUKIdQZtxICcRwt5EC3uNNMY9+WA9gNVj36fQWAdCKo/k6o3WPkuowzmgskjMZVpe8D+JY5wusVfQsLoJW1y3jbdGGvJ4gMaoaqK8TyO9RoYJ+zr4UYkgrp4tS+ss+u8n6byqKsrj/5TYVrIqfAaeY1b/Fxh9l9MGmmkUdDz3DtMImShLpW6plJbFpGmga+ZZlWnDihm6QpJDFiYCWoa1PL88w4Muji0XaYT2SXYcIKBJo02WtPQZLVcgenEvgy1IDlKNm0LIpiMmJxspJL8ICw1QM/eCrHGAbLHzD5HrIIphWwrCbhT8zM8r4RaaiaHV80JpLTsPDd1U4KGh1FzRa2EK8r6W1hYzDsPnLeezpua6qY0zRY9ZIWr8HI/R2YHf99DzNyIp9A0gMw8SptbiD/vEiLCNHxFKaXzkMVNB24gcDSHCju0XWE3f26OyCVyH9F602VKklHAzWAZ51tU61I2WnKxJlDJ+NXDpDw38mTFd3uJXfbdQ3j4ZXSlog1MZvjsEDpVjrJwKoIS9X1pVNxwUxDRdpf2siXELn7XAKlbg+3Ti4MCMPkjqA5No0J3QY3YPJsj/I6dR0U7LXXinCm4jJBvd7oEBEmrw8ysdOPs1lm9KWwqDrN07iBjRgVh2p+UnTJUT6DtcukvFnhNshB75IftokP2IFTWqfiq6FX4i2q6CpetuFrGl/jniHKA1Aq7/Cwryb792BZNtfp3dnsisSzoUpZD0xfkZfdFSiuVPYVweM20vdXACmDJ6EGMA2p8QYoI5wc3OX3gVIFDtgFsuXtpQyGPLBV3d4Nrt3ts25uhjfT2oiZcigs//sE9jMENGWldffGOhl1XbQy6n02PIcf6VvjI312wG615/Wjg3/lvF53fNza7PH5O/I3ruYwJsbT8fRJ3jikIkybneUz3mpUl9HfXejKky+TiLq0vrJ7Nh+JtRDAejZj58fn+FRVyeUc4l7Iy7IhdAvUKB5f7zOHz3BDQhWWtmgOX7ip8B3lURAZO2avD0Rlc55rjNFMubjdOI2n4jiI7ULivcjytAYaPd0LHamt6LFSdiBW+pG35jC89CnhoE8a9XA9DrwDC2TTu6AgLrMJ8ayZ3YVraUeb9N9IO06685gWABk0IAEvBrr7lH+guNngF+4CCjiHeLpQQ7Lhpw9AgQ2EHIoqOJTNzvnvg+jC8p2DkASk/6B4Yo1aJ1X6T2h7czC6wMwT2MKaKHQ30mCFECBl4uv90INca48d1gdih59R9a9t4xFJfBgkcZXB4EnA8Ki0iq+bvo9SS9++ADTi27rn+9KZDp4p6YHrbIGoQSGufEPWsVjui7YM+JYTGKga47x9wVhhVSuHc7MDYgMvx1ricqzCiZs2FwtssWhvyOELpvqNC8y2pcoHGHeNwge8wItNWGrmG2dp9oo5xIRCvZ1iA0UfyeBlvC0glzGQaIz4et4o0XnTerjvB2JMZeWhnob0+SD+DN2D0GfoflDs2es8How7paxjDGqPCt6NP6Wke6HPccU9Er06EIl+K9V9RKEfiMUrkJGX1GsfFYmaDElM2yEIJs0QuJNWCITWnnzP30U+8gLP/AHHOM6bcMCltYYFALNNmS8Q5gO4RsHIPAO2GZvyzAASsjPOYAjjDMZzaws9c2yKZ1X8xMoXclMnwG/DBtxveIgPl7ZuBuHQWoNFgrYRwyjEoMRcagFrHyp5OOoISrqMUxqduhvrocdu6bO4wCFdDdtRfSY3HGFaz01nulOmJvUqRB15VNYMVCTfGgOwMB4fWzydCwfqqWw+TEVbU56NAzG9hB5m59LH58pUC3XAQ1G9yDZG845S4G4U36ohHl5Zj9rpgaj9qajziNY/CFpf5/Tu0RH5XvztA3wy6iaEBJa/JdPShgdBjoN0rGmq2tyQHBCwg5I4z2GKVaEcbyCxHVxIxR6KRQQkzs7Fy+eKPYSho8O5bJFvjD9ctch7eGyRjqzSm0s5hCgf79CaHsFEhzLgbZv6th8Rxx6IQ8zV4+KOowmsz8IE1p4MdrtEz9u3B1HmYVz1h2Wq6zTeVFlzdyiebPON8aSnFjmJJ9sk7CAbG05onlaAntYl4Ks70pTX6eqyjukKKyN48tZ/vUPzesRZH4g4X7UtPZJmH0YVdYn+1R9ZVOl4vm7a9kBBHjky4ANzhrhM3QuGrB/KAJ3gjc98obuep+k2noaagYUO0A0bOE8P3vUwwGMTJwyQAAuQpQ9MG5CPwZIHLh6yIIPot8jJBlIwRr7RQQ/rhoP6qYYToo686/OMGsv4zMQqgec1UcMej3VR1ZX4b7Bho+5YJhbneT7rEr4Gbwsd2WygAkOVRTZdaEOudX0fxPqwWdht120XEC+TxXpM09ZwUWobiA0JNWuZnBWlnLYDg2JDKlu3QiRVdX6VA+pzoM+olwrIPHQxgQtjB/kMkQXvJIbBMxMFFy4JeV0owAgglr+Jfg+bihdHfN4X3ukXIY5Ze4/SBBYdf4Ts1kK9KbZHqXtJp7hlT+0Lh24DLZ6Ynbdvk9vA6HY6u2R+CEabI8qkeadFS+M4XeMqS2+beVvsX5dNke+4j/4hdpEM0BI09F3OtOp0vIX4anm7Se0rQI3LTpFn1VRZtIE6apKk6xRQyiq+42lEAI3K5nKzhgJT2GuwcrJOy3WeEpT8bVaI6A9t994n5KJPRxL8sFMtNn2PzbybKlnZogIkefFKlAPEs217PAGT0jn6SOkFKWFB9BZIbKNqvWsPSm2tZZgGu/MED0DEwN37o3TEuGAmNwLVxkevfnPhE58LFR0fryT4o1R8b9N2XeC+D3+10Dk7b98+AufP1TtZRajNKa75brM6i7Ic1uYph8KvD14+rUzvvlXS6lSSzuohjeoy3zSAI9DqITy4RmXFNCpJhaOkGbNzMTStmvweaJSnZDWiZKr9hpalDRkZXeSItt4UBbdhlkt2bu65pCHNmmSIYoxVikozhNk07M1gvj/6+r9OI9gYlmWVvUXyeuo+avigUgtb8w8nK1Bmt7+GzHhlDW7P15VWrnKgCV6U1YRyW2fYRsxtZwppz1kVYMNucnOwwNd9BjmpyjXqwfK8ypdW33D/x0qv2nWpBLY0A8s2VsflIDPOx6iGvr1sB5I0ox9UEu3by7Pu1G799FWBDzDdMDGMhOkCM/w5GuG5ekv2nhlT1A0fe84GU/W7aTMerIU5Pb5cz+x9R3hKReujDfS7aoAdqqg2PaCK/sn7jutYceOjjep76YW8mx7L9ABLB7/vO7zqgelHG9p3Oo89/Lx4ejjFKdj7DuXg7OjjLf4PcTT1DodrO5CDiH7f0R5IoD/GaP+OAu53kNlPD38ruHrf4R9Ibz4asH9k4dChUrAdFIgYyz0nRVyQBM5CYg7OewuaPLQPgMGVLM6iXwGJV4yBAq4W2S1iC9niZJMVKTO23V9YvM13XTscGG3ceWIi6hwYdkQRKRqD5ZICyS4k+8H/T4RV0NayLe2Fla3njnWKRhDRnFgNfGQTb5rBsdBBq4jdJ48Cx3VMx3MSz4hiaqdGkDixayaLgPqRsfDcyLe9IN3nxvns/Oe2ga1RAujH/9A0glURAboEWnrKQhRgPt1a1mlIXTtK0nRhLJwkXixS0w78JIhMx04iJzS9hWOmie1gS33fMalheakfpFESxTGqK2vahDMV2egkNzUpuaDr7LOt0RTmwDBlgWPff7Ye7KK0uUGDm20axbNdB/8n93qgEBAlZC0DgQoXoAzFK0zAUUTwmDEw5BycwBlsgXQWt4SMRTL9IPugmfL8ICDoRo1r3RjdZGgksGMZy4ogI/kpmYLBSwO8ffvJTs3HUtJ/4yk2PE0DT7uee73ZTdS5N6eu6iuK/IPChRhXD2yPGea0CH9iFR7xl05XPpSy1VClf3AVIOyuAghN0Id7+sLWx7YLdmn632tzpUW8E46PpH1W2E/mNi3u8f7T7ZQxrdB1kupVphM6Ta27V2xxvGjXxnSUVpRJL5AT9uHHqYRZ3skmSsnYmpmdD9bMHBfMoI+qUXCpCGYK/BshGlUNgvMjpvLqKk/RJsU0FujXo7jW/4yZGudSNWWdLijb43tPY0eD4R/CYLhqK3xoED9brTfoUKvY3PJR5LwcjB6NJGMK3GbnqZT6ixq2zvRHDCBTaRMlMScJWeoeiJVsUztmXOabgnkZbUFZQ1BGmrjuLJiiMWgOOKfcgGlHx00vOJjrRlpK/Si0R6iIKvA4tmu9zmmPodML+MctMKy4g1Mm+czQSD+rAX7SPJnING4Srx69TLHxgBUxkY2tHcBEjMBbaMyeRDssE8m7oykoDXf1heAvkv2yo51t5vlgOgqXWZ42aF94sdjVWIEAvmfDoWKjiSzMmMiyzAE4987D8RMOvxwpgcA9jrlktMngiSMieMexme2GhnaKFDcVuxEqY+BURNrVJDBiy3q1BxTfiG6rS/e9Kjp/zfaAYXmTA9Tlgg0jzWdkGmb4AAlYf3jy7lnfE84N8Wsm+U1Wlys2ho9ZWak0I+89p8XGlCafzXqwuJHdnUjcIsIB1PGszF484Di5JV+wqO94zCAXorp2ltVMEPNKRAzysOXHBl3NwcK/z6q6uYC4F8yKkJJVbfI/NykwgsCbr8sMt8V9+DOobzAkykdMYV41RrRvjVPTs00/tC3b0y3HCx1r4ByOnyoJ/IgIEmboP75/m/zw5m4w7NytowCxXanQ9jc3IQfIEMHtl+f5L5H977/S727X/7CWRlTkm3/chTfp//77Ni5elhf/IYXfnA1mdA19RXvh6OqlKDd1ijCoIBNVONB5cFHwgzxBOivve+Cf0uS16tZFKmcHNShcq/DSBfD/9hthb8zVy7NykydoPJvgqVu3yBTXhaIKycfHsPY9erEqX6Bl69e7HdNMtPv8ZdksUf6GJl+X5U0nJJGaojYSefTXP33706nwLUNvxW5OMhSxxlgwsJPA9qUoTWqWqegzs6hEbsrVvzWkjgFn5TohN8u7vzFOfDTWuzuK5TzrhSjyKfC4o6zSnvboWntKbNtY3+6v3ronQOAk43gytxtrYMBVm1mYBgdKJJsQgDF6XDQ3Km+1OnvL7L23zYKwr7kJ/lOypdWfNU1yLbGuMuRv/vJ1R23bVk+m4/sUmY7UNBphXO2l0UJXWaFVZdPC2JM4q2IUGEKhAaS6Yz+wzn0c9qq8ho4InPIMW9qGauWaxlkDyfEKYBfaeuVF7gGnW0sXC3hh7dPqGEVGVxpPOyNz1oBOLcLEG0pok9oi/MmYal97qCV9nSIAJX0xXQPgofe8fZsyoRpByORCH7s33G+No0hKWeCdIbv3sBMJ6GpT5d+tUCCf/D1dCC9Ve4nxpohjJqJWhdwVQHs5MvbJlRjwVBZJKcSa9TM+PaQzOTmFUk+6dfKRb05j04ncdYwdfGs82X6SJuATA7QgrLCThnCi7aNsxh/YIG7V6Nit21C5lGuYEh+KysLA/XpoKNibZJB3eK6SJReM8DPvcVjd6ZOxyeR0JcDZM8zOpg7Cqw7CFNkMq6edSpYdtqZvheNoBIf7vXPI/kSYZFUQJ5JXEu5KeNfGBgkWZdlMSnN4BIzPIWYjO8L7/A0/C8ulzR+1/3ipqnKUvNrPdzsT2Un2M3Zb0oiaJvdXCaEolX1fIn/inOshWh9r/ePS7gdxLY9D6P9xaPtPnHLnQtn9qfZ3p9kfoe3TZP1e7T5SIkdK5EiJfERK5LHZ3yP3e+R+Pz3ud5DucWlvrnHQ0d18696P5p5sJixW+eyrPZqTCDxJo0jx4iuUj05OBpbblxbXy4BtT8MZ7AAMHZomcbUpIokE73GK8Hwqe1SVnJ6S4lazWherqHbMX4tEKobfejE7jRT+zfSGCly28IBQ+VaMgnlbO+4VjCQRfnTyBM+R1UTYKWgyv/HZXaKE/sJG1fltpehrbeDy8V6NqQMuD7AGTXl/7Dy3dK/8R7nFkq5pRRv05Dmfiv/4HZw3VZrKPj7aHuP7RCc/894J8q/tJP/6w3azpXLb7orPP3x/BVXfdpt/vUuvuQPalljMVjTnWPa85Rdk768cmS+t7v2QrQYKT0SPBrx2P+hsuOi62VSp0B1ANTvmB1d29cqmYUKggPl3q+E1Eqb+oWRuloAf7rpF+z2n1SDsNp6irJAQ4TceROxeV+l+j8uckkbixOW8odYS3jYkoisfS0fpM7i2ieP4zpcve//Fe9yllPLlWS9n5Kgm4eqyqwmJBx3VwTXgR4q5Ea1T7hqZL4f9nYb1+tb7OgzjjJhSyPc/vvz28vsfL767/OaX169/enmwe7DP2OtX6KaG5wRWmFiG4dAEXQf70WLhpyb1k9Q3o8A30thKDTeMncC1o9iPF2GS2L6fBka7uKatSCBCZSjsK9L6aTtlPtnaayWzIf6TNyohenkn9D3Fjg5lDkwGft7j2SmLAaOc3Fd6gazIpPP0CenR+evesbh6CWLiAkR7I0XR9dyJ4EdLTmgojYr4dVOsm5Kxg21mLckowM+kpul+t6/Zakavgqj73yL2+4fvnrU/ecXm3knDbLsm7RGm7WInQPT2HxADC8Wuhyd2OH6ArrOkF/xwJDqZdTTPsNjWUUmrBGZpfXcIyKgKwpiZ86xbmm+gEXvJZw+FkXhT1TDm7HCkW5yjYe0ag4OqdvDhwf2wI9RecOmG6T2w+qPevflAU/Eh1qw6kg+urYNAgN8bG90T41Ht3bCREhXQj2TXcWl3QaiINLu/3DMqQFwPIN8KgvzXDo4mRK73X0/i8qBdwppBq3JtZSLbgc+GPSN4qrqalJ+NSPet2vuG19la2WTFLgy7cozjzXcEpCNGUUDewG7M/CDOX6V09ZJW6WrexYv0QEZ3BFyfl91hY6FysiQDWpPd+D6bYQpG5J2ihKzb/LuKAD0CJ9AJcLeUuTZnPxpvLjNI1q5OHlPrfJViAsH2sgW7mXu+AUvN/lt95gRf0mL99fbM6RgHy+k5DnynObz8z74pZH4uuaHsIbf7liZbnHHsmnOcTlsWBKrzKN9pfPCUZEixU2ULRAEfucB5aEXss11A8lmAQ/cqc8eT6wf6es3QoLioM8Y8dEx8iotwTxPUsgB8Sxj+xesCkLa6JgX61Ks363VZNeTPX4SuGfylP1RiwyzRGPuTsvxQf25HthMHoW/YtpFGnr0AstuKkySwqOs4NLW8hW+GHuCnPVvIBC8dRHC3XDjIiLb6LzEdaVUJQulsxg/mGdHFDtDZGIrDVGCHw9AzAye0WnYA0Xwmbtuy/K8xA54XAdRt8cCFp2Pg8257KTd8Msch3wGwaMjjUhjr2AG4e5oDYbXIUP0+x+NdgbNzBjBiTj5f2PuLggankaG0D+9an7lmDXHXyNzkPdYi9zdFeWAH56jfvFn/LcmqlOs2jS1KTtuq3D0cg4MzGcOzrUDeKLo9f7xtKNeTU+C1p3ib1sTiwL+2QuD1gkvFEKGS+vwCLQs0hA/fpHk9OsFusdeiXJXkd4TJfjWdizwjGOUjgMO5BgIZBgF696JMskWGflyftAJPZi0Ai0vxBSgBwzI1I9RM57XpnVrBqWH+owOHbqRfpWtiOl8RTI2nklJZ51NLQ7InON6MDh6n+gDJ0UyBu9Yap0IhYGhnavDTs7C3zPA65N1+ujC659hoE9mxL9CPG3rqVn1XMNfshvMG9VjQlKZpB7ppWjwFWpu3oQhPcWph+j5zEndh6b6J/n0s01Wca2jQSdTAcdxANd/rhDazNBzohmGKJ/fK5mmu7gXO2xc+JncGnnfQILylh6GVW7phBXpgqpdWdZe5/LEhAdp41kzZgbsvnBWptuNNXtEB1gF3mi+V0/ODmed8ioYHNdO4kZABWa0c3kvn8e3bgCFjLr93Utr9QbywGRmVTVMWLXply5e0920OMdMw+NbYHp0vNaEyJKPmdl0BuU4bLU8XDXJy9qw7zVFMZFxG5W3PQQ5sPKikvMw6fvR1yY3P7LUsXWaAwKUWsYSdAofwd7RRAKvAIugygdsp4GvK1YOQWax10FOMa/jiiaBs6wYz7G0Ejnrl2wlgIdu4RHGlOWK9oWaNHo6sHThoK92yvafj4k0D7dwazOB6yB/cVDf3QmbKC1ZYXZDPc0xmLNzSA6dvH/N9Y3q651tb3XYVE7q660NCOzbRPnkQIobyLCgMrUO4zEOXPXR57dgOlOdAg6CL7MGqgPRouBya+hTQIBHeHu9Z9e0KtsZrt3uRgbVbp9KxlLyWhNC5z9tLYnDt9qtpLJZh6loLmuC5K1nABtcfr8oCu1+QFyU3y5Is6TbtG8e5F3bRDYX9feYD6Ma+n0wb76Btdg1UALBO00RfL0hSqTT1wjHDbs2qMyAtv3Ncwoz6DiX6Y4sQreSfX1THIDEvn9zm/q9jK0LZ/ybuni9tGQymhBh7Qr5S0dLezZTcy9B86sqdxR1K1MRW355VHzU+99T43M3N9qbZZIwhaUe28CFRiu0apqrgVpY1HSxs/AaoN/qhBY0fhOP6G4DksqzOWA96wm4s12ZKRe8h3DaAjAicQ4TbvEkAD6Q7qn7USfvdJMQfdOIk8fOHm7z3PJngE3giSRYk7C4difFjsf5kbHBXgSZFqjVlrIvWzWSCjakYjzTEWFnDAzP2NT40W9/BIK257twugVeRAMa+mTgna08uuUxRpqZ6kdPCkxg+5gZB0pu2BnrTpjSf7+znBu87sQt0Bwlhdru6sUI9IP/rm50crxg4GEkTH6juPVQ2n9YSF70H5v0KxrfNNbR1CIRfCmn4b5HMJuwsftOsfoC9ppcvHqRzWdGbg8hoZhCH3iBamb2L/iF2BjU221b3HmTg/7fCoL5Q7KDqcmk7PjhjlcpuqdemLPMmW68Z5dW+aqsb7r+jKJnvjjyta7zN26DxqcF1WS77E3FQ+E22Ssqb+quCxoOU6klEktbXTbnW+xOJQXKZF/hpDRPckYUEuKofsub5JiLf8lKm2jSt51ViSXh3lIgGTEiCPjorkaTbLEbtJ7mFe/i6BJ55l4tJ/zAPk8JX5MiDJXMMjsYan0/4wRTuy33uWdPsPWtawrOmtUQ2w481HT2nGQ6K/zAu8NHvm8VsSPquL/uJdHTuktNTvQLproceySzHhsJ8yK4HAQaYoQ1P3/R4eRorj9c64dDTaHs66dDTfPsCWCgzQB/lwCWh5B6ZJPbgY+V7fMxt9B43FYtSjPvZJ/UomNm5+bfZ/0XjNOwd7+vdwiaXohb6bbE+ZzFzrgfEbhZVxfkTfHYnpcJvGK5U0t5w38vifYqm2g7CcPyMC035VTXuwr+8/l4DrqRIgfhImEnZZnb+/wEAAP//AKMAXP88aW5wdXQgdHlwZT0iaGlkZGVuIiBkYXRhLWNzcmY9InRydWUiIG5hbWU9ImF1dGhlbnRpY2l0eV90b2tlbiIgdmFsdWU9IlJZL1J5MlEva1FQM0NLc3NvYjdtUzlkZExTclNZZVNJTEJoaHRqUDRueVZTWlhJUER3Mjl5QVR3U2k0N21LTStOaExsclIyQS9IQzduODBQL0ova0pnPT0iIC8+7BvLkty28a6vgHlwLgKHAN/S7rpsuWI5JVdUjqOqnBQQBIf08hWSs7ujlKvyG/m9fEm6AXKG5MxKK9uxveWsRkOw0QAajUY/gJ4nZPZ3keyGoamJLEXfX1rJUNNGDoUE0KxMU1FvVUeGpimHom1VOivS+tYiw75Vl1a/S6pisJ6QxZ/oCkFLkajy0vpbsyPVrh9IokhfbGvoqaihM1KJa0WajrRd0za9IjLHIXuLpGIQNC16kZSK3hZDfrXq/qK/2Zox8iJNVX1pDd1OWSRXxTYfLi0WWOSmULdfNHeXlkMcwgKiYarriwbQmc0sclukQ26Q9YjYgsqmapta1cPU58imiUUTe4ZO9LllCLtoxZCTrChL2u1KYIq6UXWTptDvpfVNYPuE2aEvbO7DB6hxGDwp/M+5vQbD542bU/cNNnlXedDYAQAHuAAIfDQmTMn2X3Ld7QxKEerrti98YoeRR3z97ZBAI+GIL5hjc+AI9MEMDjz1cN94thcHgBrM+9XdejG3mV/aga7+HNF1GxzX8W3PgVc/920WSzvGRnbgc2oHke7aY0hZGGB7Cu1nRI+dU5t5cUnH/udMAUZ5sc2BWxS7P6miWKdbYs/vrKuLDa7H1ZOLDYjJUnQuNkb4sTJruurKANPiBvHGgllTKE5rn9KsVHcEZE3RcmuedyVBIG26VHWUm5dt19xSx5rGvEjVIIqyP/QDgp42tzUZ4bRTvRoObw2IZyn2JKVFXRa1oknZyGvrOIOLfldVotuf2brWuB1E3zbtrp2Ed74PYY/1BewoIqBJU/fWnDW/gR11rRKR0LzpindNPYhysblwI0UkFkxvJiOVDnXJ8d1xHOK++wZfPoxWMZc4H8S6V5Tg1azEKCwativX60wrVe/I4o32oDn7YY9aQnPrGWGh395ZKzkti7XOA6A4GaAYVHWviibfo4RVIFZ0V5eq72lbigHFfq2u9Z9esQmjR/JqGKN/Wgl5Fj/vVHZp5cPQ9s82m1T110PT2ltQ17vEhkW/f5CtoLIs5PWl9a0CqSyGpts/JU2raoLanox9WacsIOTPB6wvDdYpmzZivemX3Dxh7j2MtcYpbl7vv2s6mb/CrVAX9XbT7gcE0PIA6cTtphL9oLpN2sh+0ze7TqrN234QIN2bohJg2TZls23stt6ezOxLGLNsRLqiezERM42ZDO7Kg6LZjDpk3DJGj03qbER6Mio1nBqaXNiO6m44bEfY2TRp0j1pqUNA8yT0thMgSZ1eM23tKTYgZFv0A6BqxYfK7qDtjhoTEamEPQ/NW+quVE0rjt6H6QaHgt1WVFvSd/JhLEcaP4Lnn8ESXY5asQRddFwL2M1A0YyXZ6zBkZVPnpxR7O/X58t38Ky6a4sUoNK+31Xt0FCt7CestBBA2qT8Jo2v903eDNdqD6QvFfufoBf0p7AbPZtJN5lVWfQ7W22CKz4zOtrGaTMmm3JX1UTURUUzAcYO/LUMGK1HQJrfN/7Exk8oJX+w/k4ovdLliw0KheiUABLvqvZK12yaFo3R1WiOL/B7ohHUFw5GJw7puoOYGoKt0ZpdWliSqgWxy0UHy3Bp/fW7P9LIIpUa8gaYvVXD3JYWdbsbpqGwa5h2PXTNaNfFDj3UDqzCNGlqWpxQVagynbxhs6NAiUqVNyVI9pI7n+aqLIv2+Xv4R3DgrJG7fkbr6LFrIZAl+MrjauKsF274A8wuOA2aDfD5qtHyfJ9TtBCccUdMaubJUbs8OVE4pn0linq5j+b1o06aK42saVBf4DJAU9AK4GNpG0mzXVmCHoHtBVPqixuYDSwUTBtxYYpFnTV6UvPutGUB0YBmYNSg0ULG0VPrFLovygDKLcJMGdXRAVo3x9fvIY4psv2k2iaUCZyo4VaBfQIpDEibgFtYYSkLYFJl02n9SXsFVKe4qUflB0ZsrB8BR4xJDRx9ixKVr3YggLB6OSdNJ/REGdcPoMz/KLorTXKCVTMvtiymwfV+gC9T/6ls2v1zwh3OyFfF8HKXPCVf19Je2tt7m4PVXbkQoMRn/sNG1RvztoGFVLRtwGfYjyBgJTgptMmAWd1NIdUo+vkeTPjkXfzz03/smuE5RmPDW9wn5v2ZeYhalHswFL2tEQzwqXm0Yo/meMQf+5FiUFtwVRa9/FEL7aKx0UgLrG2DG1zTvEDVGmCBiSLy7BQRvOJtUYMY19u3u27ZZOLfjHU/v/lcULMDpr8tJvbUsDt/+OF0AWgOruOlFfoR5zKVcZQGUZSxyGWBFEmaqYSFmefzIBOSqyDO0ohlQcjiLOFJJpXyoyyRICnfITvQG/pFJKvtihsh9xT5oMBnHx6DaI1EP0S4zqE+YvEKROhzJ+FB6Huhk/mey91IAshx/CiM3CAMlB8mjgyzVPnSzxwpZZQpT7ppkvHIunptGPJxAvbbFwmwIruuGB4kE2dxH7FQxF7AA5UyNxCBZEw5QigZpnGUecJT0kmjOAlk7LluKDI/kEkcezFPfHB2vZAlmbVSILNpTqyyrv4yln6KZrq9vR0VE7Jg1+shHoPCMeQ+SLZOMR+xZHmoYUTsuyJynTR0My9RYeqrlDMRcinTwE+YH4OVc0PHYzII0lTFXgTi5nMuMhAbzY+zQnNQLMcTErMmT4nhOsQP7VOi2folzHItpis7Z10h0nIkc2xgimIRhrxsKtUClyCkKAZ0ro1PZy3d88NdQTr5nyDZ+qCSVNqn9O7fO0eX8gOnjdz7iNNGRP7400aIkK+pIe1Bp/gRcV64th/h4SAZC5HURY9wm8cksH2X+LYXktD2Y9uzHXj61Gbm4UILeImpjefWEddPPPqmHEq2G8LTd6kBBIAWe4CCp9sIjMw7NIOpA4ADgPl4vk5t3/QFre3AxSJhthMhiQwaQxs75FhEeBTCB+h1ATcACrG17gwqfKQHQMwOAcApoOCQsSm4duzjDADR1af5MVLDNRWOxrfdADvg+ApwrlllBzAGDA2jICXYI0PCXMDnQAl+wyz0jYELFMWEE6xgyAzoTTMXJz49bc+DSvgHImHHOBjBwYB62w8Qj2GPGp/5eq30lGDWrh36OBef4FxgyfAmwffsUK8hLIWhApmieRgbZsK4OA8OQ8Lqwir7sJafR1DlEvON58VADogD9WwPGedHVP87PUI+HOr9qjHdx5iofte2TTfM9Mpng9j2l2kzQJkatfAYTJY+WJDDQ2zWOdRHbLSCIA64EE7mBDyOIQDjXLHUZ1JmUrh+mkTMz7xUqszxndB1Qh6xwIviMHMj0M8gFS8MQ8Y4/6d4PDMWQSQigQ+PQXZeG1IfIjvnUB+x7KRpoOKMxZ6MnFhlSZh4keckfhbziIcyZCLIMplFWej7fpy5iYd+kAtOt2SArXR8hQxZC82PDt8fg7yItniIrKzRHrGcMB4zEAvhoHJxuAxY4idxIF0vCzJ0l73M8yIQHRCZ2EtcnoZSepkKYs4j5XPr6vPXX/94GRmPAR+bnAydKOoHKpazuI9YYjIwMWmasczxZOoyx3EC1xfCcwLBElemCkIn7sRRKhiLmQyl56gQtApYM6ZcJqyr70aO/Az2KMFLt0cgMEjnQ4TlBO8RC0oUJk6QoPGJHZ75LPOkG2Wp57oQZTMGUsOyMEiV8DiE48J1Yskc7qTckwEPeWxdfQFE/LiIWyTNbhhDbl1+z9GQqb/6HB/nou755fJpetE5p75NaHC4Cp5dmy9vlABzKES3n19lr3MADrf/EMyK78UdVV0H7SvV9zrmHzs+rQK6RJ+bb1NjERO1T3T96plDogQGPCyKtznXaWveLI0MglqIKZ1XDMJuCE1dCLD5vBraQGga5RBkgle8qsGoN3p17PddhfEzxuGhE0oINmOKUTiGsR4PxzLEpFFAnBLiXQxYMfqPlil1jHmmzn8JgSiLxLIW42/PxROD0H8V2J6PAbPjhe++iaE7AdGrzu6jGLmaFwfjcSAOMw/9ZTKhzhokzo1ORjxCDRDHOJ8INd1Kmwto83I8LNLiou+q8cZ8JlZp0VdF3y+vwL80QGLk66HHRL+AaN09SKxcPGHBr3kyJiwIrPGriAR27JUuygp+zfIqDQp+vYqxFGksssaiR7SIIGJJz+A5Bk9/vcIhSaRJIp4dRuskUeeePMljKgC+YeauFPV//vXvgbSq0/kXQy6GMeeCQGnIi54MRaXs5U3/TL3B8oNBKBXtQaHgffxMoYxq5VZ09VhMRF2rg4rRZPxfz/y+9czc8J0XJmryymlRH2wTyu4x21wnDIq6GXJMaRcJpp2b/EabHDxC6+pbhR4bmm40/QDsFMjoHtwaMo5nz5PFPoIy4zickAZgJO9npuzgZUARk/1KcDq14de33WcSzX56Phspc+g4E7tymHsmbVfo9LX87m3XQ0CGOZ6rzDaTyDOZj7lReKGtx0Tm/zKvzZA3dTW+VU16SEI++dWETkOb/XKiwqTJ2fsh+UgkPYwK/O9QTQHW0LQQh6ys5r3TPs35mqeD/+oa8XdmHtcGcmnogCkNxHe0bwu0YaTa0wCdn6WkTtU4wHGbvj/bDnMnzTY+8+uE102Lwo4j5ViQokvpmBh3KoeH7HPww6DH/TOCVw7PURNh/uH4aqEqAi2kUHJOMuvGEQ8Rwuqd0qQBHlW0VNlwWlmKDtrgRjWKos8FaDsDXuXGu4HT3j23zsQzZ3RbjXnvA0ZxbdIYHrR7Ou6w9RTeqa6ZY44phx/ct+NyLQdZbl8AHPbLiwnvC6M9QEXgOq2aV2A/Wz3EMXu/AbGB5ZkUAE4lUypNBIaoMEah0k/GyrEZiE6nxmTY29+SD93iEcFq2lCCGY2qkz9Ii+gdyhb+hCnfuNIhtgeejOsGBgSf3Fu6GXj5h8jUXXeB/3MK6O8qBLtvwBXJ3TdeTl1wZLgdhR54bkG4bod3sTxeOVSO1nT+TWz7SFUcBOYXVgcc/Moj2181Yx56hBzog5ZLL0xfunroPDF//qMsJIFoEuY/IWM8sDkLkYDVT6WwnNPoFE4P4646wpvb8F5V+OtLlsyVvF5JFoKMYM1zbndSgv6ZkhYeKnKwImEEFmJpSPQlMViZEFkVrn6YR0e7xtEExTaP1jZvtECwgGj3QDo4Dc6axvsitKXyWOjGmZUY87IxVR5r8qHCtI//AgAA//8BAAD//6GS5DlVEQIA){height=\"60px\" width=\"240px\"}"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Tutorial 7: Deep Energy-Based Generative Models\n", " :card_description: In this tutorial, we will look at energy-based deep learning models, and focus on their application as generative models. Energy models have been a popular tool before the...\n", " :tags: Image,GPU/TPU,UvA-DL-Course\n", " :image: _static/images/course_UvA-DL/07-deep-energy-based-generative-models.jpg"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab_type,id,colab,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7"}, "papermill": {"default_parameters": {}, "duration": 11.264697, "end_time": "2021-09-16T12:40:49.242921", "environment_variables": {}, "exception": null, "input_path": "course_UvA-DL/07-deep-energy-based-generative-models/Deep_Energy_Models.ipynb", "output_path": ".notebooks/course_UvA-DL/07-deep-energy-based-generative-models.ipynb", "parameters": {}, "start_time": "2021-09-16T12:40:37.978224", "version": "2.3.3"}}, "nbformat": 4, "nbformat_minor": 5}