{"cells": [{"cell_type": "markdown", "id": "0909ce95", "metadata": {"papermill": {"duration": 0.013689, "end_time": "2023-03-14T16:13:28.420779", "exception": false, "start_time": "2023-03-14T16:13:28.407090", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 10: Autoregressive Image Modeling\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2023-03-14T16:11:45.773564\n", "\n", "In this tutorial, we implement an autoregressive likelihood model for the task of image modeling.\n", "Autoregressive models are naturally strong generative models that constitute one of the current\n", "state-of-the-art architectures on likelihood-based image modeling,\n", "and are also the basis for large language generation models such as GPT3.\n", "We will focus on the PixelCNN architecture in this tutorial, and apply it to MNIST modeling.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/10-autoregressive-image-modeling.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/Lightning-AI/lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)"]}, {"cell_type": "markdown", "id": "3a32016d", "metadata": {"papermill": {"duration": 0.007479, "end_time": "2023-03-14T16:13:28.436124", "exception": false, "start_time": "2023-03-14T16:13:28.428645", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "9b7a79c3", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2023-03-14T16:13:28.485857Z", "iopub.status.busy": "2023-03-14T16:13:28.485400Z", "iopub.status.idle": "2023-03-14T16:13:31.786568Z", "shell.execute_reply": "2023-03-14T16:13:31.785231Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 3.344638, "end_time": "2023-03-14T16:13:31.789363", "exception": false, "start_time": "2023-03-14T16:13:28.444725", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\r\n", "\u001b[0m"]}], "source": ["! pip install --quiet \"torchmetrics>=0.7, <0.12\" \"torchvision\" \"setuptools==67.4.0\" \"seaborn\" \"pytorch-lightning>=1.4, <2.0.0\" \"lightning>=2.0.0rc0\" \"torch>=1.8.1, <1.14.0\" \"matplotlib\" \"ipython[notebook]>=8.0.0, <8.12.0\""]}, {"cell_type": "markdown", "id": "92f2e144", "metadata": {"papermill": {"duration": 0.007538, "end_time": "2023-03-14T16:13:31.810820", "exception": false, "start_time": "2023-03-14T16:13:31.803282", "status": "completed"}, "tags": []}, "source": ["
\n", "\n", "Similar to the language generation you have seen in assignment 2, autoregressive models work on images by modeling the likelihood of a pixel given all previous ones.\n", "For instance, in the picture below, we model the pixel $x_i$ as a conditional probability distribution\n", "based on all previous (here blue) pixels (figure credit - [Aaron van den Oord et al. ](https://arxiv.org/abs/1601.06759)):\n", "\n", "
\n", "\n", "Generally, autoregressive model over high-dimensional data $\\mathbf{x}$ factor the joint distribution as the following product of conditionals:\n", "\n", "$$p(\\mathbf{x})=p(x_1, ..., x_n)=\\prod_{i=1}^{n} p(x_i|x_1,...,x_{i-1})$$\n", "\n", "Learning these conditionals is often much simpler than learning the joint distribution $p(\\mathbf{x})$ all together.\n", "However, disadvantages of autoregressive models include slow sampling, especially for large images,\n", "as we need height-times-width forward passes through the model.\n", "In addition, for some applications, we require a latent space as modeled in VAEs and Normalizing Flows.\n", "For instance, in autoregressive models, we cannot interpolate between two images because of the lack of a latent representation.\n", "We will explore and discuss these benefits and drawbacks alongside with our implementation.\n", "\n", "Our implementation will focus on the [PixelCNN](https://arxiv.org/pdf/1606.05328.pdf) [2] model which has been discussed in detail in the lecture.\n", "Most current SOTA models use PixelCNN as their fundamental architecture,\n", "and various additions have been proposed to improve the performance\n", "(e.g. [PixelCNN++](https://arxiv.org/pdf/1701.05517.pdf) and [PixelSNAIL](http://proceedings.mlr.press/v80/chen18h/chen18h.pdf)).\n", "Hence, implementing PixelCNN is a good starting point for our short tutorial.\n", "\n", "First of all, we need to import our standard libraries. Similarly as in\n", "the last couple of tutorials, we will use [PyTorch\n", "Lightning](https://lightning.ai/docs/pytorch/stable/) here as\n", "well."]}, {"cell_type": "code", "execution_count": 2, "id": "e92a48cc", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:31.827827Z", "iopub.status.busy": "2023-03-14T16:13:31.827450Z", "iopub.status.idle": "2023-03-14T16:13:35.076698Z", "shell.execute_reply": "2023-03-14T16:13:35.075996Z"}, "papermill": {"duration": 3.261303, "end_time": "2023-03-14T16:13:35.079667", "exception": false, "start_time": "2023-03-14T16:13:31.818364", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Using device cuda:0\n"]}, {"data": {"text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["\n", "import math\n", "import os\n", "import urllib.request\n", "from urllib.error import HTTPError\n", "\n", "import lightning as L\n", "\n", "# Imports for plotting\n", "import matplotlib.pyplot as plt\n", "import matplotlib_inline.backend_inline\n", "import numpy as np\n", "import seaborn as sns\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "import torch.utils.data as data\n", "import torchvision\n", "from lightning.pytorch.callbacks import LearningRateMonitor, ModelCheckpoint\n", "from matplotlib.colors import to_rgb\n", "from torch import Tensor\n", "from torchvision import transforms\n", "from torchvision.datasets import MNIST\n", "from tqdm.notebook import tqdm\n", "\n", "plt.set_cmap(\"cividis\")\n", "%matplotlib inline\n", "matplotlib_inline.backend_inline.set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "\n", "# Path to the folder where the datasets are/should be downloaded (e.g. MNIST)\n", "DATASET_PATH = os.environ.get(\"PATH_DATASETS\", \"data\")\n", "# Path to the folder where the pretrained models are saved\n", "CHECKPOINT_PATH = os.environ.get(\"PATH_CHECKPOINT\", \"saved_models/tutorial12\")\n", "\n", "# Setting the seed\n", "L.seed_everything(42)\n", "\n", "# Ensure that all operations are deterministic on GPU (if used) for reproducibility\n", "torch.backends.cudnn.determinstic = True\n", "torch.backends.cudnn.benchmark = False\n", "\n", "# Fetching the device that will be used throughout this notebook\n", "device = torch.device(\"cpu\") if not torch.cuda.is_available() else torch.device(\"cuda:0\")\n", "print(\"Using device\", device)"]}, {"cell_type": "markdown", "id": "85262aec", "metadata": {"papermill": {"duration": 0.007715, "end_time": "2023-03-14T16:13:35.100160", "exception": false, "start_time": "2023-03-14T16:13:35.092445", "status": "completed"}, "tags": []}, "source": ["We again provide a pretrained model, which is downloaded below:"]}, {"cell_type": "code", "execution_count": 3, "id": "4e3094e2", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:35.116976Z", "iopub.status.busy": "2023-03-14T16:13:35.116582Z", "iopub.status.idle": "2023-03-14T16:13:36.641309Z", "shell.execute_reply": "2023-03-14T16:13:36.639868Z"}, "papermill": {"duration": 1.536466, "end_time": "2023-03-14T16:13:36.644342", "exception": false, "start_time": "2023-03-14T16:13:35.107876", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial12/PixelCNN.ckpt...\n"]}], "source": ["# Github URL where saved models are stored for this tutorial\n", "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial12/\"\n", "# Files to download\n", "pretrained_files = [\"PixelCNN.ckpt\"]\n", "# Create checkpoint path if it doesn't exist yet\n", "os.makedirs(CHECKPOINT_PATH, exist_ok=True)\n", "\n", "# For each file, check whether it already exists. If not, try downloading it.\n", "for file_name in pretrained_files:\n", " file_path = os.path.join(CHECKPOINT_PATH, file_name)\n", " if not os.path.isfile(file_path):\n", " file_url = base_url + file_name\n", " print(\"Downloading %s...\" % file_url)\n", " try:\n", " urllib.request.urlretrieve(file_url, file_path)\n", " except HTTPError as e:\n", " print(\n", " \"Something went wrong. Please try to download the file from the GDrive folder, or contact the author with the full output including the following error:\\n\",\n", " e,\n", " )"]}, {"cell_type": "markdown", "id": "00c206b5", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.010905, "end_time": "2023-03-14T16:13:36.676666", "exception": false, "start_time": "2023-03-14T16:13:36.665761", "status": "completed"}, "tags": []}, "source": ["Similar to the Normalizing Flows in Tutorial 11, we will work on the\n", "MNIST dataset and use 8-bits per pixel (values between 0 and 255). The\n", "dataset is loaded below:"]}, {"cell_type": "code", "execution_count": 4, "id": "87fecfbc", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:36.698033Z", "iopub.status.busy": "2023-03-14T16:13:36.697651Z", "iopub.status.idle": "2023-03-14T16:13:38.160315Z", "shell.execute_reply": "2023-03-14T16:13:38.159384Z"}, "papermill": {"duration": 1.475161, "end_time": "2023-03-14T16:13:38.161551", "exception": false, "start_time": "2023-03-14T16:13:36.686390", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n", "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /__w/11/s/.datasets/MNIST/raw/train-images-idx3-ubyte.gz\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "31d90245833f404f8fd0b298c7ec9b70", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/9912422 [00:00 only make them a tensor\n", "transform = transforms.Compose([transforms.ToTensor(), discretize])\n", "\n", "# Loading the training dataset. We need to split it into a training and validation part\n", "train_dataset = MNIST(root=DATASET_PATH, train=True, transform=transform, download=True)\n", "L.seed_everything(42)\n", "train_set, val_set = torch.utils.data.random_split(train_dataset, [50000, 10000])\n", "\n", "# Loading the test set\n", "test_set = MNIST(root=DATASET_PATH, train=False, transform=transform, download=True)\n", "\n", "# We define a set of data loaders that we can use for various purposes later.\n", "train_loader = data.DataLoader(train_set, batch_size=128, shuffle=True, drop_last=True, pin_memory=True, num_workers=4)\n", "val_loader = data.DataLoader(val_set, batch_size=128, shuffle=False, drop_last=False, num_workers=4)\n", "test_loader = data.DataLoader(test_set, batch_size=128, shuffle=False, drop_last=False, num_workers=4)"]}, {"cell_type": "markdown", "id": "14dcf62f", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.008628, "end_time": "2023-03-14T16:13:38.179628", "exception": false, "start_time": "2023-03-14T16:13:38.171000", "status": "completed"}, "tags": []}, "source": ["A good practice is to always visualize some data examples to get an intuition of the data:"]}, {"cell_type": "code", "execution_count": 5, "id": "fd793b34", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:38.198314Z", "iopub.status.busy": "2023-03-14T16:13:38.197980Z", "iopub.status.idle": "2023-03-14T16:13:38.357970Z", "shell.execute_reply": "2023-03-14T16:13:38.357352Z"}, "papermill": {"duration": 0.170991, "end_time": "2023-03-14T16:13:38.359132", "exception": false, "start_time": "2023-03-14T16:13:38.188141", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzQxLjY3NDgzODcwOTcgMTgwLjcyIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nFWOSw7CMAxE9z7FnCDfKkmXQKWIZWHBAaJQiCioVKLXx61AhcWzPJbHHtnk1zXlQ9xidyS5qjSSRmE6KBRmgkZkOlKserKVFs5XwdYsb79SByW84Zla2wvRmQZ4YRas4TpvB69qD+2csAbPjBPukBv+MvKrwkx8PeI/2LD4HeYgH+v3cOoh9xrNAy219AYPKzF0CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQ4CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDEgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCA0NTUgL0hlaWdodCAyMzEKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjIyICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vJycnHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKwsLCvr6+urq6tra2srKyqqqqpqamoqKinp6elpaWkpKSioqKhoaGgoKCenp6cnJyampqZmZmYmJiXl5eWlpaVlZWUlJSTk5ORkZGPj4+NjY2Li4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKAgIB+fn59fX18fHx7e3t5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5qampnZ2dmZmZkZGRjY2NiYmJgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NRUVFQUFBPT09MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj48PDw7Ozs6Ojo5OTk4ODg3Nzc1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4sLCwrKysqKipcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovQml0c1BlckNvbXBvbmVudCA4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyA0NTUgPj4gL0xlbmd0aCAxNCAwIFIgPj4Kc3RyZWFtCnic7Z37nxZVAcbZlQphIVuMELQ0tUASgVq20iKoKEouEXQxW4LASLpIUWJYQRchFoKSpTQNdEOWDYtNkizUoFrSYLm4UCjwt/Q87MyH2XnPmTlzeZfl8Hx/AeedmWfOfF98z5wz55wB3xQ+MOBiX4AoBXn0A3n0A3n0A3n0A3n0g9Djub5CeVXJk0c/8uTRjzx59CNPHv3Ik0c/8uTRjzx59CNPHi10dna+HzwB+iQvL/KYjDw65fb7PHl0yu33eZepx4Ng+/btdaAGDACTJ0+25xbNcyZPXjeYOnUqCzILVD2vCPJoRx7l8RL2+F0wHdT2prGx0Z5bJC8TWfNeAlNATQ9rQFXziiKPZuRRHvPkFUUezVxOHv8HWBdoaWlpAINAbSVXXXXVT4E5t7yCpJA1bwYIHF4NWNSq5hVFHs3IozzmySuKPJq5XDz+ExgqNQYK1HMeAmxKWLhwoX0n3uJXQfKpspbvnSDwaPgWppM172fgJoAbxsw7AUt/7NixTHnyGEMe5TFPnjzKYyQvm8eVK1d+HKQ7JM3NzXnKiRIcuxXwFJs3bzbs8UewdOnSN4HxALWtltRyupXv3AWPdXV1W4HzcZnzjh49Og+wknhFDyxx8Ncrxo0btx0458ljDHmUx0x5fe7x7Nmz/wHLweDBgxPEDRs2bCd4JuDMmTN5yvkvEJ5x7dq13ES1vITdu3d/GYwAkdh7QGo5029IQOhxypQpzsfkysOPIIW9GfwY3H///VGPADd78A7glCePMeRRHjPlyaM8VuY5eOzu7k6QBz4BPgXa29tLKCfqNpvDMw8fPnwBeBswR78XsKW3QF6UBx98cCCgx0WLFrkdkyfvHwDPjLTVHty1/fv3/zJgKghcXg+6urrS8+QxijzKY5Y8eZRHc14hj42NjXx/5RVQXjmpha2MfBU2lnfllVeyjAtBsCW5juOUF8IXcyZOnBi0rdZ0dnY6lylz3mqAq/8KMLQQcxO+UfgSD2cZm5qa0vPkMUQe5TFrXp96PAX4UU+nSpR6MAkcOnQodgz/x9+V+D905/vKR9FVq1a9I4CPr9u2beMnnwW15x9Xh7EXLfk0znmbQM0FAo9bwMM9/BqknyY17znAGzhy5EjeLcte+KdxI+CVoOaRniePIfIYQR7lMYY8luSR/V+VdZsbQCtg5WZtBaycfBTw74888kiecibzGGDHI67ErXvQOY9vcEY8Tp8+fSbgQ1ywhT2F3LQo8dEyNY8txEEPY8LV4A6HXZIbN25M2FEeY8ijPGbJk0cij+a8bB6HDh3KHjG+mDcbVFqOMmLECFYPspYzmfeAoKp1AKQfkNejnWtAR0dH3rw5gIpWr15t2eMMmDt3Lne6DZw+fTq9fPIYIo/ymCVPHok8mvMSPIbttBEGDhx4LTA0YZvho3obyFLOZEKPS5YscTsgr0d8C0dWgi/yUA5YQGWv09ySnph3FFwHqMh+JRynH/Q/Jr+bG8mTxxB5lEfXvIvikaVxspXMk8C1nCls2bLljeB94MSJE27H5PF4Bzh+/LhhJz45801F7PQCyJr3bxC+1lj5WHgYLAavB8FO6ZctjzHkUR5d8+TxPPJozbtkPOKJrQMOecZMczDm8fg9YN6JoxPGAKi0dB6m5vH5kaWIdCvySXjTpk2sQEVu3MLE0dixPHkMkUd5zJInj/Joz8vtkb1xEyZM4FstHOlGW6gdxHbio/NukKWcFn4CgrP+Cjgfl8fjh0BlPQdbjnMmSNZzdu7cmTdvGWAVZsiQIR8I4EBAbGEdjuMr3g3wn/JoKGf6jvIoj1ny5FEe7XkJHpN7GPmSzKoe7gXmnb4BspbTAB+QOcMKzsi1UZIHyuXN+wyItJPju3IoeD+X/YFtbW0fBvxkLCiQx2m47gb19fVhkwD/TeCrEw4iD7SuA87lk8cQeZTHLHl97tH83qM7t9xyC9+Bz1pOA3l/GzPlcVKQC3Pokg8CDofmXY9sXgnKyQsHIP8OBFvZAst/IrD8InAunzyGyKM85syTR3fk8WnAt3Gy+hsNeGXmGdrzePwY4JlHjRq1H2Q6NlPeqVOnbgY1Rlgb4YDw10BJeZX8ArDq4zxdiDwakEd5zJNXiTwml9NtZy89kt+CLCo5D8pvQGquc8FeBu8CPPu0adOcj8ubx4oUvzcRgXxfZsOGDRxiXn5ejDuCRmrUftwOkEcz8iiPefJiVMkjaWtrS5b3VvBkD6eTBwedy15OTirPEP7q2kc5lZdXlAJ5fJOSz478fZTHgnlFkUcz8uiUJ48l0389lkvWvE8DemQFoC/yilIg788g7JKUx4J5RZFHM/LolCePJSOPBp599tnXAXrk/O8LFixImCKxhLwSkEcD8uiaJ48lI48G5NE1r3973Lt3b9jYwLVYnn/++ermlUCBvP+CzwMuULZnz55MefJYMvJoQB5d8/q3R+W55smjH3ny6EeePPqRJ49+5MmjH3ny6EeePPqRF3oUlzby6Afy6Afy6Afy6Afy6Afy6Ad6fvQjTx79yJNHP/Lk0Y88efQjTx79yJNHP/Lk0Y88efQjTx79yJNHP/Lk0Y88efQjTx79yJNHP/L6r8e/gYd6uBpw7FxkfcTm5uaS8yrhOiy3gh8A54Mq854BE0BNTc0N4JPgO+CBBx5g2TgzL1eymjFjRjjNJNeBnDlz5hfBUeCUJ4925DE9N31HeZTHPHmVyGN6rn2HbYCLInIZ4oG9CabtIPX19dzp6+AUKJBn50eA95VzMDofFMt79NFHx4Hvgw0bNiRf6lnARSZ5B1asWMFlIIeAOXPmcFnm5Dx5tCOPCcijFXmUx8wed4HYps3A+fjkvNbWVhiqD6TV2j1e2FK7GOTNS4TLJNPjG4DzQbE8iHsF5Inn3NJcJrXm/KKittXZ5DEVeUxAHl2poseGhgY+3MS2chEsBjqdIjmvq6vrOkBFXH1xSm+CLW8HEY+8zfYnvIvssQBcSPkLQB7T8xKRxwTk0RV5dM9LxF+PBw4cMHikQ8PmxFz7Dg+DdcC+RwdYtmxZtDJ033335c2z0N3d3Qjo8RrgfFxJHvft2zcZMH7evHn4encl58mjBXlMRh6dkMeseRYuQ49sByjPoxutra1Bj+T5LsnyPT7++OPhOp4/BM7HFSgf50N+DnwboGrF5UPZfnzkyJH0PHm0II/JyKOVvvOIH0I+LBo+uQgeq/v72JceHwN33XXXWwDz+GdTU9NfgHOePFqQR6dctzw78tg7Tx4teO2xoaHB0P9I6BG1IOdctzw7Vfc4ceLE6nqEO8hqCtfvwq3le49cZPo1kDVPHi3Io1Ou83VakMfeefJowWuPs2fPtnzCNlan13QuFY/jx4+vrsdJkyaFAXNAW1tb5muM5MmjBXl0ynW+Tgvy2DtPHi1465GvidpfwuGndsuVuZmu1cD8+fMvcY8nT578A/gqGAlQiOsBh9EdBlnz5NGCPDrlOl+nBXnsnefu0d6Cyv7Ha3uAytnckX8fcIGI4YIeT4A7QTAsmScfCrZu3Wo5IE/edlBXV0eHHAr9d+B8bJ68lwCU3gs4wGo0wFeH4wKc8+TRgDzKozw6IY/mPDePNNUAYptQtxnQG1Z3+MkBsLkXRcoZ4WkQGTdHh7eD1HJmCmkGKAw9LgWZji1YPnY6Tup5tHSbJkQe7cijPMqjE/JoznPzyGoL1ezatWtxRB6qL2xXDT7ZZa0IGXKdChWno6ODM0JFPH4OlJv3KmB7J24k6xtuL8kUyDPA2TmgEjWtOk7d4ZQnjzHkUR7z5BmQxxLy/PUIcfTY0DOonH9YJlopz+M+8C2wZs2aVjAf8E88k0cccp6ohImesuRFYO0iaB+/DTgflzfPwuHDh9l6zrFzTnnyGEMe5TFPnoWSPR4IRsoFTabmdx5DzCMGzLmWT1taWviTdzNIns8KDveDonmVPAEGA3hkif8KnI+153WCTKcBXwO8CU899VR6njzGkEd5tOfJozyW4TEQ6DajIx4nLaMGKnPtn9ZGuhZrexPZMnbsWM7Snn5ReeodowA8zgXs78x0bGXeEuA83VcETkvC6taOHTvS8+TRgDzKozwGn8pjQJkeM011HDS1Ju9kz2NdINJ6mlzPCeCCJnnzzPwccMYT3MD0xk2nvI+AqWDPnj2ZzvR7QI/Lly9Pz5PHGPIoj+a8/uCxfOx5L4De0tI9lt9vxQWtgsbVl0G2whnz+Pb/HWDQoEFsLuViVivB+vXr+b6j5TR4Nr4JsIzt7e3pefIYQx7dkEd5lEdrXj/yyDcqFi9eHPN4I/gSeBE0Nzdzbttgh+r0PwZ1nJpp06ZlK1hiHh9D8RzIJlOOjmPAmDFjOOjg9gBUhNYEcIb90aNHs4wrgFOePMaQRzfkUR7l0ZrXjzySjo6OEeBuQI8o9p9AZAfekHU9VKf/kYPyyvcYgaPwuK7KwYMH94J7QE0ls2bNSlh8pTJPHmPIoxvyKI/n5NGW1888Ki9nnjz6kSePfuTJox958uhHnjz6kSePfuTJox958uhHnjz6kSePfuTJox958uhHnjz6kSePfuSFHsWljTz6gTz6gTz6gTz6gTz6gTz6wf8BhQQ12wplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjM0NDQKZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjcuMSkgL0NyZWF0aW9uRGF0ZSAoRDoyMDIzMDMxNDE2MTMzOFopCj4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDUxNTUgMDAwMDAgbiAKMDAwMDAwMDYwNyAwMDAwMCBuIAowMDAwMDAwNjI4IDAwMDAwIG4gCjAwMDAwMDA2ODggMDAwMDAgbiAKMDAwMDAwMDcwOSAwMDAwMCBuIAowMDAwMDAwNzMwIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM0NCAwMDAwMCBuIAowMDAwMDAwNTg3IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU2NyAwMDAwMCBuIAowMDAwMDAwNzYyIDAwMDAwIG4gCjAwMDAwMDUxMzQgMDAwMDAgbiAKMDAwMDAwNTIxNSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDE2IC9Sb290IDEgMCBSIC9JbmZvIDE1IDAgUiA+PgpzdGFydHhyZWYKNTM2NgolJUVPRgo=", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["def show_imgs(imgs):\n", " num_imgs = imgs.shape[0] if isinstance(imgs, Tensor) else len(imgs)\n", " nrow = min(num_imgs, 4)\n", " ncol = int(math.ceil(num_imgs / nrow))\n", " imgs = torchvision.utils.make_grid(imgs, nrow=nrow, pad_value=128)\n", " imgs = imgs.clamp(min=0, max=255)\n", " np_imgs = imgs.cpu().numpy()\n", " plt.figure(figsize=(1.5 * nrow, 1.5 * ncol))\n", " plt.imshow(np.transpose(np_imgs, (1, 2, 0)), interpolation=\"nearest\")\n", " plt.axis(\"off\")\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "show_imgs([train_set[i][0] for i in range(8)])"]}, {"cell_type": "markdown", "id": "78c05d07", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.009176, "end_time": "2023-03-14T16:13:38.379326", "exception": false, "start_time": "2023-03-14T16:13:38.370150", "status": "completed"}, "tags": []}, "source": ["## Masked autoregressive convolutions\n", "\n", "The core module of PixelCNN is its masked convolutions.\n", "In contrast to language models, we don't apply an LSTM on each pixel one-by-one.\n", "This would be inefficient because images are grids instead of sequences.\n", "Thus, it is better to rely on convolutions that have shown great success in deep CNN classification models.\n", "\n", "Nevertheless, we cannot just apply standard convolutions without any changes.\n", "Remember that during training of autoregressive models, we want to use teacher forcing which both helps the model training, and significantly reduces the time needed for training.\n", "For image modeling, teacher forcing is implemented by using a training image as input to the model, and we want to obtain as output the prediction for each pixel based on *only* its predecessors.\n", "Thus, we need to ensure that the prediction for a specific pixel can only be influenced by its predecessors and not by its own value or any \"future\" pixels.\n", "For this, we apply convolutions with a mask.\n", "\n", "Which mask we use depends on the ordering of pixels we decide on, i.e. which is the first pixel we predict,\n", "which is the second one, etc.\n", "The most commonly used ordering is to denote the upper left pixel as the start pixel,\n", "and sort the pixels row by row, as shown in the visualization at the top of the tutorial.\n", "Thus, the second pixel is on the right of the first one (first row, second column),\n", "and once we reach the end of the row, we start in the second row, first column.\n", "If we now want to apply this to our convolutions, we need to ensure that the prediction of pixel 1\n", "is not influenced by its own \"true\" input, and all pixels on its right and in any lower row.\n", "In convolutions, this means that we want to set those entries of the weight matrix to zero that take pixels on the right and below into account.\n", "As an example for a 5x5 kernel, see a mask below (figure credit - [Aaron van den Oord](https://arxiv.org/pdf/1606.05328.pdf)):\n", "\n", "
\n", "\n", "Before looking into the application of masked convolutions in PixelCNN\n", "in detail, let's first implement a module that allows us to apply an\n", "arbitrary mask to a convolution:"]}, {"cell_type": "code", "execution_count": 6, "id": "b7f0b172", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:38.398926Z", "iopub.status.busy": "2023-03-14T16:13:38.398526Z", "iopub.status.idle": "2023-03-14T16:13:38.406778Z", "shell.execute_reply": "2023-03-14T16:13:38.405852Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.019953, "end_time": "2023-03-14T16:13:38.408358", "exception": false, "start_time": "2023-03-14T16:13:38.388405", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class MaskedConvolution(nn.Module):\n", " def __init__(self, c_in, c_out, mask, **kwargs):\n", " \"\"\"Implements a convolution with mask applied on its weights.\n", "\n", " Args:\n", " c_in: Number of input channels\n", " c_out: Number of output channels\n", " mask: Tensor of shape [kernel_size_H, kernel_size_W] with 0s where\n", " the convolution should be masked, and 1s otherwise.\n", " kwargs: Additional arguments for the convolution\n", " \"\"\"\n", " super().__init__()\n", " # For simplicity: calculate padding automatically\n", " kernel_size = (mask.shape[0], mask.shape[1])\n", " dilation = 1 if \"dilation\" not in kwargs else kwargs[\"dilation\"]\n", " padding = tuple(dilation * (kernel_size[i] - 1) // 2 for i in range(2))\n", " # Actual convolution\n", " self.conv = nn.Conv2d(c_in, c_out, kernel_size, padding=padding, **kwargs)\n", "\n", " # Mask as buffer => it is no parameter but still a tensor of the module\n", " # (must be moved with the devices)\n", " self.register_buffer(\"mask\", mask[None, None])\n", "\n", " def forward(self, x):\n", " self.conv.weight.data *= self.mask # Ensures zero's at masked positions\n", " return self.conv(x)"]}, {"cell_type": "markdown", "id": "a4de1bbc", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.009681, "end_time": "2023-03-14T16:13:38.431177", "exception": false, "start_time": "2023-03-14T16:13:38.421496", "status": "completed"}, "tags": []}, "source": ["### Vertical and horizontal convolution stacks\n", "\n", "To build our own autoregressive image model, we could simply stack a few masked convolutions on top of each other.\n", "This was actually the case for the original PixelCNN model, discussed in the paper\n", "[Pixel Recurrent Neural Networks](https://arxiv.org/pdf/1601.06759.pdf), but this leads to a considerable issue.\n", "When sequentially applying a couple of masked convolutions, the receptive field of a pixel\n", "show to have a \"blind spot\" on the right upper side, as shown in the figure below\n", "(figure credit - [Aaron van den Oord et al. ](https://arxiv.org/pdf/1606.05328.pdf)):\n", "\n", "
\n", "\n", "Although a pixel should be able to take into account all other pixels above and left of it,\n", "a stack of masked convolutions does not allow us to look to the upper pixels on the right.\n", "This is because the features of the pixels above, which we use for convolution,\n", "do not contain any information of the pixels on the right of the same row.\n", "If they would, we would be \"cheating\" and actually looking into the future.\n", "To overcome this issue, van den Oord et.\n", "al [2] proposed to split the convolutions into a vertical and a horizontal stack.\n", "The vertical stack looks at all pixels above the current one, while the horizontal takes into account all on the left.\n", "While keeping both of them separate, we can actually look at the pixels on the right with the vertical stack without breaking any of our assumptions.\n", "The two convolutions are also shown in the figure above.\n", "\n", "Let us implement them here as follows:"]}, {"cell_type": "code", "execution_count": 7, "id": "97b535aa", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:38.450380Z", "iopub.status.busy": "2023-03-14T16:13:38.449893Z", "iopub.status.idle": "2023-03-14T16:13:38.459719Z", "shell.execute_reply": "2023-03-14T16:13:38.459235Z"}, "papermill": {"duration": 0.022494, "end_time": "2023-03-14T16:13:38.462513", "exception": false, "start_time": "2023-03-14T16:13:38.440019", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class VerticalStackConvolution(MaskedConvolution):\n", " def __init__(self, c_in, c_out, kernel_size=3, mask_center=False, **kwargs):\n", " # Mask out all pixels below. For efficiency, we could also reduce the kernel\n", " # size in height, but for simplicity, we stick with masking here.\n", " mask = torch.ones(kernel_size, kernel_size)\n", " mask[kernel_size // 2 + 1 :, :] = 0\n", "\n", " # For the very first convolution, we will also mask the center row\n", " if mask_center:\n", " mask[kernel_size // 2, :] = 0\n", "\n", " super().__init__(c_in, c_out, mask, **kwargs)\n", "\n", "\n", "class HorizontalStackConvolution(MaskedConvolution):\n", " def __init__(self, c_in, c_out, kernel_size=3, mask_center=False, **kwargs):\n", " # Mask out all pixels on the left. Note that our kernel has a size of 1\n", " # in height because we only look at the pixel in the same row.\n", " mask = torch.ones(1, kernel_size)\n", " mask[0, kernel_size // 2 + 1 :] = 0\n", "\n", " # For the very first convolution, we will also mask the center pixel\n", " if mask_center:\n", " mask[0, kernel_size // 2] = 0\n", "\n", " super().__init__(c_in, c_out, mask, **kwargs)"]}, {"cell_type": "markdown", "id": "f9c58750", "metadata": {"papermill": {"duration": 0.008882, "end_time": "2023-03-14T16:13:38.484630", "exception": false, "start_time": "2023-03-14T16:13:38.475748", "status": "completed"}, "tags": []}, "source": ["Note that we have an input argument called `mask_center`. Remember that\n", "the input to the model is the actual input image. Hence, the very first\n", "convolution we apply cannot use the center pixel as input, but must be\n", "masked. All consecutive convolutions, however, should use the center\n", "pixel as we otherwise lose the features of the previous layer. Hence,\n", "the input argument `mask_center` is True for the very first\n", "convolutions, and False for all others."]}, {"cell_type": "markdown", "id": "4da9569f", "metadata": {"papermill": {"duration": 0.008849, "end_time": "2023-03-14T16:13:38.502394", "exception": false, "start_time": "2023-03-14T16:13:38.493545", "status": "completed"}, "tags": []}, "source": ["### Visualizing the receptive field\n", "\n", "To validate our implementation of masked convolutions, we can visualize the receptive field we obtain with such convolutions.\n", "We should see that with increasing number of convolutional layers, the receptive field grows in both vertical and horizontal direction, without the issue of a blind spot.\n", "The receptive field can be empirically measured by backpropagating an arbitrary loss for the output features of a speicifc pixel with respect to the input.\n", "We implement this idea below, and visualize the receptive field below."]}, {"cell_type": "code", "execution_count": 8, "id": "8f07bd67", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:38.521677Z", "iopub.status.busy": "2023-03-14T16:13:38.521303Z", "iopub.status.idle": "2023-03-14T16:13:38.725791Z", "shell.execute_reply": "2023-03-14T16:13:38.725197Z"}, "papermill": {"duration": 0.217186, "end_time": "2023-03-14T16:13:38.728503", "exception": false, "start_time": "2023-03-14T16:13:38.511317", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1Qy27CMBC871fMEQ51vHYdJ8dSWtTeaCNxQD1UwYREhDaAQP37LkHpA1Uqra2Rduy1Z3aiYdiVeXgYDXD9SNEnyzfEqAQFNCrBHoyRoCAtrCbrWTkj5bIrOWWVOGO8lTP9nS6I5tTAK9OCY6Os8cdtT+k6YIIVoitR24hkJdiL0ujEYNM+9AcTuis7hbxGdMcYvmBMYzTSUHz96cCpkfE0LuQt2CnWLjE2iW0M9k4G+vCT1zTIKLqVNoNs3gaQzWiK3qQPlyiXWu/QC2Wx2IYZ1n0YrRLbLbnJw+u23AXMy7Cc9fGE7J5uMjpaM5pVmnZy/82HDlP9ns8PYm1U5uyoDKdKa81sJarL87MalKvn9dvf0qF3+fSWJgplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjI4NAplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagoxOSAwIG9iago8PCAvTGVuZ3RoIDI2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UrmRAzEMy7cKlsBfUj2+uXFg958ewD07MTFLEQBB925RORs/bSXLj/zYZWdJ5Jb3oG3yuqLqBqmbIHPJcckVYpbyuBIkFi1lJtZnqoPycQ1qFb7wEzMT0yFJxBJyUo8irI+vg9f1HNxfN+n8GhkfdGxQekuSq6BUw75ytBI7lupdg+yDppvS6jPTruyApfGGrNSkTn8d9b8jLMKk3khFByEWv9PLHbIspBzU27l+A+Fd7YJYT6087BBp3lZ6SxXM5swETBltO6yAtVljwlQJ8BbNIdRaiMwXOq2I+eTc0cE0VXkaIsNShYPtPaM1XOgaEkvD+UnGBOa/8PqsyG1//wBwaGe6CmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0xlbmd0aCA5MiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9jcENwDAIA/9MwQgQAsT7VFUf6f7fJhHqBx8G2RhgYbM14MHZwJfS2je9pEWT2ghWtUXdUJ67FKVYXUelTMJPmTt/UnQc7XAO29/W5ThN4+hf99D9AQ9KHgsKZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvTGVuZ3RoIDMwNyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9kktuAzEMQ/c+hS4QwPrZnvOkKLqY3n/bJyXpihzZFkVqlrpMWVMekDSThH/p8HCxnfI7bM9mZuBaopeJ5ZTn0BVi7qJ82cxGXVknxeqEZjq36FE5Fwc2Taqfqyyl3S54Dtcmnlv2ET+80KAe1DUuCTd0V6NlKTRjqvt/0nv8jDLgakxdbFKrex88XkRV6OgHR4kiY5cX5+NBCelKwmhaiJV3RQNB7vK0ynsJ7tveasiyB6mYzjspZrDrdFIubheHIR7I8qjw5aPYa0LP+LArJfRI2IYzcifuaMbm1MjikP7ejQRLj65oIfPgr27WLmC8UzpFYmROcqxpi1VO91AU07nDvQwQ9WxFQylzkdXqX8POC2uWbBZ4SvoFHqPdJksOVtnbqE7vrTzZ0PcfWtd0HwplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9MZW5ndGggMjMyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVRSW7EMAy7+xX8wADW7rwnxaCH9v/XUsoUCEAltrglYmMjAi8x+DmI3PiSNaMmfmdyV/wsT4VHwq3gSRSBl+FedoLLG8ZlPw4zH7yXVs6kxpMMyEU2PTwRMtglEDowuwZ12Gbaib4h4bMjUs1GltPXEvTSKgTKU7bf6YISbav6c/usC2372hNOdnvqSeUTiOeWrMBl4xWTxVgGPVG5SzF9kOpsoSehvCifg2w+aohElyhn4InBwSjQDuy57WfiVSFoXd2nbWOoRkrH078NTU2SCPlECWe2NO4W/n/Pvb7X+w9OIVQRCmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0xlbmd0aCAyMzEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicNU85kgQhDMt5hT4wVRjbQL+np7Y22Pl/upKZTpDwIcnTEx2ZeJkjI7Bmx9taZCBm4FNMxb/2tA8TqvfgHiKUiwthhpFw1qzjbp6OF/92lc9YB+82+IpZXhDYwkzWVxZnLtsFY2mcxDnJboxdE7GNda2nU1hHMKEMhHS2w5Qgc1Sk9MmOMuboOJEnnovv9tssdjl+DusLNo0hFef4KnqCNoOi7HnvAhpyQf9d3fgeRbvoJSAbCRbWUWLunOWEX712dB61KBJzQppBLhMhzekqphCaUKyzo6BSUXCpPqforJ9/5V9cLQplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9MZW5ndGggMjQ5IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvTGVuZ3RoIDEzNiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxNj0EOAzEIA+95hZ9AIEB4z1ZVD9v/X0vYdtMLHsmAbFEGgSWHeIcb4dHbD99FNhVn45xfUiliIZhPcJ8wUxyNKXfyY4+AcZRqLKdoeF5Lzk3DFy13Ey2lrZeTGW+47pf3R5VtkQ1Fzy0LQtdskvkygQd8GJhHdeNppcfd9myv9vwAzmw0SQplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9MZW5ndGggMzQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEVSS25EMQjbv1NwgUjhl5DztKq6mN5/W5tM1c3gCWBseMtTpmTKsLklIyTXlE99IkOspvw0ciQipvhJCQV2lY/Ha0usjeyRqBSf2vHjsfRGptkVWvXu0aXNolHNysg5yBChnhW6snvUDtnwelxIuu+UzSEcy/9QgSxl3XIKJUFb0HfsEd8PHa6CK4JhsGsug+1lMtT/+ocWXO9992LHLoAWrOe+wQ4AqKcTtAXIGdruNiloAFW6i0nCo/J6bnaibKNV6fkcADMOMHLAiCVbHb7R3gCWfV3oRY2K/StAUVlA/MjVdsHeMclIcBbmBo69cDzFmXBLOMYCQIq94hh68CXY5i9Xroia8Al1umQvvMKe2ubnQpMId60ADl5kw62ro6iW7ek8gvZnRXJGjNSLODohklrSOYLi0qAeWuNcN7HibSOxuVff7h/hnC9c9usXS+yExAplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9MZW5ndGggMTY0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQx3EFMQxD76oCJTCACvWsx/MP6/6vhvTTQXoYQgxiT8KwXFdxYXTDj7ctMw1/RxnuxvoyY7zVWCAn6AMMkYmr0aT6dsUZqvTk1WKuo6JcLzoiEsyS46tAI3w6sseTtrYz/XReH+wh7xP/KirnbmEBLqruQPlSH/HUj9lR6pqhjyorax5q2leEXRFK2z4upzJO3b0DWuG9las92u8/HnY68gplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9MZW5ndGggNzIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZcQL6piblCLhdIDMTKAbMMgLQlnIKIZ4CYIG0QxSAWRLGZiRlEHZwBkcvgSgMAJdsWyQplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9MZW5ndGggNDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZclhBWLhdMLAfMAtGWcAoinsGVBgC5Zw0nCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0xlbmd0aCAxNjMgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA7EgMhDEN7TqEj+CMDPs9mMik2929j2GxSwNNYIIO7E4LU2oKJ6IKHtiXdBe+tBGdj/Ok2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlDcPVf9b9i3TmbiYHJyh0IzepT3Pk2O6K6usn+pMfcrNd+K+xVYWlZS8sJt527ZkAJ3FM52qs9Px8KOvYKZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iago8PCAvTGVuZ3RoIDIzOSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxNUMltBDEM+7sKNTDA6By7HgeLPLL9f0PKCZKXaEviofKUW5bKZfcjOW/JuuVDh06VafJu0M2vsf6jDAJ2/1BUEK0lsUrMXNJusTRJL9nDOI2Xa7WO56l7hFmjePDj2NMpgek9MsFms705MKs9zg6QTrjGr+rTO5UkA4m6kPNCpQrrHtQloo8r25hSnU4t5RiXn+h7fI4APcXejdzRx8sXjEa1LajRapU4DzATU9GVcauRgZQTBkNnR1c0C6XIynpCNcKNOaGZvcNwYAPLs4Skpa1SvA9lAegCXdo64zRKgo4Awt8ojPX6Bqr8XjcKZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iago8PCAvTGVuZ3RoIDE2MCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjw8IC9MZW5ndGggMTggL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicMza0UDCAwxRDrjQAHeYDUgplbmRzdHJlYW0KZW5kb2JqCjM0IDAgb2JqCjw8IC9MZW5ndGggMTMzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWPSw4EIQhE95yijsDHH+dxMumFc//tgJ1uE2M9hVSBuYKhPS5rA50VHyEZtvG3qZaORVk+VHpSVg/J4Iesxssh3KAs8IJJKoYhUIuYGpEtZW63gNs2DbKylVOljrCLozCP9rRsFR5folsidZI/g8QqL9zjuh3Ipda73qKLvn+kATEJCmVuZHN0cmVhbQplbmRvYmoKMzUgMCBvYmoKPDwgL0xlbmd0aCA3NSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJwztTRSMFAwNgASpmZGCqYm5gophlxAPoiVy2VoZApm5XAZWZopWFgAGSZm5lAhmIYcLmNTc6ABQEXGpmAaqj+HK4MrDQCVkBLvCmVuZHN0cmVhbQplbmRvYmoKMzYgMCBvYmoKPDwgL0xlbmd0aCAxNDEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY/BDsMwCEPv+Qr/QKTYKaF8T6dqh+7/ryNLuwt6AmOMhdDQG6qaw4Zgm+PF0iVUa/gUxUAlN8iZYA6lpNIdR5F6YjgYXB60G47isej6EbuSZn3QxkK6JWiAe6xTadymcRPEHTUF6inqnKO8ELmfqWfYNJLdNLOSc7gNv3vPU9f/p6u8y/kFvXcu/gplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9CYXNlRm9udCAvQk1RUURWK0RlamFWdVNhbnMgL0ZpcnN0Q2hhciAwIC9MYXN0Q2hhciAyNTUKL0ZvbnREZXNjcmlwdG9yIDE2IDAgUiAvU3VidHlwZSAvVHlwZTMgL05hbWUgL0JNUVFEVitEZWphVnVTYW5zCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0KL0NoYXJQcm9jcyAxOCAwIFIKL0VuY29kaW5nIDw8IC9UeXBlIC9FbmNvZGluZwovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNjYgL0IgODcgL1cgOTcgL2EgOTkgL2MgL2QgL2UgL2YgL2cgL2ggL2kgMTA4IC9sIDExMCAvbiAxMTIgL3AKMTE0IC9yIDExNiAvdCAxMTggL3YgMTIxIC95IF0KPj4KL1dpZHRocyAxNSAwIFIgPj4KZW5kb2JqCjE2IDAgb2JqCjw8IC9UeXBlIC9Gb250RGVzY3JpcHRvciAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zIC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Bc2NlbnQgOTI5IC9EZXNjZW50IC0yMzYgL0NhcEhlaWdodCAwCi9YSGVpZ2h0IDAgL0l0YWxpY0FuZ2xlIDAgL1N0ZW1WIDAgL01heFdpZHRoIDEzNDIgPj4KZW5kb2JqCjE1IDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE4IDAgb2JqCjw8IC9CIDE5IDAgUiAvVyAyMCAwIFIgL2EgMjEgMCBSIC9jIDIyIDAgUiAvZCAyMyAwIFIgL2UgMjQgMCBSIC9mIDI1IDAgUgovZyAyNiAwIFIgL2ggMjcgMCBSIC9pIDI4IDAgUiAvbCAyOSAwIFIgL24gMzAgMCBSIC9wIDMxIDAgUiAvciAzMiAwIFIKL3NwYWNlIDMzIDAgUiAvdCAzNCAwIFIgL3YgMzUgMCBSIC95IDM2IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTcgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgL0kyIDE0IDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjYgL0hlaWdodCAyMjYKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMSAo/ec3ACJNKV0gL0JpdHNQZXJDb21wb25lbnQgOAovRmlsdGVyIC9GbGF0ZURlY29kZQovRGVjb2RlUGFybXMgPDwgL1ByZWRpY3RvciAxMCAvQ29sb3JzIDEgL0NvbHVtbnMgMjI2ID4+IC9MZW5ndGggMzcgMCBSID4+CnN0cmVhbQp4nO3PwQ3DABDDsHT/pbtAnz0HEKgFbD6ffM/bB+5DLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCPF/Oz9aTa92EC+nVzuIl9OrHcTL6dUO4uX0agfxcnq1g3g5vdpBvJxe7SBeTq92EC+nVzt94oshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFjoCxKCxfUKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iagozMDYKZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDEgKP3nNwAiTSldIC9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM4IDAgUiA+PgpzdHJlYW0KeJztz8ENwwAQw7B0/6W7QJ89BxCoBWw+n3zP2wfuQyyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAjxfzs/Wk2vdhAvp1c7iJfTqx3Ey+nVDuLl9GoH8XJ6tYN4Ob3aQbycXu0gXk6vdhAvp1c7feKLIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRY6AsSgsX1CmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKMzA2CmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagozOSAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My43LjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My43LjEpIC9DcmVhdGlvbkRhdGUgKEQ6MjAyMzAzMTQxNjEzMzhaKQo+PgplbmRvYmoKeHJlZgowIDQwCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA4MzIxIDAwMDAwIG4gCjAwMDAwMDY5OTAgMDAwMDAgbiAKMDAwMDAwNzAyMiAwMDAwMCBuIAowMDAwMDA3MDgyIDAwMDAwIG4gCjAwMDAwMDcxMDMgMDAwMDAgbiAKMDAwMDAwNzEyNCAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDQgMDAwMDAgbiAKMDAwMDAwMDcyMyAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA3MDMgMDAwMDAgbiAKMDAwMDAwNzE2NyAwMDAwMCBuIAowMDAwMDA3NzQ0IDAwMDAwIG4gCjAwMDAwMDU3MzEgMDAwMDAgbiAKMDAwMDAwNTUyNCAwMDAwMCBuIAowMDAwMDA1MTE2IDAwMDAwIG4gCjAwMDAwMDY3ODQgMDAwMDAgbiAKMDAwMDAwMDc0MyAwMDAwMCBuIAowMDAwMDAxMDgwIDAwMDAwIG4gCjAwMDAwMDEyNDQgMDAwMDAgbiAKMDAwMDAwMTYyNCAwMDAwMCBuIAowMDAwMDAxOTI5IDAwMDAwIG4gCjAwMDAwMDIyMzMgMDAwMDAgbiAKMDAwMDAwMjU1NSAwMDAwMCBuIAowMDAwMDAyNzY0IDAwMDAwIG4gCjAwMDAwMDMxNzggMDAwMDAgbiAKMDAwMDAwMzQxNSAwMDAwMCBuIAowMDAwMDAzNTU5IDAwMDAwIG4gCjAwMDAwMDM2NzggMDAwMDAgbiAKMDAwMDAwMzkxNCAwMDAwMCBuIAowMDAwMDA0MjI2IDAwMDAwIG4gCjAwMDAwMDQ0NTkgMDAwMDAgbiAKMDAwMDAwNDU0OSAwMDAwMCBuIAowMDAwMDA0NzU1IDAwMDAwIG4gCjAwMDAwMDQ5MDIgMDAwMDAgbiAKMDAwMDAwNzcyNCAwMDAwMCBuIAowMDAwMDA4MzAxIDAwMDAwIG4gCjAwMDAwMDgzODEgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSA0MCAvUm9vdCAxIDAgUiAvSW5mbyAzOSAwIFIgPj4Kc3RhcnR4cmVmCjg1MzIKJSVFT0YK", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["inp_img = torch.zeros(1, 1, 11, 11)\n", "inp_img.requires_grad_()\n", "\n", "\n", "def show_center_recep_field(img, out):\n", " \"\"\"Calculates the gradients of the input with respect to the output center pixel, and visualizes the overall\n", " receptive field.\n", "\n", " Args:\n", " img: Input image for which we want to calculate the receptive field on.\n", " out: Output features/loss which is used for backpropagation, and should be\n", " the output of the network/computation graph.\n", " \"\"\"\n", " # Determine gradients\n", " loss = out[0, :, img.shape[2] // 2, img.shape[3] // 2].sum() # L1 loss for simplicity\n", " # Retain graph as we want to stack multiple layers and show the receptive field of all of them\n", " loss.backward(retain_graph=True)\n", " img_grads = img.grad.abs()\n", " img.grad.fill_(0) # Reset grads\n", "\n", " # Plot receptive field\n", " img = img_grads.squeeze().cpu().numpy()\n", " fig, ax = plt.subplots(1, 2)\n", " _ = ax[0].imshow(img)\n", " ax[1].imshow(img > 0)\n", " # Mark the center pixel in red if it doesn't have any gradients (should be\n", " # the case for standard autoregressive models)\n", " show_center = img[img.shape[0] // 2, img.shape[1] // 2] == 0\n", " if show_center:\n", " center_pixel = np.zeros(img.shape + (4,))\n", " center_pixel[center_pixel.shape[0] // 2, center_pixel.shape[1] // 2, :] = np.array([1.0, 0.0, 0.0, 1.0])\n", " for i in range(2):\n", " ax[i].axis(\"off\")\n", " if show_center:\n", " ax[i].imshow(center_pixel)\n", " ax[0].set_title(\"Weighted receptive field\")\n", " ax[1].set_title(\"Binary receptive field\")\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "show_center_recep_field(inp_img, inp_img)"]}, {"cell_type": "markdown", "id": "e338e8e3", "metadata": {"papermill": {"duration": 0.009539, "end_time": "2023-03-14T16:13:38.752675", "exception": false, "start_time": "2023-03-14T16:13:38.743136", "status": "completed"}, "tags": []}, "source": ["Let's first visualize the receptive field of a horizontal convolution\n", "without the center pixel. We use a small, arbitrary input image\n", "($11\\times 11$ pixels), and calculate the loss for the center pixel. For\n", "simplicity, we initialize all weights with 1 and the bias with 0, and\n", "use a single channel. This is sufficient for our visualization purposes."]}, {"cell_type": "code", "execution_count": 9, "id": "b1f9a2d2", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:38.773402Z", "iopub.status.busy": "2023-03-14T16:13:38.773004Z", "iopub.status.idle": "2023-03-14T16:13:39.002904Z", "shell.execute_reply": "2023-03-14T16:13:39.002298Z"}, "papermill": {"duration": 0.241939, "end_time": "2023-03-14T16:13:39.004164", "exception": false, "start_time": "2023-03-14T16:13:38.762225", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP3nNwAiTf8AACldIC9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM3IDAgUiA+PgpzdHJlYW0KeJztz8ENAzEMBDEn/Rd9DfgVSLhgwClAWp5PvvP2gP0QCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQvzp5K3vpfnX9z0LJxER5/csnEREnN+zcBIRcX7PwklExPk9CycREef3LJxERJzfs3CyT/y3EAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIs9AC1QMdxCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKMzA2CmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvSW1hZ2UgL1dpZHRoIDIyNiAvSGVpZ2h0IDIyNgovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyICj95zcAIk3/AAApXSAvQml0c1BlckNvbXBvbmVudCA4Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyAyMjYgPj4gL0xlbmd0aCAzOCAwIFIgPj4Kc3RyZWFtCnic7c/BDQMxDAQxJ/0XfQ34FUi4YMApQFqeT77z9oD9EAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkL86eSt76X51/c9CycREef3LJxERJzfs3ASEXF+z8JJRMT5PQsnERHn9yycRESc37Nwsk/8txALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLPQAtUDHcQplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjMwNgplbmRvYmoKMiAwIG9iago8PCAvVHlwZSAvUGFnZXMgL0tpZHMgWyAxMSAwIFIgXSAvQ291bnQgMSA+PgplbmRvYmoKMzkgMCBvYmoKPDwgL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNy4xLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNy4xKSAvQ3JlYXRpb25EYXRlIChEOjIwMjMwMzE0MTYxMzM4WikKPj4KZW5kb2JqCnhyZWYKMCA0MAowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwODMyNSAwMDAwMCBuIAowMDAwMDA2OTg4IDAwMDAwIG4gCjAwMDAwMDcwMjAgMDAwMDAgbiAKMDAwMDAwNzA4MCAwMDAwMCBuIAowMDAwMDA3MTAxIDAwMDAwIG4gCjAwMDAwMDcxMjIgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQ0IDAwMDAwIG4gCjAwMDAwMDA3MjEgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNzAxIDAwMDAwIG4gCjAwMDAwMDcxNjUgMDAwMDAgbiAKMDAwMDAwNzc0NSAwMDAwMCBuIAowMDAwMDA1NzI5IDAwMDAwIG4gCjAwMDAwMDU1MjIgMDAwMDAgbiAKMDAwMDAwNTExNCAwMDAwMCBuIAowMDAwMDA2NzgyIDAwMDAwIG4gCjAwMDAwMDA3NDEgMDAwMDAgbiAKMDAwMDAwMTA3OCAwMDAwMCBuIAowMDAwMDAxMjQyIDAwMDAwIG4gCjAwMDAwMDE2MjIgMDAwMDAgbiAKMDAwMDAwMTkyNyAwMDAwMCBuIAowMDAwMDAyMjMxIDAwMDAwIG4gCjAwMDAwMDI1NTMgMDAwMDAgbiAKMDAwMDAwMjc2MiAwMDAwMCBuIAowMDAwMDAzMTc2IDAwMDAwIG4gCjAwMDAwMDM0MTMgMDAwMDAgbiAKMDAwMDAwMzU1NyAwMDAwMCBuIAowMDAwMDAzNjc2IDAwMDAwIG4gCjAwMDAwMDM5MTIgMDAwMDAgbiAKMDAwMDAwNDIyNCAwMDAwMCBuIAowMDAwMDA0NDU3IDAwMDAwIG4gCjAwMDAwMDQ1NDcgMDAwMDAgbiAKMDAwMDAwNDc1MyAwMDAwMCBuIAowMDAwMDA0OTAwIDAwMDAwIG4gCjAwMDAwMDc3MjUgMDAwMDAgbiAKMDAwMDAwODMwNSAwMDAwMCBuIAowMDAwMDA4Mzg1IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgNDAgL1Jvb3QgMSAwIFIgL0luZm8gMzkgMCBSID4+CnN0YXJ0eHJlZgo4NTM2CiUlRU9GCg==", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["horiz_conv = HorizontalStackConvolution(c_in=1, c_out=1, kernel_size=3, mask_center=True)\n", "horiz_conv.conv.weight.data.fill_(1)\n", "horiz_conv.conv.bias.data.fill_(0)\n", "horiz_img = horiz_conv(inp_img)\n", "show_center_recep_field(inp_img, horiz_img)"]}, {"cell_type": "markdown", "id": "5f9c3bba", "metadata": {"papermill": {"duration": 0.00991, "end_time": "2023-03-14T16:13:39.024315", "exception": false, "start_time": "2023-03-14T16:13:39.014405", "status": "completed"}, "tags": []}, "source": ["The receptive field is shown in yellow, the center pixel in red, and all other pixels outside of the receptive field are dark blue.\n", "As expected, the receptive field of a single horizontal convolution with the center pixel masked and a $3\\times3$ kernel is only the pixel on the left.\n", "If we use a larger kernel size, more pixels would be taken into account on the left.\n", "\n", "Next, let's take a look at the vertical convolution:"]}, {"cell_type": "code", "execution_count": 10, "id": "c2d5757e", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:39.045390Z", "iopub.status.busy": "2023-03-14T16:13:39.045130Z", "iopub.status.idle": "2023-03-14T16:13:39.265495Z", "shell.execute_reply": "2023-03-14T16:13:39.264909Z"}, "papermill": {"duration": 0.232545, "end_time": "2023-03-14T16:13:39.266774", "exception": false, "start_time": "2023-03-14T16:13:39.034229", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP3nNwAiTf8AACldIC9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM3IDAgUiA+PgpzdHJlYW0KeJztz8FtwwAMBEEl/RftBvQLzwEWswWQN89Pvue/B+xDLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWGhCfPzXYMziJiHi/Z3ASEfF+z+AkIuL9nsFJRMT7PYOTiIj3ewYnERHv9wxOIgaIr/2+9KXXiGchLkM8C3EZ4lmIyxDPQlyGeBbiMsSzEJchFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsdAHEijD/wplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjMxMwplbmRvYmoKMTQgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjYgL0hlaWdodCAyMjYKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMiAo/ec3ACJN/wAAKV0gL0JpdHNQZXJDb21wb25lbnQgOAovRmlsdGVyIC9GbGF0ZURlY29kZQovRGVjb2RlUGFybXMgPDwgL1ByZWRpY3RvciAxMCAvQ29sb3JzIDEgL0NvbHVtbnMgMjI2ID4+IC9MZW5ndGggMzggMCBSID4+CnN0cmVhbQp4nO3PwW3DAAwEQSX9F+0G9AvPARazBZA3z0++578H7EMshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRYaEJ8/NdgzOImIeL9ncBIR8X7P4CQi4v2ewUlExPs9g5OIiPd7BicREe/3DE4iBoiv/b70pdeIZyEuQzwLcRniWYjLEM9CXIZ4FuIyxLMQlyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAix0AcSKMP/CmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKMzEzCmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagozOSAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My43LjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My43LjEpIC9DcmVhdGlvbkRhdGUgKEQ6MjAyMzAzMTQxNjEzMzlaKQo+PgplbmRvYmoKeHJlZgowIDQwCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA4MzM5IDAwMDAwIG4gCjAwMDAwMDY5ODggMDAwMDAgbiAKMDAwMDAwNzAyMCAwMDAwMCBuIAowMDAwMDA3MDgwIDAwMDAwIG4gCjAwMDAwMDcxMDEgMDAwMDAgbiAKMDAwMDAwNzEyMiAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDQgMDAwMDAgbiAKMDAwMDAwMDcyMSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA3MDEgMDAwMDAgbiAKMDAwMDAwNzE2NSAwMDAwMCBuIAowMDAwMDA3NzUyIDAwMDAwIG4gCjAwMDAwMDU3MjkgMDAwMDAgbiAKMDAwMDAwNTUyMiAwMDAwMCBuIAowMDAwMDA1MTE0IDAwMDAwIG4gCjAwMDAwMDY3ODIgMDAwMDAgbiAKMDAwMDAwMDc0MSAwMDAwMCBuIAowMDAwMDAxMDc4IDAwMDAwIG4gCjAwMDAwMDEyNDIgMDAwMDAgbiAKMDAwMDAwMTYyMiAwMDAwMCBuIAowMDAwMDAxOTI3IDAwMDAwIG4gCjAwMDAwMDIyMzEgMDAwMDAgbiAKMDAwMDAwMjU1MyAwMDAwMCBuIAowMDAwMDAyNzYyIDAwMDAwIG4gCjAwMDAwMDMxNzYgMDAwMDAgbiAKMDAwMDAwMzQxMyAwMDAwMCBuIAowMDAwMDAzNTU3IDAwMDAwIG4gCjAwMDAwMDM2NzYgMDAwMDAgbiAKMDAwMDAwMzkxMiAwMDAwMCBuIAowMDAwMDA0MjI0IDAwMDAwIG4gCjAwMDAwMDQ0NTcgMDAwMDAgbiAKMDAwMDAwNDU0NyAwMDAwMCBuIAowMDAwMDA0NzUzIDAwMDAwIG4gCjAwMDAwMDQ5MDAgMDAwMDAgbiAKMDAwMDAwNzczMiAwMDAwMCBuIAowMDAwMDA4MzE5IDAwMDAwIG4gCjAwMDAwMDgzOTkgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSA0MCAvUm9vdCAxIDAgUiAvSW5mbyAzOSAwIFIgPj4Kc3RhcnR4cmVmCjg1NTAKJSVFT0YK", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["vert_conv = VerticalStackConvolution(c_in=1, c_out=1, kernel_size=3, mask_center=True)\n", "vert_conv.conv.weight.data.fill_(1)\n", "vert_conv.conv.bias.data.fill_(0)\n", "vert_img = vert_conv(inp_img)\n", "show_center_recep_field(inp_img, vert_img)"]}, {"cell_type": "markdown", "id": "5ad48dde", "metadata": {"papermill": {"duration": 0.010586, "end_time": "2023-03-14T16:13:39.288150", "exception": false, "start_time": "2023-03-14T16:13:39.277564", "status": "completed"}, "tags": []}, "source": ["The vertical convolution takes all pixels above into account. Combining\n", "these two, we get the L-shaped receptive field of the original masked\n", "convolution:"]}, {"cell_type": "code", "execution_count": 11, "id": "18807b1c", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:39.310996Z", "iopub.status.busy": "2023-03-14T16:13:39.310539Z", "iopub.status.idle": "2023-03-14T16:13:39.530846Z", "shell.execute_reply": "2023-03-14T16:13:39.530262Z"}, "papermill": {"duration": 0.233421, "end_time": "2023-03-14T16:13:39.532086", "exception": false, "start_time": "2023-03-14T16:13:39.298665", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP3nNwAiTf8AACldIC9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM3IDAgUiA+PgpzdHJlYW0KeJztz8FtwwAMBEEl/RftBvQLDw4WswWQN89PvufbA/YhFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALDYjPnxrsGZxERLzfMziJiHi/Z3ASEfF+z+AkIuL9nsFJRMT7PYOTiIj3ewYnEavE35fuX7/vGZxERLzfMziJiHi/Z3ASEfF+z+AkIuL9nsFJRMT7PYOTiIj3ewYn+8T/FmIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhT6DL8JbCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKMzI3CmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvSW1hZ2UgL1dpZHRoIDIyNiAvSGVpZ2h0IDIyNgovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyICj95zcAIk3/AAApXSAvQml0c1BlckNvbXBvbmVudCA4Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyAyMjYgPj4gL0xlbmd0aCAzOCAwIFIgPj4Kc3RyZWFtCnic7c/BbcMADARBJf0X7Qb0Cw8OFrMFkDfPT77n2wP2IRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCw2Iz58a7BmcRES83zM4iYh4v2dwEhHxfs/gJCLi/Z7BSUTE+z2Dk4iI93sGJxGrxN+X7l+/7xmcRES83zM4iYh4v2dwEhHxfs/gJCLi/Z7BSUTE+z2Dk4iI93sGJ/vE/xZiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIU+gy/CWwplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjMyNwplbmRvYmoKMiAwIG9iago8PCAvVHlwZSAvUGFnZXMgL0tpZHMgWyAxMSAwIFIgXSAvQ291bnQgMSA+PgplbmRvYmoKMzkgMCBvYmoKPDwgL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNy4xLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNy4xKSAvQ3JlYXRpb25EYXRlIChEOjIwMjMwMzE0MTYxMzM5WikKPj4KZW5kb2JqCnhyZWYKMCA0MAowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwODM2NyAwMDAwMCBuIAowMDAwMDA2OTg4IDAwMDAwIG4gCjAwMDAwMDcwMjAgMDAwMDAgbiAKMDAwMDAwNzA4MCAwMDAwMCBuIAowMDAwMDA3MTAxIDAwMDAwIG4gCjAwMDAwMDcxMjIgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQ0IDAwMDAwIG4gCjAwMDAwMDA3MjEgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNzAxIDAwMDAwIG4gCjAwMDAwMDcxNjUgMDAwMDAgbiAKMDAwMDAwNzc2NiAwMDAwMCBuIAowMDAwMDA1NzI5IDAwMDAwIG4gCjAwMDAwMDU1MjIgMDAwMDAgbiAKMDAwMDAwNTExNCAwMDAwMCBuIAowMDAwMDA2NzgyIDAwMDAwIG4gCjAwMDAwMDA3NDEgMDAwMDAgbiAKMDAwMDAwMTA3OCAwMDAwMCBuIAowMDAwMDAxMjQyIDAwMDAwIG4gCjAwMDAwMDE2MjIgMDAwMDAgbiAKMDAwMDAwMTkyNyAwMDAwMCBuIAowMDAwMDAyMjMxIDAwMDAwIG4gCjAwMDAwMDI1NTMgMDAwMDAgbiAKMDAwMDAwMjc2MiAwMDAwMCBuIAowMDAwMDAzMTc2IDAwMDAwIG4gCjAwMDAwMDM0MTMgMDAwMDAgbiAKMDAwMDAwMzU1NyAwMDAwMCBuIAowMDAwMDAzNjc2IDAwMDAwIG4gCjAwMDAwMDM5MTIgMDAwMDAgbiAKMDAwMDAwNDIyNCAwMDAwMCBuIAowMDAwMDA0NDU3IDAwMDAwIG4gCjAwMDAwMDQ1NDcgMDAwMDAgbiAKMDAwMDAwNDc1MyAwMDAwMCBuIAowMDAwMDA0OTAwIDAwMDAwIG4gCjAwMDAwMDc3NDYgMDAwMDAgbiAKMDAwMDAwODM0NyAwMDAwMCBuIAowMDAwMDA4NDI3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgNDAgL1Jvb3QgMSAwIFIgL0luZm8gMzkgMCBSID4+CnN0YXJ0eHJlZgo4NTc4CiUlRU9GCg==", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["horiz_img = vert_img + horiz_img\n", "show_center_recep_field(inp_img, horiz_img)"]}, {"cell_type": "markdown", "id": "4b47f51a", "metadata": {"papermill": {"duration": 0.01113, "end_time": "2023-03-14T16:13:39.554659", "exception": false, "start_time": "2023-03-14T16:13:39.543529", "status": "completed"}, "tags": []}, "source": ["If we stack multiple horizontal and vertical convolutions, we need to take two aspects into account:\n", "\n", "1.\n", "The center should not be masked anymore for the following convolutions as the features at the pixel's position are already independent of its actual value.\n", "If it is hard to imagine why we can do this, just change the value below to `mask_center=True` and see what happens.\n", "2.\n", "The vertical convolution is not allowed to work on features from the horizontal convolution.\n", "In the feature map of the horizontal convolutions, a pixel contains information about all of the \"true\" pixels on the left.\n", "If we apply a vertical convolution which also uses features from the right, we effectively expand our receptive field to the true input which we want to prevent.\n", "Thus, the feature maps can only be merged for the horizontal convolution.\n", "\n", "Using this, we can stack the convolutions in the following way. We have\n", "two feature streams: one for the vertical stack, and one for the\n", "horizontal stack. The horizontal convolutions can operate on the joint\n", "features of the previous horizontals and vertical convolutions, while\n", "the vertical stack only takes its own previous features as input. For a\n", "quick implementation, we can therefore sum the horizontal and vertical\n", "output features at each layer, and use those as final output features to\n", "calculate the loss on. An implementation of 4 consecutive layers is\n", "shown below. Note that we reuse the features from the other convolutions\n", "with `mask_center=True` from above."]}, {"cell_type": "code", "execution_count": 12, "id": "4597b204", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:39.578521Z", "iopub.status.busy": "2023-03-14T16:13:39.578121Z", "iopub.status.idle": "2023-03-14T16:13:40.679037Z", "shell.execute_reply": "2023-03-14T16:13:40.678393Z"}, "papermill": {"duration": 1.115033, "end_time": "2023-03-14T16:13:40.680823", "exception": false, "start_time": "2023-03-14T16:13:39.565790", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Layer 2\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDYgKP3nN8i3ZZSOd2ZpcDVFbAAiTf8AACldCi9CaXRzUGVyQ29tcG9uZW50IDggL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM3IDAgUiA+PgpzdHJlYW0KeJztz1sKwkAQRcEYH/tfshtomGAw4qHO9zB9a3vm23494PshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCJ4mPofvUPjQ+nH5ERERERERERERERERERERERPykaeRtahua2DsiIiIiIiIiIiIiIiIiIiIiIuLhpkFjr6Fzpw+HuAoR8ZoQVyEiXhPiKkTEa0JchYh4TYirEAvEfwixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEJvvjO8jwplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjMwNwplbmRvYmoKMTQgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjYgL0hlaWdodCAyMjYKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMiAo/ec3ACJN/wAAKV0gL0JpdHNQZXJDb21wb25lbnQgOAovRmlsdGVyIC9GbGF0ZURlY29kZQovRGVjb2RlUGFybXMgPDwgL1ByZWRpY3RvciAxMCAvQ29sb3JzIDEgL0NvbHVtbnMgMjI2ID4+IC9MZW5ndGggMzggMCBSID4+CnN0cmVhbQp4nO3b0QnDMBAFQcX9F+0GAjIIX2Az+//QTQFan3zr1we8H2IhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLHRIXCMhIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIlaJ15fOnn5+4+EcEXEkxO0cEXEkxO0cEXEkxO0cEXEkxO0cEXEkP98KIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLPQHxBu7fbWLCmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKMjkyCmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagozOSAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My43LjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My43LjEpIC9DcmVhdGlvbkRhdGUgKEQ6MjAyMzAzMTQxNjEzMzlaKQo+PgplbmRvYmoKeHJlZgowIDQwCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA4MzI0IDAwMDAwIG4gCjAwMDAwMDY5ODggMDAwMDAgbiAKMDAwMDAwNzAyMCAwMDAwMCBuIAowMDAwMDA3MDgwIDAwMDAwIG4gCjAwMDAwMDcxMDEgMDAwMDAgbiAKMDAwMDAwNzEyMiAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDQgMDAwMDAgbiAKMDAwMDAwMDcyMSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA3MDEgMDAwMDAgbiAKMDAwMDAwNzE2NSAwMDAwMCBuIAowMDAwMDA3NzU4IDAwMDAwIG4gCjAwMDAwMDU3MjkgMDAwMDAgbiAKMDAwMDAwNTUyMiAwMDAwMCBuIAowMDAwMDA1MTE0IDAwMDAwIG4gCjAwMDAwMDY3ODIgMDAwMDAgbiAKMDAwMDAwMDc0MSAwMDAwMCBuIAowMDAwMDAxMDc4IDAwMDAwIG4gCjAwMDAwMDEyNDIgMDAwMDAgbiAKMDAwMDAwMTYyMiAwMDAwMCBuIAowMDAwMDAxOTI3IDAwMDAwIG4gCjAwMDAwMDIyMzEgMDAwMDAgbiAKMDAwMDAwMjU1MyAwMDAwMCBuIAowMDAwMDAyNzYyIDAwMDAwIG4gCjAwMDAwMDMxNzYgMDAwMDAgbiAKMDAwMDAwMzQxMyAwMDAwMCBuIAowMDAwMDAzNTU3IDAwMDAwIG4gCjAwMDAwMDM2NzYgMDAwMDAgbiAKMDAwMDAwMzkxMiAwMDAwMCBuIAowMDAwMDA0MjI0IDAwMDAwIG4gCjAwMDAwMDQ0NTcgMDAwMDAgbiAKMDAwMDAwNDU0NyAwMDAwMCBuIAowMDAwMDA0NzUzIDAwMDAwIG4gCjAwMDAwMDQ5MDAgMDAwMDAgbiAKMDAwMDAwNzczOCAwMDAwMCBuIAowMDAwMDA4MzA0IDAwMDAwIG4gCjAwMDAwMDgzODQgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSA0MCAvUm9vdCAxIDAgUiAvSW5mbyAzOSAwIFIgPj4Kc3RhcnR4cmVmCjg1MzUKJSVFT0YK", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Layer 3\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDEyICj95zfjzVLGtWeCgHh2dnZfY25SWWw3RmwnPW0LM3AAK2QAIk3/AAApXQovQml0c1BlckNvbXBvbmVudCA4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyAyMjYgPj4gL0xlbmd0aCAzNyAwIFIgPj4Kc3RyZWFtCnic7dg5UgQxEEXBYR+W+58Xt41vEBDN8shnK1RKmXV5zXf56QecH2IhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQD72srqPH1cNoHlw3ztGIiIiIiIiIiIiIiIiIiIiIiP+KuGZf70c3q8vodrX+4nmFiIiIiIiIiIiIiIiIiIiIiPgJ4nz56m71NPp16ylERERERERERERERERERERExG8nzjlrNTYPvo0+PPprIR5CREQ8M8RDiIiIZ4Z4CBER8cwQDyEi/gninw2xEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxELvp9vUBQplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjMzMAplbmRvYmoKMTQgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjYgL0hlaWdodCAyMjYKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMiAo/ec3ACJN/wAAKV0gL0JpdHNQZXJDb21wb25lbnQgOAovRmlsdGVyIC9GbGF0ZURlY29kZQovRGVjb2RlUGFybXMgPDwgL1ByZWRpY3RvciAxMCAvQ29sb3JzIDEgL0NvbHVtbnMgMjI2ID4+IC9MZW5ndGggMzggMCBSID4+CnN0cmVhbQp4nO3PMQpCURAEQfX+hzZ9gYEgX8emOh7Yrds93+3XD1wfYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRDP5ViIiIgbISIiboSIiLgRIiLiRoiIiBshIiJuhIiIuBEiIuJGiIiIGyEiIm6EiIi4ESIi4kaIiIgbISIiboSIiLgR4veJjxe9ffqzEM8lIiLihSGeS0RExAtDPJeIiIgXhnguERH/gfi3IRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRY6Amau6FdCmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKMjk1CmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagozOSAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My43LjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My43LjEpIC9DcmVhdGlvbkRhdGUgKEQ6MjAyMzAzMTQxNjEzNDBaKQo+PgplbmRvYmoKeHJlZgowIDQwCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA4MzY5IDAwMDAwIG4gCjAwMDAwMDY5ODggMDAwMDAgbiAKMDAwMDAwNzAyMCAwMDAwMCBuIAowMDAwMDA3MDgwIDAwMDAwIG4gCjAwMDAwMDcxMDEgMDAwMDAgbiAKMDAwMDAwNzEyMiAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDQgMDAwMDAgbiAKMDAwMDAwMDcyMSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA3MDEgMDAwMDAgbiAKMDAwMDAwNzE2NSAwMDAwMCBuIAowMDAwMDA3ODAwIDAwMDAwIG4gCjAwMDAwMDU3MjkgMDAwMDAgbiAKMDAwMDAwNTUyMiAwMDAwMCBuIAowMDAwMDA1MTE0IDAwMDAwIG4gCjAwMDAwMDY3ODIgMDAwMDAgbiAKMDAwMDAwMDc0MSAwMDAwMCBuIAowMDAwMDAxMDc4IDAwMDAwIG4gCjAwMDAwMDEyNDIgMDAwMDAgbiAKMDAwMDAwMTYyMiAwMDAwMCBuIAowMDAwMDAxOTI3IDAwMDAwIG4gCjAwMDAwMDIyMzEgMDAwMDAgbiAKMDAwMDAwMjU1MyAwMDAwMCBuIAowMDAwMDAyNzYyIDAwMDAwIG4gCjAwMDAwMDMxNzYgMDAwMDAgbiAKMDAwMDAwMzQxMyAwMDAwMCBuIAowMDAwMDAzNTU3IDAwMDAwIG4gCjAwMDAwMDM2NzYgMDAwMDAgbiAKMDAwMDAwMzkxMiAwMDAwMCBuIAowMDAwMDA0MjI0IDAwMDAwIG4gCjAwMDAwMDQ0NTcgMDAwMDAgbiAKMDAwMDAwNDU0NyAwMDAwMCBuIAowMDAwMDA0NzUzIDAwMDAwIG4gCjAwMDAwMDQ5MDAgMDAwMDAgbiAKMDAwMDAwNzc4MCAwMDAwMCBuIAowMDAwMDA4MzQ5IDAwMDAwIG4gCjAwMDAwMDg0MjkgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSA0MCAvUm9vdCAxIDAgUiAvSW5mbyAzOSAwIFIgPj4Kc3RhcnR4cmVmCjg1ODAKJSVFT0YK", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Layer 4\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDI0ICj95zfr1Eu/sGqmnHObk3aYkXZ9fHh4eHZsbXJgZG5WXFxtRlBrN0ZsLkFsKj9tJT1tITtuETVvAC1pACxmACpgAFwoWwAjUAAiTf8AACldCi9CaXRzUGVyQ29tcG9uZW50IDggL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM3IDAgUiA+PgpzdHJlYW0KeJzt2ktSHDEUBdF2G2PzN782+18pG7gDBoAjsk+OS9I7Girq8C/f4X8P8PUhFkIshFgIsRBiIcRCiIUQCyEWOlviafSyehzdrW5H88O14zx6zYiIiIiIiIiIiIiIiIiIiIh4DsS15Rzoz+jn6ji6WF2N/q5eR4iIiIiIiIiIiIiIiIiIiIiICeLT6Gb1a/RjdRgt9nFd2sMKERERERERERERERERERER8VyJH36eWo9Jl6vfo+vV/eh59V1/TyEiIiIiIiIiIiIiIiIiIiIifjtxLV8vXrO1+PQ2mkd/foiIiIiIiIiIiIiIiIiIiIgxYirEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAu9Aw6NfSIKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iagozNTYKZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP3nNwAiTf8AACldIC9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM4IDAgUiA+PgpzdHJlYW0KeJztz8ENw0AAw7BL9x+6C+SZIrBKLWDzXPnO2wd+H2IhxEKIhRALIRZCLIRYCLEQYqG/JZ7REBEnQkScCBFxIkTEiRARJ0JEnAgRcSJExIkQESdCRJwIEXEiRMSJEBEnQkScCBFxIkTEiRARJ0JEnAgRcSJExIkQESdCRJwIEXEiRMSJEBEnQkScCBFxoleJn5tup58PERERERERERERERERERERMUZMhVgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBioS8tiYe9CmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKMjgyCmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagozOSAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My43LjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My43LjEpIC9DcmVhdGlvbkRhdGUgKEQ6MjAyMzAzMTQxNjEzNDBaKQo+PgplbmRvYmoKeHJlZgowIDQwCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA4NDIwIDAwMDAwIG4gCjAwMDAwMDY5ODggMDAwMDAgbiAKMDAwMDAwNzAyMCAwMDAwMCBuIAowMDAwMDA3MDgwIDAwMDAwIG4gCjAwMDAwMDcxMDEgMDAwMDAgbiAKMDAwMDAwNzEyMiAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDQgMDAwMDAgbiAKMDAwMDAwMDcyMSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA3MDEgMDAwMDAgbiAKMDAwMDAwNzE2NSAwMDAwMCBuIAowMDAwMDA3ODY0IDAwMDAwIG4gCjAwMDAwMDU3MjkgMDAwMDAgbiAKMDAwMDAwNTUyMiAwMDAwMCBuIAowMDAwMDA1MTE0IDAwMDAwIG4gCjAwMDAwMDY3ODIgMDAwMDAgbiAKMDAwMDAwMDc0MSAwMDAwMCBuIAowMDAwMDAxMDc4IDAwMDAwIG4gCjAwMDAwMDEyNDIgMDAwMDAgbiAKMDAwMDAwMTYyMiAwMDAwMCBuIAowMDAwMDAxOTI3IDAwMDAwIG4gCjAwMDAwMDIyMzEgMDAwMDAgbiAKMDAwMDAwMjU1MyAwMDAwMCBuIAowMDAwMDAyNzYyIDAwMDAwIG4gCjAwMDAwMDMxNzYgMDAwMDAgbiAKMDAwMDAwMzQxMyAwMDAwMCBuIAowMDAwMDAzNTU3IDAwMDAwIG4gCjAwMDAwMDM2NzYgMDAwMDAgbiAKMDAwMDAwMzkxMiAwMDAwMCBuIAowMDAwMDA0MjI0IDAwMDAwIG4gCjAwMDAwMDQ0NTcgMDAwMDAgbiAKMDAwMDAwNDU0NyAwMDAwMCBuIAowMDAwMDA0NzUzIDAwMDAwIG4gCjAwMDAwMDQ5MDAgMDAwMDAgbiAKMDAwMDAwNzg0NCAwMDAwMCBuIAowMDAwMDA4NDAwIDAwMDAwIG4gCjAwMDAwMDg0ODAgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSA0MCAvUm9vdCAxIDAgUiAvSW5mbyAzOSAwIFIgPj4Kc3RhcnR4cmVmCjg2MzEKJSVFT0YK", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Layer 5\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDM3ICj95zfs1UrUwV7Sv2DNvGKlm3Ogl3WUjneIhXiCgHh7e3d2dnZmaXBYXW1XXW1UWmxRWGxMVGxET2tBTWs3RmwuQWwrP20kPG4WNm8UNm8LM3AAL28ALmsALGYAK2QAKmIAXChbACZVACRSACNQACJN/wAAKV0KL0JpdHNQZXJDb21wb25lbnQgOCAvRmlsdGVyIC9GbGF0ZURlY29kZQovRGVjb2RlUGFybXMgPDwgL1ByZWRpY3RvciAxMCAvQ29sb3JzIDEgL0NvbHVtbnMgMjI2ID4+IC9MZW5ndGggMzcgMCBSID4+CnN0cmVhbQp4nO3buU7DQBiFUbPvEPadhATe/xVpXdwuGJmr85Wj0djH5ej38JXahFapt9BL6ikUN6YT46PTO0bMgIiIiIiIiIiIiIiIiIiIiIg4X2LauXkPPaQuQqep49BZahF6TC1DEYOIiIiIiIiIiIiIiIiIiIj4f4mfqefQZeowtJMaQnupk9Bt6jW0TiEiIiIiIiIiIiIiIiIiIiIiNhCvUkeh/dRu6CB1HrpPfYS2vIFDREREREREREREREREREREnBsxrqbpqTjCdBNK80+L69BdKn3dNCi1TDdRW/75hoiIiIiIiIiIiIiIiIiIiIj458S4mtjpyHXaGE/8DsWNvx8iIuIoxClDREQchThliIiIoxCnDBERcdTsiFUhNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDiA0hNoTYEGJDP23ZIpsKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago0MDIKZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP3nNwAiTf8AACldIC9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM4IDAgUiA+PgpzdHJlYW0KeJztz8EJwDAQwLBr9x860HffCTHSAsYzAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMDn/fHsYdGiRYsWLVq0aNGiRYsWLVq0eO/ips5BFgssFlgssFhgscBigcUCiwUWCywWWCywWGCxwGKBxQKLBRYLLBZYLLBYYLHAYoHFAosFFgssFlgssFhgscBigcUCiwUWCywWWCywWGCxwGKBxQKLBRYLLBZYLLBYYLHAYoHFAosFFgssFlgsWN2kZhsKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iagoxODgKZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjM5IDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjcuMSkgL0NyZWF0aW9uRGF0ZSAoRDoyMDIzMDMxNDE2MTM0MFopCj4+CmVuZG9iagp4cmVmCjAgNDAKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDg0MTAgMDAwMDAgbiAKMDAwMDAwNjk4OCAwMDAwMCBuIAowMDAwMDA3MDIwIDAwMDAwIG4gCjAwMDAwMDcwODAgMDAwMDAgbiAKMDAwMDAwNzEwMSAwMDAwMCBuIAowMDAwMDA3MTIyIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM0NCAwMDAwMCBuIAowMDAwMDAwNzIxIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDcwMSAwMDAwMCBuIAowMDAwMDA3MTY1IDAwMDAwIG4gCjAwMDAwMDc5NDggMDAwMDAgbiAKMDAwMDAwNTcyOSAwMDAwMCBuIAowMDAwMDA1NTIyIDAwMDAwIG4gCjAwMDAwMDUxMTQgMDAwMDAgbiAKMDAwMDAwNjc4MiAwMDAwMCBuIAowMDAwMDAwNzQxIDAwMDAwIG4gCjAwMDAwMDEwNzggMDAwMDAgbiAKMDAwMDAwMTI0MiAwMDAwMCBuIAowMDAwMDAxNjIyIDAwMDAwIG4gCjAwMDAwMDE5MjcgMDAwMDAgbiAKMDAwMDAwMjIzMSAwMDAwMCBuIAowMDAwMDAyNTUzIDAwMDAwIG4gCjAwMDAwMDI3NjIgMDAwMDAgbiAKMDAwMDAwMzE3NiAwMDAwMCBuIAowMDAwMDAzNDEzIDAwMDAwIG4gCjAwMDAwMDM1NTcgMDAwMDAgbiAKMDAwMDAwMzY3NiAwMDAwMCBuIAowMDAwMDAzOTEyIDAwMDAwIG4gCjAwMDAwMDQyMjQgMDAwMDAgbiAKMDAwMDAwNDQ1NyAwMDAwMCBuIAowMDAwMDA0NTQ3IDAwMDAwIG4gCjAwMDAwMDQ3NTMgMDAwMDAgbiAKMDAwMDAwNDkwMCAwMDAwMCBuIAowMDAwMDA3OTI4IDAwMDAwIG4gCjAwMDAwMDgzOTAgMDAwMDAgbiAKMDAwMDAwODQ3MCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDQwIC9Sb290IDEgMCBSIC9JbmZvIDM5IDAgUiA+PgpzdGFydHhyZWYKODYyMQolJUVPRgo=", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["# Initialize convolutions with equal weight to all input pixels\n", "horiz_conv = HorizontalStackConvolution(c_in=1, c_out=1, kernel_size=3, mask_center=False)\n", "horiz_conv.conv.weight.data.fill_(1)\n", "horiz_conv.conv.bias.data.fill_(0)\n", "vert_conv = VerticalStackConvolution(c_in=1, c_out=1, kernel_size=3, mask_center=False)\n", "vert_conv.conv.weight.data.fill_(1)\n", "vert_conv.conv.bias.data.fill_(0)\n", "\n", "# We reuse our convolutions for the 4 layers here. Note that in a standard network,\n", "# we don't do that, and instead learn 4 separate convolution. As this cell is only for\n", "# visualization purposes, we reuse the convolutions for all layers.\n", "for l_idx in range(4):\n", " vert_img = vert_conv(vert_img)\n", " horiz_img = horiz_conv(horiz_img) + vert_img\n", " print(\"Layer %i\" % (l_idx + 2))\n", " show_center_recep_field(inp_img, horiz_img)"]}, {"cell_type": "markdown", "id": "8b579284", "metadata": {"papermill": {"duration": 0.013563, "end_time": "2023-03-14T16:13:40.710054", "exception": false, "start_time": "2023-03-14T16:13:40.696491", "status": "completed"}, "tags": []}, "source": ["The receptive field above it visualized for the horizontal stack, which includes the features of the vertical convolutions.\n", "It grows over layers without any blind spot as we had before.\n", "The difference between \"weighted\" and \"binary\" receptive field is that for the latter, we check whether there are any gradients flowing back to this pixel.\n", "This indicates that the center pixel indeed can use information from this pixel.\n", "Nevertheless, due to the convolution weights, some pixels have a stronger effect on the prediction than others.\n", "This is visualized in the weighted receptive field by plotting the gradient magnitude for each pixel instead of a binary yes/no.\n", "\n", "\n", "Another receptive field we can check is the one for the vertical stack\n", "as the one above is for the horizontal stack. Let's visualize it below:"]}, {"cell_type": "code", "execution_count": 13, "id": "405d15d7", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:40.737736Z", "iopub.status.busy": "2023-03-14T16:13:40.737547Z", "iopub.status.idle": "2023-03-14T16:13:40.956702Z", "shell.execute_reply": "2023-03-14T16:13:40.956156Z"}, "papermill": {"duration": 0.235975, "end_time": "2023-03-14T16:13:40.959252", "exception": false, "start_time": "2023-03-14T16:13:40.723277", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDE2ICj95zffylalm3ORjHdkZ29OVmw0RGwqP20iO24IM3AALWkAKmIAJlUAJVQAJFIAIk3/AAApXQovQml0c1BlckNvbXBvbmVudCA4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyAyMjYgPj4gL0xlbmd0aCAzNyAwIFIgPj4Kc3RyZWFtCnic7ds5bgQxEATB1X0f/3+tPlDmjKBNRdoEu8MlwMvX6mP0tnoZPa+eRvPgunGOXjtOzAURERERERERERERERERERER8e8SP1evo8fV/ehudTuaB9eNc/TacWIQERERERERERERERERERERr5f4vlpvRA+rteTN6jKaB9eNc/TacWIQEREREREREREREREREREREf8w8R+8wCEiIiIiIiIiIiIiIiIiIiJeLfEf/HxDRERERERERERERERERERERLwy4vF9j35pNOJhIZ4Z4mEhnhniYSGeGeJhIZ4Z4mEhnhniYSGeGWIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRAL/QAzgEqWCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKMzQ3CmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvSW1hZ2UgL1dpZHRoIDIyNiAvSGVpZ2h0IDIyNgovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyICj95zcAIk3/AAApXSAvQml0c1BlckNvbXBvbmVudCA4Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyAyMjYgPj4gL0xlbmd0aCAzOCAwIFIgPj4Kc3RyZWFtCnic7c/BDYBADMAwYP+hWYBne0KRvUCU6wIAAAAAAAAAAAAAAAAAAAAAAAAAAACAv7vPeD4cSlscY3GTxTEWN1kcY3GTxTEWN1kcY3GTxTEWN1kssFhgscBigcUCiwUWCywWWCywWGCxwGKBxQKLBRYLLBZYLLBYYLHAYoHFAosFFgssFlgssFhgscBigcUCiwUWCywWWCywWGCxwGKBxQKLBRYLLBZYLLBYYLHAYoHFAosFFgssFlgssFhgscBiwQtGUW4nCmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKMTk3CmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagozOSAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My43LjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My43LjEpIC9DcmVhdGlvbkRhdGUgKEQ6MjAyMzAzMTQxNjEzNDBaKQo+PgplbmRvYmoKeHJlZgowIDQwCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA4MzAwIDAwMDAwIG4gCjAwMDAwMDY5ODggMDAwMDAgbiAKMDAwMDAwNzAyMCAwMDAwMCBuIAowMDAwMDA3MDgwIDAwMDAwIG4gCjAwMDAwMDcxMDEgMDAwMDAgbiAKMDAwMDAwNzEyMiAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDQgMDAwMDAgbiAKMDAwMDAwMDcyMSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA3MDEgMDAwMDAgbiAKMDAwMDAwNzE2NSAwMDAwMCBuIAowMDAwMDA3ODI5IDAwMDAwIG4gCjAwMDAwMDU3MjkgMDAwMDAgbiAKMDAwMDAwNTUyMiAwMDAwMCBuIAowMDAwMDA1MTE0IDAwMDAwIG4gCjAwMDAwMDY3ODIgMDAwMDAgbiAKMDAwMDAwMDc0MSAwMDAwMCBuIAowMDAwMDAxMDc4IDAwMDAwIG4gCjAwMDAwMDEyNDIgMDAwMDAgbiAKMDAwMDAwMTYyMiAwMDAwMCBuIAowMDAwMDAxOTI3IDAwMDAwIG4gCjAwMDAwMDIyMzEgMDAwMDAgbiAKMDAwMDAwMjU1MyAwMDAwMCBuIAowMDAwMDAyNzYyIDAwMDAwIG4gCjAwMDAwMDMxNzYgMDAwMDAgbiAKMDAwMDAwMzQxMyAwMDAwMCBuIAowMDAwMDAzNTU3IDAwMDAwIG4gCjAwMDAwMDM2NzYgMDAwMDAgbiAKMDAwMDAwMzkxMiAwMDAwMCBuIAowMDAwMDA0MjI0IDAwMDAwIG4gCjAwMDAwMDQ0NTcgMDAwMDAgbiAKMDAwMDAwNDU0NyAwMDAwMCBuIAowMDAwMDA0NzUzIDAwMDAwIG4gCjAwMDAwMDQ5MDAgMDAwMDAgbiAKMDAwMDAwNzgwOSAwMDAwMCBuIAowMDAwMDA4MjgwIDAwMDAwIG4gCjAwMDAwMDgzNjAgMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSA0MCAvUm9vdCAxIDAgUiAvSW5mbyAzOSAwIFIgPj4Kc3RhcnR4cmVmCjg1MTEKJSVFT0YK", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["show_center_recep_field(inp_img, vert_img)"]}, {"cell_type": "markdown", "id": "2be44851", "metadata": {"papermill": {"duration": 0.014076, "end_time": "2023-03-14T16:13:40.994268", "exception": false, "start_time": "2023-03-14T16:13:40.980192", "status": "completed"}, "tags": []}, "source": ["As we have discussed before, the vertical stack only looks at pixels above the one we want to predict.\n", "Hence, we can validate that our implementation works as we initially expected it to.\n", "As a final step, let's clean up the computation graph we still had kept\n", "in memory for the visualization of the receptive field:"]}, {"cell_type": "code", "execution_count": 14, "id": "31aee31b", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:41.023719Z", "iopub.status.busy": "2023-03-14T16:13:41.023470Z", "iopub.status.idle": "2023-03-14T16:13:41.027522Z", "shell.execute_reply": "2023-03-14T16:13:41.026787Z"}, "papermill": {"duration": 0.020523, "end_time": "2023-03-14T16:13:41.028823", "exception": false, "start_time": "2023-03-14T16:13:41.008300", "status": "completed"}, "tags": []}, "outputs": [], "source": ["del inp_img, horiz_conv, vert_conv"]}, {"cell_type": "markdown", "id": "bd4be8fc", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.014487, "end_time": "2023-03-14T16:13:41.060199", "exception": false, "start_time": "2023-03-14T16:13:41.045712", "status": "completed"}, "tags": []}, "source": ["## Gated PixelCNN\n", "\n", "
\n", "\n", "In the next step, we will use the masked convolutions to build a full autoregressive model, called Gated PixelCNN.\n", "The difference between the original PixelCNN and Gated PixelCNN is the use of separate horizontal and vertical stacks.\n", "However, in literature, you often see that people refer to the Gated PixelCNN simply as \"PixelCNN\".\n", "Hence, in the following, if we say \"PixelCNN\", we usually mean the gated version.\n", "What \"Gated\" refers to in the model name is explained next.\n", "\n", "### Gated Convolutions\n", "\n", "For visualizing the receptive field, we assumed a very simplified stack of vertical and horizontal convolutions.\n", "Obviously, there are more sophisticated ways of doing it, and PixelCNN uses gated convolutions for this.\n", "Specifically, the Gated Convolution block in PixelCNN looks as follows\n", "(figure credit - [Aaron van den Oord et al. ](https://arxiv.org/pdf/1606.05328.pdf)):\n", "\n", "
\n", "\n", "The left path is the vertical stack (the $N\\times N$ convolution is masked correspondingly),\n", "and the right path is the horizontal stack.\n", "Gated convolutions are implemented by having a twice as large output channel size,\n", "and combine them by a element-wise multiplication of $\\tanh$ and a sigmoid.\n", "For a linear layer, we can express a gated activation unit as follows:\n", "\n", "$$\\mathbf{y} = \\tanh\\left(\\mathbf{W}_{f}\\mathbf{x}\\right)\\odot\\sigma\\left(\\mathbf{W}_{g}\\mathbf{x}\\right)$$\n", "\n", "For simplicity, biases have been neglected and the linear layer split into two part, $\\mathbf{W}_{f}$ and $\\mathbf{W}_{g}$.\n", "This concept resembles the input and modulation gate in an LSTM, and has been used in many other architectures as well.\n", "The main motivation behind this gated activation is that it might allow to model more complex interactions and simplifies learning.\n", "But as in any other architecture, this is mostly a design choice and can be considered a hyperparameters.\n", "\n", "Besides the gated convolutions, we also see that the horizontal stack uses a residual connection while the vertical stack does not.\n", "This is because we use the output of the horizontal stack for prediction.\n", "Each convolution in the vertical stack also receives a strong gradient signal\n", "as it is only two $1\\times 1$ convolutions away from the residual connection,\n", "and does not require another residual connection to all its earleri layers.\n", "\n", "The implementation in PyTorch is fairly straight forward for this block,\n", "because the visualization above gives us a computation graph to follow:"]}, {"cell_type": "code", "execution_count": 15, "id": "92099906", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:41.088895Z", "iopub.status.busy": "2023-03-14T16:13:41.088677Z", "iopub.status.idle": "2023-03-14T16:13:41.097747Z", "shell.execute_reply": "2023-03-14T16:13:41.096885Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.026057, "end_time": "2023-03-14T16:13:41.100076", "exception": false, "start_time": "2023-03-14T16:13:41.074019", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class GatedMaskedConv(nn.Module):\n", " def __init__(self, c_in, **kwargs):\n", " \"\"\"Gated Convolution block implemented the computation graph shown above.\"\"\"\n", " super().__init__()\n", " self.conv_vert = VerticalStackConvolution(c_in, c_out=2 * c_in, **kwargs)\n", " self.conv_horiz = HorizontalStackConvolution(c_in, c_out=2 * c_in, **kwargs)\n", " self.conv_vert_to_horiz = nn.Conv2d(2 * c_in, 2 * c_in, kernel_size=1, padding=0)\n", " self.conv_horiz_1x1 = nn.Conv2d(c_in, c_in, kernel_size=1, padding=0)\n", "\n", " def forward(self, v_stack, h_stack):\n", " # Vertical stack (left)\n", " v_stack_feat = self.conv_vert(v_stack)\n", " v_val, v_gate = v_stack_feat.chunk(2, dim=1)\n", " v_stack_out = torch.tanh(v_val) * torch.sigmoid(v_gate)\n", "\n", " # Horizontal stack (right)\n", " h_stack_feat = self.conv_horiz(h_stack)\n", " h_stack_feat = h_stack_feat + self.conv_vert_to_horiz(v_stack_feat)\n", " h_val, h_gate = h_stack_feat.chunk(2, dim=1)\n", " h_stack_feat = torch.tanh(h_val) * torch.sigmoid(h_gate)\n", " h_stack_out = self.conv_horiz_1x1(h_stack_feat)\n", " h_stack_out = h_stack_out + h_stack\n", "\n", " return v_stack_out, h_stack_out"]}, {"cell_type": "markdown", "id": "3f7f76a6", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.013807, "end_time": "2023-03-14T16:13:41.131970", "exception": false, "start_time": "2023-03-14T16:13:41.118163", "status": "completed"}, "tags": []}, "source": ["### Building the model\n", "\n", "Using the gated convolutions, we can now build our PixelCNN model.\n", "The architecture consists of multiple stacked GatedMaskedConv blocks, where we add an additional dilation factor to a few convolutions.\n", "This is used to increase the receptive field of the model and allows to take a larger context into accout during generation.\n", "As a reminder, dilation on a convolution works looks as follows\n", "(figure credit - [Vincent Dumoulin and Francesco Visin](https://arxiv.org/pdf/1603.07285.pdf)):\n", "\n", "
\n", "\n", "Note that the smaller output size is only because the animation assumes no padding.\n", "In our implementation, we will pad the input image correspondingly.\n", "Alternatively to dilated convolutions, we could downsample the input and use a encoder-decoder architecture as in PixelCNN++ [3].\n", "This is especially beneficial if we want to build a very deep autoregressive model.\n", "Nonetheless, as we seek to train a reasonably small model, dilated convolutions are the more efficient option to use here.\n", "\n", "Below, we implement the PixelCNN model as a PyTorch Lightning module.\n", "Besides the stack of gated convolutions, we also have the initial\n", "horizontal and vertical convolutions which mask the center pixel, and a\n", "final $1\\times 1$ convolution which maps the output features to class\n", "predictions. To determine the likelihood of a batch of images, we first\n", "create our initial features using the masked horizontal and vertical\n", "input convolution. Next, we forward the features through the stack of\n", "gated convolutions. Finally, we take the output features of the\n", "horizontal stack, and apply the $1\\times 1$ convolution for\n", "classification. We use the bits per dimension metric for the likelihood,\n", "similarly to Tutorial 11 and assignment 3."]}, {"cell_type": "code", "execution_count": 16, "id": "23384a77", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:41.160618Z", "iopub.status.busy": "2023-03-14T16:13:41.160397Z", "iopub.status.idle": "2023-03-14T16:13:41.177766Z", "shell.execute_reply": "2023-03-14T16:13:41.177218Z"}, "papermill": {"duration": 0.034335, "end_time": "2023-03-14T16:13:41.180065", "exception": false, "start_time": "2023-03-14T16:13:41.145730", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class PixelCNN(L.LightningModule):\n", " def __init__(self, c_in, c_hidden):\n", " super().__init__()\n", " self.save_hyperparameters()\n", "\n", " # Initial convolutions skipping the center pixel\n", " self.conv_vstack = VerticalStackConvolution(c_in, c_hidden, mask_center=True)\n", " self.conv_hstack = HorizontalStackConvolution(c_in, c_hidden, mask_center=True)\n", " # Convolution block of PixelCNN. We use dilation instead of downscaling\n", " self.conv_layers = nn.ModuleList(\n", " [\n", " GatedMaskedConv(c_hidden),\n", " GatedMaskedConv(c_hidden, dilation=2),\n", " GatedMaskedConv(c_hidden),\n", " GatedMaskedConv(c_hidden, dilation=4),\n", " GatedMaskedConv(c_hidden),\n", " GatedMaskedConv(c_hidden, dilation=2),\n", " GatedMaskedConv(c_hidden),\n", " ]\n", " )\n", " # Output classification convolution (1x1)\n", " self.conv_out = nn.Conv2d(c_hidden, c_in * 256, kernel_size=1, padding=0)\n", "\n", " self.example_input_array = train_set[0][0][None]\n", "\n", " def forward(self, x):\n", " \"\"\"Forward image through model and return logits for each pixel.\n", "\n", " Args:\n", " x: Image tensor with integer values between 0 and 255.\n", " \"\"\"\n", " # Scale input from 0 to 255 back to -1 to 1\n", " x = (x.float() / 255.0) * 2 - 1\n", "\n", " # Initial convolutions\n", " v_stack = self.conv_vstack(x)\n", " h_stack = self.conv_hstack(x)\n", " # Gated Convolutions\n", " for layer in self.conv_layers:\n", " v_stack, h_stack = layer(v_stack, h_stack)\n", " # 1x1 classification convolution\n", " # Apply ELU before 1x1 convolution for non-linearity on residual connection\n", " out = self.conv_out(F.elu(h_stack))\n", "\n", " # Output dimensions: [Batch, Classes, Channels, Height, Width]\n", " out = out.reshape(out.shape[0], 256, out.shape[1] // 256, out.shape[2], out.shape[3])\n", " return out\n", "\n", " def calc_likelihood(self, x):\n", " # Forward pass with bpd likelihood calculation\n", " pred = self.forward(x)\n", " nll = F.cross_entropy(pred, x, reduction=\"none\")\n", " bpd = nll.mean(dim=[1, 2, 3]) * np.log2(np.exp(1))\n", " return bpd.mean()\n", "\n", " @torch.no_grad()\n", " def sample(self, img_shape, img=None):\n", " \"\"\"Sampling function for the autoregressive model.\n", "\n", " Args:\n", " img_shape: Shape of the image to generate (B,C,H,W)\n", " img (optional): If given, this tensor will be used as\n", " a starting image. The pixels to fill\n", " should be -1 in the input tensor.\n", " \"\"\"\n", " # Create empty image\n", " if img is None:\n", " img = torch.zeros(img_shape, dtype=torch.long).to(device) - 1\n", " # Generation loop\n", " for h in tqdm(range(img_shape[2]), leave=False):\n", " for w in range(img_shape[3]):\n", " for c in range(img_shape[1]):\n", " # Skip if not to be filled (-1)\n", " if (img[:, c, h, w] != -1).all().item():\n", " continue\n", " # For efficiency, we only have to input the upper part of the image\n", " # as all other parts will be skipped by the masked convolutions anyways\n", " pred = self.forward(img[:, :, : h + 1, :])\n", " probs = F.softmax(pred[:, :, c, h, w], dim=-1)\n", " img[:, c, h, w] = torch.multinomial(probs, num_samples=1).squeeze(dim=-1)\n", " return img\n", "\n", " def configure_optimizers(self):\n", " optimizer = optim.Adam(self.parameters(), lr=1e-3)\n", " scheduler = optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.99)\n", " return [optimizer], [scheduler]\n", "\n", " def training_step(self, batch, batch_idx):\n", " loss = self.calc_likelihood(batch[0])\n", " self.log(\"train_bpd\", loss)\n", " return loss\n", "\n", " def validation_step(self, batch, batch_idx):\n", " loss = self.calc_likelihood(batch[0])\n", " self.log(\"val_bpd\", loss)\n", "\n", " def test_step(self, batch, batch_idx):\n", " loss = self.calc_likelihood(batch[0])\n", " self.log(\"test_bpd\", loss)"]}, {"cell_type": "markdown", "id": "430ab7ef", "metadata": {"papermill": {"duration": 0.014313, "end_time": "2023-03-14T16:13:41.220796", "exception": false, "start_time": "2023-03-14T16:13:41.206483", "status": "completed"}, "tags": []}, "source": ["To sample from the autoregressive model, we need to iterate over all dimensions of the input.\n", "We start with an empty image, and fill the pixels one by one, starting from the upper left corner.\n", "Note that as for predicting $x_i$, all pixels below it have no influence on the prediction.\n", "Hence, we can cut the image in height without changing the prediction while increasing efficiency.\n", "Nevertheless, all the loops in the sampling function already show that it will take us quite some time to sample.\n", "A lot of computation could be reused across loop iterations as those the features on the already predicted pixels will not change over iterations.\n", "Nevertheless, this takes quite some effort to implement, and is often not done in implementations because in the end,\n", "autoregressive sampling remains sequential and slow.\n", "Hence, we settle with the default implementation here.\n", "\n", "Before training the model, we can check the full receptive field of the model on an MNIST image of size $28\\times 28$:"]}, {"cell_type": "code", "execution_count": 17, "id": "6c4334df", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:41.249916Z", "iopub.status.busy": "2023-03-14T16:13:41.249698Z", "iopub.status.idle": "2023-03-14T16:13:41.559017Z", "shell.execute_reply": "2023-03-14T16:13:41.558345Z"}, "papermill": {"duration": 0.32633, "end_time": "2023-03-14T16:13:41.561298", "exception": false, "start_time": "2023-03-14T16:13:41.234968", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMzcxLjUyIDE5MS44NTIyNzI3MjczIF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nJ1QTU8CMRC9z694RzjY7UztdvcookRv6CYcjAezlGUJoAsE4r93gKwfXEQ7ecm8Tqfz5iX9uK3L+DDo4fqRki9WrokxU1SwmCl2YAwUFVllC3KBjRdN523KOZvMiwSnd/YnnRJNqEEwcgCnYpyEY7hTuooYYYnkSqetdeRMsdNJgxOBzaEx7EXYNg0a5QLJHaP/iiEN0Wix+v7LnlOjq1lcaB/YG7Y+E5elLgUHr8t8aikX1CsoudVngmJyWL4Y0xM6oy58Znzugkcn1tV0E8dYdSHWZK49Winj26beRkzqOB938Yzinm4KOkoTyybP23H/9Yb2W/3ujVhp3ZGz3RHOjbWW2ak7l+fb06uXL6v3vxlCH5GVksMKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyODIKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAyNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVK5kQMxDMu3CpbAX1I9vrlxYPefHsA9OzExSxEAQfduUTkbP20ly4/82GVnSeSW96Bt8rqi6gapmyBzyXHJFWKW8rgSJBYtZSbWZ6qD8nENahW+8BMzE9MhScQSclKPIqyPr4PX9RzcXzfp/BoZH3RsUHpLkqugVMO+crQSO5bqXYPsg6ab0uoz067sgKXxhqzUpE5/HfW/IyzCpN5IRQchFr/Tyx2yLKQc1Nu5fgPhXe2CWE+tPOwQad5WeksVzObMBEwZbTusgLVZY8JUCfAWzSHUWojMFzqtiPnk3NHBNFV5GiLDUoWD7T2jNVzoGhJLw/lJxgTmv/D6rMhtf/8AcGhnugplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9MZW5ndGggOTIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY3BDcAwCAP/TMEIEALE+1RVH+n+3yYR6gcfBtkYYGGzNeDB2cCX0to3vaRFk9oIVrVF3VCeuxSlWF1HpUzCT5k7f1J0HO1wDtvf1uU4TePoX/fQ/QEPSh4LCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDIzMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjMxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAxMzYgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDM0MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvTGVuZ3RoIDE2NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvTGVuZ3RoIDcyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvTGVuZ3RoIDQ3IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9MZW5ndGggMTYzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0xlbmd0aCAyMzkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVDJbQQxDPu7CjUwwOgcux4Hizyy/X9DygmSl2hL4qHylFuWymX3IzlvybrlQ4dOlWnybtDNr7H+owwCdv9QVBCtJbFKzFzSbrE0SS/ZwziNl2u1juepe4RZo3jw49jTKYHpPTLBZrO9OTCrPc4OkE64xq/q0zuVJAOJupDzQqUK6x7UJaKPK9uYUp1OLeUYl5/oe3yOAD3F3o3c0cfLF4xGtS2o0WqVOA8wE1PRlXGrkYGUEwZDZ0dXNAulyMp6QjXCjTmhmb3DcGADy7OEpKWtUrwPZQHoAl3aOuM0SoKOAMLfKIz1+gaq/F43CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0xlbmd0aCAxNjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9MZW5ndGggMTQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNiAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTggMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY2IC9CIDg3IC9XIDk3IC9hIDk5IC9jIC9kIC9lIC9mIC9nIC9oIC9pIDEwOCAvbCAxMTAgL24gMTEyIC9wCjExNCAvciAxMTYgL3QgMTE4IC92IDEyMSAveSBdCj4+Ci9XaWR0aHMgMTUgMCBSID4+CmVuZG9iagoxNiAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxNSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxOCAwIG9iago8PCAvQiAxOSAwIFIgL1cgMjAgMCBSIC9hIDIxIDAgUiAvYyAyMiAwIFIgL2QgMjMgMCBSIC9lIDI0IDAgUiAvZiAyNSAwIFIKL2cgMjYgMCBSIC9oIDI3IDAgUiAvaSAyOCAwIFIgL2wgMjkgMCBSIC9uIDMwIDAgUiAvcCAzMSAwIFIgL3IgMzIgMCBSCi9zcGFjZSAzMyAwIFIgL3QgMzQgMCBSIC92IDM1IDAgUiAveSAzNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE3IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSIC9JMiAxNCAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDE1ICj95zchO24dOW4EMnAAMXAALGcAK2QAXChcXABcKFsAJlcAJlUAJFIAI1AAI08AIk3/AAApXQovQml0c1BlckNvbXBvbmVudCA4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlCi9EZWNvZGVQYXJtcyA8PCAvUHJlZGljdG9yIDEwIC9Db2xvcnMgMSAvQ29sdW1ucyAyMjYgPj4gL0xlbmd0aCAzNyAwIFIgPj4Kc3RyZWFtCnic7dXJigIxFEBRex7t/v+/7U2CUCBoldJ4PWeXRxbvZpPdT97uvxe4PokFEgskFkgskFggsUBigcQCiQUSCyQWSCyQWCCxQGKBxAKJBRILJBZILJBYILFAYoHEAokFEgskFkgskFggsUBigcQCiQUSCyQWSCyQWCCxQGKBxBW+F+Z8PyznX8Ocz3sXW0jiChIlSlxD4gp3kDgtk/YLx+4tn2AziRtIlCjxHBI3CCUuV55f+Ty/D6/DTPgYjj3F5sUmiSeQKFHiJUg8wR0kTse+7pnyMjwP897bcLFFliSeQaJEiVtIPMPtJ86NPof5Q+/n4HGYKU/Dw7AbfoerJS1JPJAoUeJVSTy4g8TbJbFAYoHEAokFEgskFkgskFggsUBigcQCiQUSCyQWSCyQWCCxQGKBxAKJBRILJBZILJBYILFAYoHEAokFEgskFkgskFggsUBigcQCiQUSCyQWSCyQWCCxQGKBxAKJBRILJBZILJBYILFAYoHEAokFf/QFwkcKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iagozODUKZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9JbWFnZSAvV2lkdGggMjI2IC9IZWlnaHQgMjI2Ci9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP3nNwAiTf8AACldIC9CaXRzUGVyQ29tcG9uZW50IDgKL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNiA+PiAvTGVuZ3RoIDM4IDAgUiA+PgpzdHJlYW0KeJztz7ENgDAQwMAP+w9Ng+gpEiTrPIFvVr75e2B/iIUQCyEWeonTDbEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFjhOvp3UqRERERETE78RuiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQiyEWAixEGIhxEKIhRALIRZCLIRYCLEQYiHEQoiFEAshFkIshFgIsRBiIcRCiIUQCyEWQix0A6cmaxMKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iagoyODEKZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjM5IDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjcuMSkgL0NyZWF0aW9uRGF0ZSAoRDoyMDIzMDMxNDE2MTM0MVopCj4+CmVuZG9iagp4cmVmCjAgNDAKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDg0MjIgMDAwMDAgbiAKMDAwMDAwNjk4OCAwMDAwMCBuIAowMDAwMDA3MDIwIDAwMDAwIG4gCjAwMDAwMDcwODAgMDAwMDAgbiAKMDAwMDAwNzEwMSAwMDAwMCBuIAowMDAwMDA3MTIyIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM0NCAwMDAwMCBuIAowMDAwMDAwNzIxIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDcwMSAwMDAwMCBuIAowMDAwMDA3MTY1IDAwMDAwIG4gCjAwMDAwMDc4NjcgMDAwMDAgbiAKMDAwMDAwNTcyOSAwMDAwMCBuIAowMDAwMDA1NTIyIDAwMDAwIG4gCjAwMDAwMDUxMTQgMDAwMDAgbiAKMDAwMDAwNjc4MiAwMDAwMCBuIAowMDAwMDAwNzQxIDAwMDAwIG4gCjAwMDAwMDEwNzggMDAwMDAgbiAKMDAwMDAwMTI0MiAwMDAwMCBuIAowMDAwMDAxNjIyIDAwMDAwIG4gCjAwMDAwMDE5MjcgMDAwMDAgbiAKMDAwMDAwMjIzMSAwMDAwMCBuIAowMDAwMDAyNTUzIDAwMDAwIG4gCjAwMDAwMDI3NjIgMDAwMDAgbiAKMDAwMDAwMzE3NiAwMDAwMCBuIAowMDAwMDAzNDEzIDAwMDAwIG4gCjAwMDAwMDM1NTcgMDAwMDAgbiAKMDAwMDAwMzY3NiAwMDAwMCBuIAowMDAwMDAzOTEyIDAwMDAwIG4gCjAwMDAwMDQyMjQgMDAwMDAgbiAKMDAwMDAwNDQ1NyAwMDAwMCBuIAowMDAwMDA0NTQ3IDAwMDAwIG4gCjAwMDAwMDQ3NTMgMDAwMDAgbiAKMDAwMDAwNDkwMCAwMDAwMCBuIAowMDAwMDA3ODQ3IDAwMDAwIG4gCjAwMDAwMDg0MDIgMDAwMDAgbiAKMDAwMDAwODQ4MiAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDQwIC9Sb290IDEgMCBSIC9JbmZvIDM5IDAgUiA+PgpzdGFydHhyZWYKODYzMwolJUVPRgo=", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["test_model = PixelCNN(c_in=1, c_hidden=64)\n", "inp = torch.zeros(1, 1, 28, 28)\n", "inp.requires_grad_()\n", "out = test_model(inp)\n", "show_center_recep_field(inp, out.squeeze(dim=2))\n", "del inp, out, test_model"]}, {"cell_type": "markdown", "id": "ebb748d6", "metadata": {"papermill": {"duration": 0.014545, "end_time": "2023-03-14T16:13:41.594412", "exception": false, "start_time": "2023-03-14T16:13:41.579867", "status": "completed"}, "tags": []}, "source": ["The visualization shows that for predicting any pixel, we can take almost half of the image into account.\n", "However, keep in mind that this is the \"theoretical\" receptive field and not necessarily\n", "the [effective receptive field](https://arxiv.org/pdf/1701.04128.pdf), which is usually much smaller.\n", "For a stronger model, we should therefore try to increase the receptive\n", "field even further. Especially, for the pixel on the bottom right, the\n", "very last pixel, we would be allowed to take into account the whole\n", "image. However, our current receptive field only spans across 1/4 of the\n", "image. An encoder-decoder architecture can help with this, but it also\n", "shows that we require a much deeper, more complex network in\n", "autoregressive models than in VAEs or energy-based models."]}, {"cell_type": "markdown", "id": "2e74a7c4", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.014282, "end_time": "2023-03-14T16:13:41.623307", "exception": false, "start_time": "2023-03-14T16:13:41.609025", "status": "completed"}, "tags": []}, "source": ["### Training loop\n", "\n", "To train the model, we again can rely on PyTorch Lightning and write a\n", "function below for loading the pretrained model if it exists. To reduce\n", "the computational cost, we have saved the validation and test score in\n", "the checkpoint already:"]}, {"cell_type": "code", "execution_count": 18, "id": "c1ac3f4a", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:41.653251Z", "iopub.status.busy": "2023-03-14T16:13:41.652916Z", "iopub.status.idle": "2023-03-14T16:13:41.664414Z", "shell.execute_reply": "2023-03-14T16:13:41.663876Z"}, "papermill": {"duration": 0.028946, "end_time": "2023-03-14T16:13:41.666554", "exception": false, "start_time": "2023-03-14T16:13:41.637608", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def train_model(**kwargs):\n", " # Create a PyTorch Lightning trainer with the generation callback\n", " trainer = L.Trainer(\n", " default_root_dir=os.path.join(CHECKPOINT_PATH, \"PixelCNN\"),\n", " accelerator=\"auto\",\n", " devices=1,\n", " max_epochs=150,\n", " callbacks=[\n", " ModelCheckpoint(save_weights_only=True, mode=\"min\", monitor=\"val_bpd\"),\n", " LearningRateMonitor(\"epoch\"),\n", " ],\n", " )\n", " result = None\n", " # Check whether pretrained model exists. If yes, load it and skip training\n", " pretrained_filename = os.path.join(CHECKPOINT_PATH, \"PixelCNN.ckpt\")\n", " if os.path.isfile(pretrained_filename):\n", " print(\"Found pretrained model, loading...\")\n", " model = PixelCNN.load_from_checkpoint(pretrained_filename)\n", " ckpt = torch.load(pretrained_filename, map_location=device)\n", " result = ckpt.get(\"result\", None)\n", " else:\n", " model = PixelCNN(**kwargs)\n", " trainer.fit(model, train_loader, val_loader)\n", " model = model.to(device)\n", "\n", " if result is None:\n", " # Test best model on validation and test set\n", " val_result = trainer.test(model, dataloaders=val_loader, verbose=False)\n", " test_result = trainer.test(model, dataloaders=test_loader, verbose=False)\n", " result = {\"test\": test_result, \"val\": val_result}\n", " return model, result"]}, {"cell_type": "markdown", "id": "3596cc75", "metadata": {"papermill": {"duration": 0.014335, "end_time": "2023-03-14T16:13:41.698274", "exception": false, "start_time": "2023-03-14T16:13:41.683939", "status": "completed"}, "tags": []}, "source": ["Training the model is time consuming and we recommend using the provided pre-trained model for going through this notebook.\n", "However, feel free to play around with the hyperparameter like number of layers etc.\n", "if you want to get a feeling for those.\n", "\n", "When calling the training function with a pre-trained model, we automatically load it and print its test performance:"]}, {"cell_type": "code", "execution_count": 19, "id": "1d6f5478", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:41.728262Z", "iopub.status.busy": "2023-03-14T16:13:41.728099Z", "iopub.status.idle": "2023-03-14T16:13:43.271303Z", "shell.execute_reply": "2023-03-14T16:13:43.270004Z"}, "papermill": {"duration": 1.56105, "end_time": "2023-03-14T16:13:43.273803", "exception": false, "start_time": "2023-03-14T16:13:41.712753", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True (cuda), used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Lightning automatically upgraded your loaded checkpoint from v0.9.0 to v2.0.0rc0. To apply the upgrade to your files permanently, run `python -m lightning.pytorch.utilities.upgrade_checkpoint --file saved_models/tutorial12/PixelCNN.ckpt`\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model, loading...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Test bits per dimension: 0.808bpd\n"]}], "source": ["model, result = train_model(c_in=1, c_hidden=64)\n", "test_res = result[\"test\"][0]\n", "print(\n", " \"Test bits per dimension: %4.3fbpd\" % (test_res[\"test_loss\"] if \"test_loss\" in test_res else test_res[\"test_bpd\"])\n", ")"]}, {"cell_type": "markdown", "id": "4c7ae506", "metadata": {"papermill": {"duration": 0.014712, "end_time": "2023-03-14T16:13:43.307215", "exception": false, "start_time": "2023-03-14T16:13:43.292503", "status": "completed"}, "tags": []}, "source": ["With a test performance of 0.809bpd, the PixelCNN significantly outperforms the normalizing flows we have seen in Tutorial 11.\n", "Considering image modeling as an autoregressive problem simplifies the learning process as predicting\n", "one pixel given the ground truth of all others is much easier than predicting all pixels at once.\n", "In addition, PixelCNN can explicitly predict the pixel values by a discrete softmax while\n", "Normalizing Flows have to learn transformations in continuous latent space.\n", "These two aspects allow the PixelCNN to achieve a notably better performance.\n", "\n", "To fully compare the models, let's also measure the number of parameters of the PixelCNN:"]}, {"cell_type": "code", "execution_count": 20, "id": "5536439d", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:43.338744Z", "iopub.status.busy": "2023-03-14T16:13:43.338328Z", "iopub.status.idle": "2023-03-14T16:13:43.346474Z", "shell.execute_reply": "2023-03-14T16:13:43.345329Z"}, "papermill": {"duration": 0.02678, "end_time": "2023-03-14T16:13:43.348899", "exception": false, "start_time": "2023-03-14T16:13:43.322119", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Number of parameters: 852,160\n"]}], "source": ["num_params = sum(np.prod(param.shape) for param in model.parameters())\n", "print(f\"Number of parameters: {num_params:,}\")"]}, {"cell_type": "markdown", "id": "5ddfc6fe", "metadata": {"papermill": {"duration": 0.01493, "end_time": "2023-03-14T16:13:43.384448", "exception": false, "start_time": "2023-03-14T16:13:43.369518", "status": "completed"}, "tags": []}, "source": ["Compared to the multi-scale normalizing flows, the PixelCNN has considerably less parameters.\n", "Of course, the number of parameters depend on our hyperparameter choices.\n", "Nevertheless, in general, it can be said that autoregressive models\n", "require considerably less parameters than normalizing flows to reach\n", "good performance, based on the reasons stated above. Still,\n", "autoregressive models are much slower in sampling than normalizing\n", "flows, which limits their possible applications."]}, {"cell_type": "markdown", "id": "dba52ce0", "metadata": {"papermill": {"duration": 0.014745, "end_time": "2023-03-14T16:13:43.414182", "exception": false, "start_time": "2023-03-14T16:13:43.399437", "status": "completed"}, "tags": []}, "source": ["## Sampling\n", "\n", "One way of qualitatively analysing generative models is by looking at the actual samples.\n", "Let's therefore use our sampling function to generate a few digits:"]}, {"cell_type": "code", "execution_count": 21, "id": "797b6092", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:43.445993Z", "iopub.status.busy": "2023-03-14T16:13:43.445629Z", "iopub.status.idle": "2023-03-14T16:13:48.695374Z", "shell.execute_reply": "2023-03-14T16:13:48.694592Z"}, "papermill": {"duration": 5.269017, "end_time": "2023-03-14T16:13:48.697995", "exception": false, "start_time": "2023-03-14T16:13:43.428978", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 1\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "25db831959f548fd9381c36990fab9c0", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/28 [00:00\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["L.seed_everything(1)\n", "samples = model.sample(img_shape=(16, 1, 28, 28))\n", "show_imgs(samples.cpu())"]}, {"cell_type": "markdown", "id": "7d979873", "metadata": {"papermill": {"duration": 0.016051, "end_time": "2023-03-14T16:13:48.737269", "exception": false, "start_time": "2023-03-14T16:13:48.721218", "status": "completed"}, "tags": []}, "source": ["Most of the samples can be identified as digits, and overall we achieve a better quality than we had in normalizing flows.\n", "This goes along with the lower likelihood we achieved with autoregressive models.\n", "Nevertheless, we also see that there is still place for improvement\n", "as a considerable amount of samples cannot be identified (for example the first row).\n", "Deeper autoregressive models are expected to achieve better quality,\n", "as they can take more context into account for generating the pixels.\n", "\n", "Note that on Google Colab, you might see different results, specifically with a white line at the top.\n", "After some debugging, it seemed that the difference occurs inside the dilated convolution,\n", "as it gives different results for different batch sizes.\n", "However, it is hard to debug this further as it might be a bug of the installed PyTorch version on Google Colab.\n", "\n", "The trained model itself is not restricted to any specific image size.\n", "However, what happens if we actually sample a larger image than we had\n", "seen in our training dataset? Let's try below to sample images of size\n", "$64\\times64$ instead of $28\\times28$:"]}, {"cell_type": "code", "execution_count": 22, "id": "3ac6c40a", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:13:48.771810Z", "iopub.status.busy": "2023-03-14T16:13:48.771428Z", "iopub.status.idle": "2023-03-14T16:14:07.470230Z", "shell.execute_reply": "2023-03-14T16:14:07.469527Z"}, "papermill": {"duration": 18.718523, "end_time": "2023-03-14T16:14:07.472228", "exception": false, "start_time": "2023-03-14T16:13:48.753705", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 1\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "2dcb52e8c9194f45a2ba294e584342c0", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/64 [00:00\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["L.seed_everything(1)\n", "samples = model.sample(img_shape=(8, 1, 64, 64))\n", "show_imgs(samples.cpu())"]}, {"cell_type": "markdown", "id": "720c6e17", "metadata": {"papermill": {"duration": 0.016713, "end_time": "2023-03-14T16:14:07.510386", "exception": false, "start_time": "2023-03-14T16:14:07.493673", "status": "completed"}, "tags": []}, "source": ["The larger images show that changing the size of the image during testing confuses the model\n", "and generates abstract figures (you can sometimes spot a digit in the upper left corner).\n", "In addition, sampling for images of 64x64 pixels take more than a minute on a GPU.\n", "Clearly, autoregressive models cannot be scaled to large images without changing the sampling procedure such as with [forecasting](https://arxiv.org/abs/2002.09928).\n", "Our implementation is also not the most efficient as many computations can be stored and reused throughout the sampling process.\n", "Nevertheless, the sampling procedure stays sequential which is\n", "inherently slower than parallel generation like done in normalizing\n", "flows."]}, {"cell_type": "markdown", "id": "4997f1ba", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.016637, "end_time": "2023-03-14T16:14:07.543845", "exception": false, "start_time": "2023-03-14T16:14:07.527208", "status": "completed"}, "tags": []}, "source": ["### Autocompletion\n", "\n", "One common application done with autoregressive models is\n", "auto-completing an image. As autoregressive models predict pixels one by\n", "one, we can set the first $N$ pixels to predefined values and check how\n", "the model completes the image. For implementing this, we just need to\n", "skip the iterations in the sampling loop that already have a value\n", "unequals -1. See above in our PyTorch Lightning module for the specific\n", "implementation. In the cell below, we randomly take three images from\n", "the training set, mask about the lower half of the image, and let the\n", "model autocomplete it. To see the diversity of samples, we do this 12\n", "times for each image:"]}, {"cell_type": "code", "execution_count": 23, "id": "8390b92c", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:14:07.578284Z", "iopub.status.busy": "2023-03-14T16:14:07.578111Z", "iopub.status.idle": "2023-03-14T16:14:14.028150Z", "shell.execute_reply": "2023-03-14T16:14:14.027370Z"}, "papermill": {"duration": 6.470137, "end_time": "2023-03-14T16:14:14.030600", "exception": false, "start_time": "2023-03-14T16:14:07.560463", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Original image and input image to sampling:\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMTc1LjUyMjUgOTcuNTYgXSAvQ29udGVudHMgOSAwIFIgL0Fubm90cyAxMCAwIFIgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0xlbmd0aCAxMiAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicVY4xDsIwDEV3n+KfIImDkpQRqBQxFgYOEIVCRItKJXp93A6tGJ7kZ9nf1nX+PlO+xCNOV9KbpZEYRWhhUIQJjCi0ZMQ64uCUs9aJvDbZB+W8NMxaPYjuNCAou8CeFc+T1U6xxyfjhh76IMGjpBdhksiI/1+GZdFWmI/LqrNrYuqgz4z6jYYa+gGxYy7BCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQzCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDEgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjQgL0hlaWdodCAxMTYKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgNzAgKP////39/fv7+/r6+vX19fLy8vDw8O/v7+3t7ezs7Ovr6+jo6OTk5OPj49/f39jY2NHR0dDQ0M/Pz8fHx8PDw8LCwsHBwb+/v7q6urm5ubS0tK6urqysrKWlpaCgoJWVlZOTk4+Pj4uLi4qKioiIiICAgH19fXJycmNjY2BgYF9fX1ZWVlFRUVBQUE9PT0xMTEdHR0RERDs7OzU1NTIyMjAwMCwsLCoqKiMjIyIiIh0dHRsbGxoaGhkZGRQUFBMTExISEg4ODgwMDFxuXG5cbgYGBgQEBAAAACldCi9CaXRzUGVyQ29tcG9uZW50IDggL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNCA+PiAvTGVuZ3RoIDE0IDAgUiA+PgpzdHJlYW0KeJzt3GtT2kAYxfFEQaUipRQv4IVK0baKVqC0RUEBhe//kXp2NnZUkrjJPKuT5fxfAsPh9yLcwTtxPO+9b4DtCMx6BGa9pQHO36KTdxgj0MImgZJjBFrYJFByjEALmwRKjhFoYZNAyTECLWwSKDlGoIVNAiXHCLSwSaDkmBHwB/I87wuKvMwDms1mJpuvjnl6LPoyestojMBgk8DsAieTySfk+/45Wjz/F2q1WhuoUqlcoNc2o8cmeszXY5FbLb1VidsicHmAo9HI152h4GbcoV6vd4w2kf+/ryg9cPRs7JGst3oLW37cFoEEOgPE0R5c3wfUQKVSyQ+tWq3eo/TA82djwVb4WFWPEUjgHNeyj/L5/IuJXC63h9QTuOCU2GPCBCgZgY8RmHVgULfb/YY+6k5Rp9NRZxwh4NbQcDg02SRQJAJf5DwwtCukXggCeIkMNwkUiUCTPiP1AFgoFK6R4SaBIhFo0tIAm81mgk0CRSLQJLeB7XZ7HW2j6XSaYJNAkQiMrY+AU8ffAUq2SaBIBMbmPPA7Ct4r/ImSbRIoEoHRjcdj9RUA4HZQ7OdJoZsEikRgdM4DU93BPNkkUCQCo3MeWK/Xla5YLP5FyTcJFInA8G5RuVxWwFqtlm6TQJEIDM95oPrWNHBbqN/vp9skUCQCw3MeeIgA3EWpNwkUicDF/qBVBOAKajQaNyj5JoEiEbiY88DfKHivQv2cYTAYpNskUCQC7W0SKDlGoIVNAiXHCLSwSaDk2NL8DbWzEZj1CMx6zgP/Afg5osMKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago1ODcKZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjcuMSkgL0NyZWF0aW9uRGF0ZSAoRDoyMDIzMDMxNDE2MTQwN1opCj4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDE4MTkgMDAwMDAgbiAKMDAwMDAwMDU5NSAwMDAwMCBuIAowMDAwMDAwNjE2IDAwMDAwIG4gCjAwMDAwMDA2NzYgMDAwMDAgbiAKMDAwMDAwMDY5NyAwMDAwMCBuIAowMDAwMDAwNzE4IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDMzNyAwMDAwMCBuIAowMDAwMDAwNTc1IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU1NSAwMDAwMCBuIAowMDAwMDAwNzUwIDAwMDAwIG4gCjAwMDAwMDE3OTkgMDAwMDAgbiAKMDAwMDAwMTg3OSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDE2IC9Sb290IDEgMCBSIC9JbmZvIDE1IDAgUiA+PgpzdGFydHhyZWYKMjAzMAolJUVPRgo=", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["Global seed set to 1\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "f5bf61676010434faf1e705c33370955", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/28 [00:00\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Original image and input image to sampling:\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMTc1LjUyMjUgOTcuNTYgXSAvQ29udGVudHMgOSAwIFIgL0Fubm90cyAxMCAwIFIgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0xlbmd0aCAxMiAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicVY4xDsIwDEV3n+KfIImDkpQRqBQxFgYOEIVCRItKJXp93A6tGJ7kZ9nf1nX+PlO+xCNOV9KbpZEYRWhhUIQJjCi0ZMQ64uCUs9aJvDbZB+W8NMxaPYjuNCAou8CeFc+T1U6xxyfjhh76IMGjpBdhksiI/1+GZdFWmI/LqrNrYuqgz4z6jYYa+gGxYy7BCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQzCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDEgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjQgL0hlaWdodCAxMTYKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgNjggKP////7+/vz8/Pv7+/n5+ff39/Ly8vHx8erq6unp6ejo6Obm5uXl5d3d3dnZ2dfX19bW1tXV1czMzMbGxsHBwb+/v729vby8vLu7u7q6uri4uLW1tbS0tLKysq2traysrKGhoZiYmJWVlZGRkYCAgHV1dXNzc3FxcWpqamdnZ2JiYlpaWldXV1NTU1FRUU9PT0xMTEtLS0JCQkBAQDs7Ozo6OjAwMCAgIBsbGxgYGBcXFxUVFRMTEw4ODlxyXHJcclxuXG5cbggICAYGBgMDAwICAgAAACldCi9CaXRzUGVyQ29tcG9uZW50IDggL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNCA+PiAvTGVuZ3RoIDE0IDAgUiA+PgpzdHJlYW0KeJztmtmS0kAYRokbuC8sQUVkdEQEFBURZJHR4f2fya8rsQZCSC90N5XwnctOp07OTfJXktJlwSmd+gJcw8C8w8C8czaBGx9cnkDGQAdOrzIGOnB6lTHQgdOrzDDwF6jVat+BvtNAVotkOmcxUOZk4EGngcxv4Hq9fgmCIHgL9J06snUkCyKZzlUyMFPKwCynjsx74AKEYRhEfAL6TlVZ7NqSabgYmC0tbuAbEAsrlcofoO9Ule26KnouBipJGZjqVJWdJPApiKX9fl9HuNEO3HUZyhiYKTVzqsoYqCIzDLwHRqORmVNV9j8wchnKGJguLXygGBE1hRvjQCMXA7OlDMxyqsq8B34EtwGc74GxU1UWuwIjFwPTnQUOXCwWL0A8Ook328ZOucwGDEzAQHWnXGYD3cDhcBjcEAd+AZ8jvgFFp1xmAwYmYGDeA6vV6lbga9But2+BeOUuwErmBMBAmzAwwW7gQR6BnyDLKZfZgIEJGFiswAfg4R7lcrkEHoPUWZWBNmFggnMKrNfrv8H+nvF4fB+IPdPp9KBTLrMBAxOcU2C3203fs1wunwEROZvNDjrlMhswMAEDN8UJbDab+zcZsRKGoYj7AbKccpkNGJig8IGdTmdrVPsK5vO5OHANJpPJK4ADz4HMKZfZgIEJGJj3wNVqtfX/pqDRaLwDF+Bm8QOQOeUyGzAwAQPzHgiuwBMQpCFe3fd6vb9A5lSSHQ0D9yl8oEA8AFut1lbZHTAYDMS3UEWnsuwoGJgOAxWcyrKjMAy04fQqY6ADp1cZAx04vcoY6MDpVcZAB06vMgY6cHqVMdCB06uMgQ6cXmUMdOD0KmOgA6dXGQMdOL3KziawsDAw7zAw7zAw7/wD8IOXcgplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjY1OAplbmRvYmoKMiAwIG9iago8PCAvVHlwZSAvUGFnZXMgL0tpZHMgWyAxMSAwIFIgXSAvQ291bnQgMSA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNy4xLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNy4xKSAvQ3JlYXRpb25EYXRlIChEOjIwMjMwMzE0MTYxNDA5WikKPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwMTg4NyAwMDAwMCBuIAowMDAwMDAwNTk1IDAwMDAwIG4gCjAwMDAwMDA2MTYgMDAwMDAgbiAKMDAwMDAwMDY3NiAwMDAwMCBuIAowMDAwMDAwNjk3IDAwMDAwIG4gCjAwMDAwMDA3MTggMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzM3IDAwMDAwIG4gCjAwMDAwMDA1NzUgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNTU1IDAwMDAwIG4gCjAwMDAwMDA3NTAgMDAwMDAgbiAKMDAwMDAwMTg2NyAwMDAwMCBuIAowMDAwMDAxOTQ3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgMTYgL1Jvb3QgMSAwIFIgL0luZm8gMTUgMCBSID4+CnN0YXJ0eHJlZgoyMDk4CiUlRU9GCg==", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["Global seed set to 1\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "418b15f4e98c47bc965e3234817bc316", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/28 [00:00\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stdout", "output_type": "stream", "text": ["Original image and input image to sampling:\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgMTc1LjUyMjUgOTcuNTYgXSAvQ29udGVudHMgOSAwIFIgL0Fubm90cyAxMCAwIFIgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0xlbmd0aCAxMiAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicVY4xDsIwDEV3n+KfIImDkpQRqBQxFgYOEIVCRItKJXp93A6tGJ7kZ9nf1nX+PlO+xCNOV9KbpZEYRWhhUIQJjCi0ZMQ64uCUs9aJvDbZB+W8NMxaPYjuNCAou8CeFc+T1U6xxyfjhh76IMGjpBdhksiI/1+GZdFWmI/LqrNrYuqgz4z6jYYa+gGxYy7BCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQzCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDEgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0ltYWdlIC9XaWR0aCAyMjQgL0hlaWdodCAxMTYKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgNzggKP////7+/v39/fz8/Pn5+ff39/T09PPz8+/v7+3t7ezs7Ovr6+np6ejo6Obm5uHh4d7e3tvb29ra2tPT087OzsnJyb+/v76+vru7u7q6uq6urqmpqaioqKenp6SkpJycnJiYmJaWlpOTk5GRkY+Pj42NjYmJiYWFhYCAgHd3d3Z2dnFxcWpqamRkZGBgYF5eXl1dXVlZWVdXV1VVVVRUVFBQUENDQ0BAQD8/Pz4+Pjg4ODExMSsrKyYmJiMjIyAgIB8fHx0dHRsbGxYWFhISEhAQEA8PDw4ODgsLCwcHBwYGBgUFBQQEBAICAgAAACldCi9CaXRzUGVyQ29tcG9uZW50IDggL0ZpbHRlciAvRmxhdGVEZWNvZGUKL0RlY29kZVBhcm1zIDw8IC9QcmVkaWN0b3IgMTAgL0NvbG9ycyAxIC9Db2x1bW5zIDIyNCA+PiAvTGVuZ3RoIDE0IDAgUiA+PgpzdHJlYW0KeJztnGtz0kAYRqGIIMpFwCviDZQqUm9FQEBEkQAq///n+OxkM0CFZjO8a2V5zqc27OTkzJTdbjIQO3Wc2FVfgG0YeOgw8NA5msDlv+D0CmQMtOBkoKSMgRacDJSUMdCCk4GSMgZacDJQUsZAC04GSsoYaMHJQEkZAy04GSgpixr4DhSLxRMQj8cfgjPgeV4Ep6lMu4q+K75yRZExcLuUgZc5/8/A6XT6GCRBwgde/VOiVCp9AobOUNnUl226TlaukoGLgUcXiL9+dfpboNVqvQZrgeA66PV6IoFnvky7Wr7rgqxnKGMgA10I/AawHqlTfwY48gWc+zwA2lsoFH6A/QJ9V3HlWu5yFS5zMfCoAt8AvAlegl9g80V1pNls3gQY8wzsFygFA9dg4EEHDofDDMjlcmqu2T5mNpvdBtjTPAJhTgaKwMAA5wPr9brehu0e0+l01BhMMu9BmJOBIjAwoFKpqIvHUrj99d+gWq2qMeVy+ScIczJQBAYGOB2o7t/l83l18TtP9AroPVq73TZxMlAEBiqcD/wO9MX/vYBPJpMX4BrQYwydDBSBgQFYB9V95bVt0Hg8/gBisdjJiufA0MlAERgY4Hxgo9FQE0gqlbqnyWaz6kg6nVY3KO4A/MbAJQNlMQ5cLBZPQSaT0atdIplM3gf6CWTW730LDJ0MFIGBAc4HajzP049Zz7vdrj6o/le9AVD/FRg6GSgCAy/gfOBWPgI17dRqtQhOBorAQBPuArVXwhszgpOBIjDQBLcDR6ORWgPVJMPADScDRWBgKIPBINghMnDDyUARGBgKA3c5GSjC/oHz+fwJUJ/p6ff7EZwMFIGB9pwMlJQx0IKTgZIyBlpwMlBSdjRfQ+0sDDx0GHjoOB/4B/2r8sQKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago2NjgKZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjcuMSkgL0NyZWF0aW9uRGF0ZSAoRDoyMDIzMDMxNDE2MTQxMVopCj4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDE5MjEgMDAwMDAgbiAKMDAwMDAwMDU5NSAwMDAwMCBuIAowMDAwMDAwNjE2IDAwMDAwIG4gCjAwMDAwMDA2NzYgMDAwMDAgbiAKMDAwMDAwMDY5NyAwMDAwMCBuIAowMDAwMDAwNzE4IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDMzNyAwMDAwMCBuIAowMDAwMDAwNTc1IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU1NSAwMDAwMCBuIAowMDAwMDAwNzUwIDAwMDAwIG4gCjAwMDAwMDE5MDEgMDAwMDAgbiAKMDAwMDAwMTk4MSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDE2IC9Sb290IDEgMCBSIC9JbmZvIDE1IDAgUiA+PgpzdGFydHhyZWYKMjEzMgolJUVPRgo=", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}, {"name": "stderr", "output_type": "stream", "text": ["Global seed set to 1\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "dc61c791bbc44c5a8d0a9d011354ebad", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/28 [00:00\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["def autocomplete_image(img):\n", " # Remove lower half of the image\n", " img_init = img.clone()\n", " img_init[:, 10:, :] = -1\n", " print(\"Original image and input image to sampling:\")\n", " show_imgs([img, img_init])\n", " # Generate 12 example completions\n", " img_init = img_init.unsqueeze(dim=0).expand(12, -1, -1, -1).to(device)\n", " L.seed_everything(1)\n", " img_generated = model.sample(img_init.shape, img_init)\n", " print(\"Autocompletion samples:\")\n", " show_imgs(img_generated)\n", "\n", "\n", "for i in range(1, 4):\n", " img = train_set[i][0]\n", " autocomplete_image(img)"]}, {"cell_type": "markdown", "id": "3a429ef2", "metadata": {"papermill": {"duration": 0.018749, "end_time": "2023-03-14T16:14:14.073447", "exception": false, "start_time": "2023-03-14T16:14:14.054698", "status": "completed"}, "tags": []}, "source": ["For the first two digits (7 and 6), we see that the 12 samples all\n", "result in a shape which resemble the original digit. Nevertheless, there\n", "are some style difference in writing the 7, and some deformed sixes in\n", "the samples. When autocompleting the 9 below, we see that the model can\n", "fit multiple digits to it. We obtain diverse samples from 0, 3, 8 and 9.\n", "This shows that despite having no latent space, we can still obtain\n", "diverse samples from an autoregressive model."]}, {"cell_type": "markdown", "id": "29ea7146", "metadata": {"papermill": {"duration": 0.018612, "end_time": "2023-03-14T16:14:14.110771", "exception": false, "start_time": "2023-03-14T16:14:14.092159", "status": "completed"}, "tags": []}, "source": ["### Visualization of the predictive distribution (softmax)\n", "\n", "Autoregressive models use a softmax over 256 values to predict the next pixel.\n", "This gives the model a large flexibility as the probabilities for each pixel value can be learned independently if necessary.\n", "However, the values are actually not independent because the values 32 and 33 are much closer than 32 and 255.\n", "In the following, we visualize the softmax distribution that the model predicts to gain insights how it has learned the relationships of close-by pixels.\n", "\n", "To do this, we first run the model on a batch of images and store the output softmax distributions:"]}, {"cell_type": "code", "execution_count": 24, "id": "d882d822", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:14:14.149965Z", "iopub.status.busy": "2023-03-14T16:14:14.149583Z", "iopub.status.idle": "2023-03-14T16:14:14.241067Z", "shell.execute_reply": "2023-03-14T16:14:14.240320Z"}, "papermill": {"duration": 0.11407, "end_time": "2023-03-14T16:14:14.243446", "exception": false, "start_time": "2023-03-14T16:14:14.129376", "status": "completed"}, "tags": []}, "outputs": [], "source": ["det_loader = data.DataLoader(train_set, batch_size=128, shuffle=False, drop_last=False)\n", "imgs, _ = next(iter(det_loader))\n", "imgs = imgs.to(device)\n", "with torch.no_grad():\n", " out = model(imgs)\n", " out = F.softmax(out, dim=1)\n", " mean_out = out.mean(dim=[0, 2, 3, 4]).cpu().numpy()\n", " out = out.cpu().numpy()"]}, {"cell_type": "markdown", "id": "361e2dd4", "metadata": {"papermill": {"duration": 0.018535, "end_time": "2023-03-14T16:14:14.285455", "exception": false, "start_time": "2023-03-14T16:14:14.266920", "status": "completed"}, "tags": []}, "source": ["Before diving into the model, let's visualize the distribution of the pixel values in the whole dataset:"]}, {"cell_type": "code", "execution_count": 25, "id": "21b1f8a0", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:14:14.324560Z", "iopub.status.busy": "2023-03-14T16:14:14.324198Z", "iopub.status.idle": "2023-03-14T16:14:15.680768Z", "shell.execute_reply": "2023-03-14T16:14:15.680126Z"}, "papermill": {"duration": 1.380679, "end_time": "2023-03-14T16:14:15.684889", "exception": false, "start_time": "2023-03-14T16:14:14.304210", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNDA3LjAyIDMwMC42NTI2MjUgXSAvQ29udGVudHMgOSAwIFIgL0Fubm90cyAxMCAwIFIgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0xlbmd0aCAxMiAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnic3V1Nj1xHcrz3r+jj7kHF+v44SpZNQDdJhH0wfFjQ3LUEUoZW2PXfd2bV9MssdpRBL5c2qgQQmsmYjunI6fe6oiLf61ffvvvrT2/f/fD6m/s//Hh7Jd+9/e3m7j/Tvz/d7f1n+vdfd3d/Tf/+dLP03YdbtMVYT1++f3wZrDU5+ewT1ez87X/cbn+8vfqaHv7b3ZrmSo7FplqfvonNupZtqfc/829+Pf3A7X/66dstelPuvpgQLf/WD7fQmqleld5fpRZMfDy58Thd6U/31/tHhCEV4+ixORtH///zu/u/3H+5v/rasyp3/47+/Uz/RqeklzfqZaqmBV/8/PxUVf/224+37++/PoitcYn+Cg/u/u3rl+rtV/oL2ftXlqCUTOxknbCZWB2xvf1w++bN7dU/ubtz9zd/7H+8N/9++9f77+zv7/92f/Pd7R/f3L7vv+/vp9ZFa6KL9Nef5Ory5+t1IbywEX/8BME5fjnF3jtTW070StSKdfnzFXvnTGC2kqJ1n6DY+frlJAcbTCg12TAfc6r8d5DcvKnM5nMJ9VMkN/8FJddoarbRz39lXf58yaFEE5itVFvSJ0j2KX85yZ2pVlNsq02dU6X0t+oshlhIarLJtRJSfpFIwozNlaSTPNH7O/f7+5ufb9m0VlNILmU/OsAnMgJcMIn+y5kl36PxzrYgJIVkMEvoP0yN+pV/jS3lzo18+dJF46wPkR9HDKX1p/H2w50Yvvr23c9/+Oe//PiHX3776sNPv/zlt/u3/3n//sv02yV6QdXqUpCGq9rndNwla5KvpZQavf8/abnfoeV8prakhJTLukFqn9Ny7zIdK8WG1CItW6TlrvQj+0v03H16z0NtvYP/Dz2v1J4Yc26q51L72xcG3PRKbxw1+RYttevzXuek0oZqa6JOr1tuHy3/X/UqDGW+VRuad4n+UCaOBa4PiRa8+bHEjX1FbFKXvvj5+/TzP7y+feJP8upZFqNf5VQbvfQdvUQq/WkysfSHfYS8V4ivtDahl2V/41PrWl2mVfU3e/am67TNtufeOBNLX4A+9eZC6CvCSnaPjnUmXd24M6SSSs09d4ak0FtMzM+duZBSqUfWJ/fSr8Gkqxt3hp5woGMgPHcm0td9ufjUmQspfJ7PpbSXfg0mXd24M6TSpkin5afOJFoC1Pzxi+m9AHRazznxq+S94tHVjfvCIkMOHx8x1JdsUszl6Sh7rxCyDclFcpuPdnUmXd24M6zSVffxEUOdKaa62J6OsvcKKc3kkoP0qzPp6sadYZXPXWnG1uCejrD3CimZVYb+zjxYdGXjjpBCWlg/rVY+3IqltYgL4eMVznuF0HuPpTOsLS+9Gky6um9nWGWkk8LHaxXqDK1FXCMf/tyZC6nBhJwdmcb3iklXN+4Mq4z0VvLcGbYRpdjnM68gvItRqQnp0a/OpKsbd4ZVQldA7yx0GkGuQBB9jhWmM868XSX0BCWZ6KEnEESv5ITpjPUdq3TQE9A7TG7QEwiiz7XCdMgZmFRW6AlKMS1DTyAIdyN699KtwSO1jbtCChP0A3Sm8B75gQugs0rKKbvy0qnBo6sb94VFQj9Q6YEN+gFBqAd0xDSXH+0afkBV9+1MVwn9QHWmZOgHBCHfGPmr+OhXZ9LVjTvDKp+7EowN0A8IQu9CrdKrpDx6NVU27ggp9NAP8ArNQj8gCHmA5nOr7aVXg0lXN+5MpJ+FfqAmOn9CPyCIPpMI0yHnF1YJ/UDNpgboBwTR61thOmPV21VCP1CrcTglECQ18tmV5D361Zl0dePOsEroByrpwxmBINkaso0l5Ee/Rkagqht3hlTijKDRz+KMQJDMj661lZd+DSZd3bczrBJnBPR1wxmBIMVywhR7TiJMurpxZ0glzgha4OAceIILyJ53XmJf/QuPrm7cFxYJPUGj9SvOCATJyZTisvWPdnUmXd24M6wSeoJG+nBGIEimtV3LdFw9+jU8gapu3BlW+dyVTKt76AkEofOttSV0TzRYdGXjjpBCnBG0ajzOCATRu3TCdMbeHavEGUFrJuGMQBCdIAnTGblSVwk9AS3WTMEhgYKm3lxcZ/RmyIS2wFl2hNAXKChRuVmrejacgSrv3B3WCa2B42lpnBUoKCU6yQRb3dW0TqbLO3eHdOK4gN54TcJ5gYL0xp0iO2M/r+vEkQF56sfY+XN3LkgbJUV2hn/qOnF04Cw/eeQTBCn06BZLD68VlS7v3Bt+htAqOFv5QhLkFRRUoqHfVMPVssdA0aO6c29YJTQLzjaTcYKgIFJYXIo1Xi3rZLq8c3dY53NnnDUNpwgK0mbyhegMf9k14iCBnrHxOElQEEnjlV6Lj4a9ZAmqvHN3vHE4THAu8GQvcg4Kml43QnbKa4d1YvPgIk/3QvMgUC4mezr1xKtpnUyXd+4O68TuwWWe8YXuQaAUjKOib1fThntQ5Z27wzqxe3DFeJwsKGjuzkV2SndIJw4XnKsm4XRBQYm/osWgezRtkOnyzt0hGThgcK6ZihMGBU3eSsgO8VasE4cMji/9gymDIPTW5HwqLjxa9pIzqPLGvekysXvwnqd/oXsQaHrHErJD3rG6TuwffOAJYOgfBNIXfymyM64JGzpBZ/gKZuwfBNKjNS9EZ0zbdI04dHCenjVOHRSkhwEU2RkzAl0nDh4c6Yg4eVBQ5nvQOL4vxntNpss7d4d1Yv/gK88CQ/8gUOJLv1OI7WpaJ9PlnbvDOrF/8I1ngqF/EGjuzkV2SndYJ/YPwRm/SB8EmnZJheyQXVLWuUgfgjdxkT4INHVHyE7pDulcpA8hmLJIHwRK3riQfEmPpg0yXd65O6RzkT6ExLPCyD9cyLQGFKpD1oBdJvYPIfO0MPQPAqVsaiqxpqtnnUyXd+4O68T+IRSeGIb+QSA9zKbIzphxGzpBZ6qpi/xBID1M/EJ0xnxx17jIHyJ5gEX+INDkrITsEHdFOu0if4iOp4ahfxCoJfoqlOhemvZCpss7d4d1Yv9AYvIifxAo8503ve+DkIpMl3fuDuvE/iEGnh+G/kGgaZdUyA7ZJe06sX+IybhF/iBQrrwoHhvswqWqO/eGVC7Sh5jpodg9CJRoYZNt6rdnUWS6vHN3SOcifYg8KYHdg0DTlIWQHTJlwToX6UOsPFWM3MOF6NF0RXXGxPqQid1DsjxXDN2DQNP6WMgOWR93ndg9JMeTxdA9CDRlnkJ2SObZdYLOkJBF+iDQlOkNokPyPNa4SB9SNHaRPgg07eQI2SE7OaxzkT7Qe3FYpA8Cza+bi+yU1w7rxO4hZZ4ohu5BoGmPVMgO2SPtOrF7oCdfF+mDQNMaUMgOWQN2ndg9pEZPHrsHgabUU8gOST1Z5yJ9IIVhkT4INO2SCtkhu6Ssc5E+8C75In0QaLpuRsgOuW6mBwXYP9A6t+H04UKmPQuhOmTPosvE/oEUukX6IBDZqFJrrtKz4R9UeefusE7sH2jFEhfpg0AxGW+jLf5qWifT5Z27wzpBZ7Ipi/RBoMrrZVr8uathU2nnrpDGRfqQq7GL9EGgaR5QyA6ZByQZbZE+5MYzxdA/CFSK8c4F3/crhEyXd+4O68T+oVieKYb+QaBpgl3IDplg7zqxfyiOp4qhfxCob/7RWi9cTetkurxzd1gn9g8lGLtIHwSapiyE7JApC9a5yB/4Y6QW+YNAMZia6WVSHk0bZLq8c3dI5yJ/KMnkRf4g0OSuhOwQd8U6F/lDyTxdjPzDhUzZjFAdks10mdg/lMqzxdA/CDQ5TyE7xHl2ndg/lMazxdA/CDR5TyE7xHt2nc+dqfSTi/xBoGm3axAdstPFGhf5Q3WmLfIHgaZET8gOSfRY5yJ/qIEHJaB/EGhy5EJ2iCPvOrF/IHcdF/mDQFPeKWSH5J1dJ/YPNfFUMfQPAk3+QcgO8Q9dJ/YPla8Nwv5BoOmdXMgOeSdnnYv8oVbiwf5BoCkVFrJDUmHWucgfajNpkT8INF2TJmSHXJPGOhf5Q7M8XYz8w4VMqadQHZJ6dpnYPzTPE1rQPwgUaOkXXj42U5Hp8s7dYZ3YPzT+PCnsHwSaVjtCdshqp+sEnYkmL/IHgabcahAdklmxxkX+0Oi9eJE/CJQiSaPHt0fDBpku79wdHqXB/qEVniqG/kGgyV0J2SHuquvE/qFVniqG/kGgadZWyA6Zte06sX9ozZRF/iDQfGRdZKccWawT+gdvLXkA6B8UFGu/UYPzV9M6mS7v252uE+cP3nrjcf6gIG2jFNkZ7qrrxPmDt8EknD8oSN/8TpGdcU+8rhPnD7SK4/li4B8EKYWvTXN9KayodHnn3rBM6B+8zTxdjPyDgnSAp8jOyPWGTugfvC08XYz8g4L0lroiO2OnfegEnakm4fxBQXrj5oXojL2crhHnD942U3H+oCC9nFFkZ6xyuk6cP3jneKoY+QcFaZOpyM7wnkMn9A/esUOC/kFBuZrSWulb6YpMl3fuDuuE/sG7wHPFyD8oaDqyhOyQI6vrxP7BRdNw/qAgbRQU2SH+gXXi/ME7etY4f1CQjvAU2RnJXteJ8wfviok4f1CQHpBUZGfMTXadOH/wrvJ8MfIPFzI5T6E6xHl2mdg/uMbTxdA/CDQ5TyE7xHl2ndg/eMfTxdA/CDQfVxfZIcdV1wk6403C+YOC+geF0HqvfyL9C9NU27kvpBInEN6Tr8YJhIKm9yohO+S9inQWnEB4z5kudhACTc5TyA5xn10ndhA+81wxdBAC6ctkFNkZV88MndhBkI6MEwgF6aEBRXbGLMHQiR2EZxeAHYRA85F1kZ1yZJHORQIReD8POwiB9C0aFNkZd27oOhcJRHAmLhIIgfStExXZGXdU7DoXCUTwPGGMHMSFTLvIQnXILnKXiR1ECDxfDB2EQPMr5yI75ZXDOrGDCInni6GDEEiPDiiyMyYKhk7QmWziIoEQaH7ddKJTXjOkcZFAhGLKIoEQaHofF7JD3sdZ5yKBCI3niqF/EEhfKq3IzriCeujE/iFaniuG/kGg+XxzkR1yvuk6sX+IjieLoX8QSF/qoMjOuAJi6MT+gcTURQIh0JTPCNkh+QzrXCQQMdLRgf2DQPqDlBXZGZ+v3HUuEojId8bG/kGg+ci6yE45skjnIoGImSeMkX+4EH3zbEV1xj21h0zsH+ivXxcJhED6QnJFdsb15UMn9g+x8Xwx9A8CJSo3a1XTRgKhyjt3h3U+dybRwxYJhEDTfs4gOmQvhzUu8gf6u5dF/iDQlOkJ2SGZHunMi/yBHFJb5A8CTROTQnbIxGTXif0DreP8In8QaHLlQnaIM+86sX9IiSeLoX8QaDobC9khZ+OuE/uHlOncgf2DQJN/ELJD/APrXOQPqRq7yB8EmuZQhOyQORTWucgf+NyxyB8E4g99qC4P/yBkurxzd0jnIn8gF5Bw/nAh036gUB2yH9hlYv+QHc8XQ/8g0JTOCNkh6UzXif1DDjxfDP2DQPqjTRXZGZ94OnSCzkQTFvmDQNNRNYhOOaJI4yJ/IA+QF/mDQJOzErJD3BXrXOQP9KzbIn8QSN9QUpGdcZ/JoRP7h1x5rhj6B4Hm7lxkp3SHdWL/kBtPFkP/INDkH4TsEP/QdWL/UKwpi/xBoGhNazblcDVtXAGhyht3h3Uu8ofiSCH2DwJNu8hCdsguMutc5A8lGL/IHwSix6fQbLyaNsh0eefukM5F/lAizxcj/3Ah9OiUcint0bJBpcs794ZlYv9QEk8XQ/8gULX8icI1pqtnnUyXd+4O68T+oRSeLob+QaDJPwjZIf6h6wSdqSYs8geBpt2uQXTIThdrXOQPpZm8yB8Emq4RFrJDrhEmnWmRP/A5Y5E/CDR35yI7pDtdJ/YPld019g8CTWmwkB2SBned2D/UwHPF0D8IpG9Ir8jOuE/90In9Q40mL/IHgaYddiE7ZIeddS7yh0oKF/mDQNN5R8hOOe+QzkX+UHkyFPsHgchQxRJjco+mDTJd3rk7pHORP9TK88XIP1xI8Ybsd3P50bKXOzCp8s69YZnYP9TG08XQPwg0+XIhO8SXd53YP/RvsH8QiKRFG/y4nlHIdHnj7nSdoDPe+EX+INDkHwbRIf6BNS7yh8Z7Dtg/CDR5TiE7xHOyzkX+0CJPFUP/INA0oSNkh0zodJ3YP7TMU8XQPwg0pXpCdkiy13Vi/9AKzxVD/yDQdI8hITvkHkNdJ/YPrZq8yB8Emo+si+yUI4t0LvKH1kxd5A8CTamwkB2SCrNOnD8E64zD+YOC9EdaKbIzPumq68T5Q7Ce54uBfxCETjTF2tIPIEWlyzv3hmVC/xBs4Oli5B8UpA8gRXbGcTV0Qv8QbOTpYuQfFJQrf/JVv2eO4lLVnXvDKkFfsvE4fVCQvqXkC9EZd5nsGnH6EGwxCacPCtLWUpGd4ThZZ8TpQ7CVZ4qRe1CQ3tBSZGfscw2d0D0EZ3mmGLkHBelIXJGdkZQPndA9BOd4qhi5BwVVenwuoeSraZ1Ml3fuDuuE7iE4bxJOHxSkP5RIkZ3xWUVdJ04fggum4vRBQfoGBIrsjPsSdJ04faA3a3rW2D0IpG8oqcjOuM9k14nTh+AyTxcj93Ah00pHqA5Z7XSZ2D24wrPF0D0IpD9iUJGd8cmDQyd2D67ybDF0DwLp0VFFdsZE6dD53BlvjcPpg4IKl10eK51BpEsbd4U14vQheGciTh8UpMcFFNkZUwRdJ04fgvc8Uwz9g0D6sipFdsbVVkMn9g8+8Ewx9A8C6dBXkZ2RBQ+d2D/4xFPF0D8IpDfSFdkZ++tDJ/YPPpuE0wcF6UtkFNkZV850nTh9IBNJKznsHwTSt1lSZGfcfanrXKQPvvG9jaF/EEgPcSmyM2a7us5F+hAsTxcj/3Ah81nnojrkrNNlYv8QHM8WQ/8gUGErnnL/4FdFpss7d4d1Yv8QPM8WQ/8gUImGflMNV886l6ru3BtWCfoSjVukDwJN65xBdMgahzUu0oeQTFykDwKVQOfkFMujX4NLVXfuTTJhkT2EzPPE0DsINJ2JheyUMzHrxN4hFJ4nht5BIP0hBorsjM82GDqxdwiNJ4qhdxBI58DCdUYO3FVi5xB7Lgedg0DTDqmQHbJDyjoXyUN0piySB4H0jaEV2Rn3i+46F8lDpDedRfIg0JSWC9khaTnrXCQPMfJcMXIOF6I/dEdRnfFZPEMmdg4x8VQxdA4CTZMEQnbIJEHXiZ1DzDxVDJ2DQJWMpqWDKV5N62S6vHN3WCfoTDV2kTwINB9VneiUI4o0LpKHyGtc7B0EmjIZITskl2Gdi+QhWZ4mhu5BIH0JsCI748rgoRO7h+R4mhi6B4FK5O10H8rVtDG5pMo7d4d1YvfAMxKL5EEgfRsYRXbG3WGGTuwfUjRxkTwINPkHITvEP7DORfJA3jovkgeBCl/Saf1Y5wiZLu/cHdK5SB5SNm2RPAik97uE64z9Lla5yB1S5bli5B4uZErKheqQpLzLxO4hNZ4qhu5BoOqMt+3xwrnIdHnn7rBO7B7o718WuYNAhRbEoTifrqaN4EGVN+5O1wk6441dJA8CTVfLDKJDrpRhjYvkIQcTFsmDQNM+l5Adss9FOv0ie8iRp4mhexBIJ3nCdUaS11Vi70DHRl0kDwJV4k/B5XS1rJPp8s7dYZ3YO+TC08TQOwhEb0mp2tg/NkWR6fLO3WGd2DvQ+j8ssgeBSjOeXIItV9M6mS7v3B3SucgecjN5kT0IxLeTdLGOTE/IdHnn7jRejkDvUHgaAHsHgegQajGnEB5NG2S6vHF3WOcieyiep4qRe7iQad5NqA6Zd+sysXsg6xgX2YNAfK/JkoPq2cgeVHnn7rBO7B7ozFEW2YNA016gkB2yF9h1gs4kUofdg0CVH1vrWAYOIl3auSt8eQJ2D+wAFtmDQJX3u4KP/tGwQabLO3eHdC6yh1J5lhi6B4Gcd4boQvCPrg22qb5zf1gpdhB0Ti2L9EEgR2ub1ug35KtvnW2q79wfVoo9BK1W7CJ/EMjVwPc6qa1dfetsU33j/nSl2EVUb8IigRDI0Xt4LbQm9FffOttU37k/pHSRQdArIC0yCIFc4j6k4dCFTJev7rz62rNwf//u7ujsTprc/fX91bfv/vrT23c/vP7m/va320cd/PBSaLRWSD73/cXbj7fA9zHy0889Sh/95BPf00PVj8ljl3zf3/4brljuhQplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjYxMjkKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTcgMCBvYmoKPDwgL0xlbmd0aCAzOTUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPVJLbsVACNvnFFyg0vCbz3lSVd28+29rQ1KpKryJMcYwfcqQueVLXRJxhcm3Xq5bPKZ8LltamXmIu4uNJT623JfuIbZddC6xOB1H8gsynSpEqM2q0aH4QpaFB5BO8KELwn05/uMvgMHXsA244T0yQbAk5ilCxm5RGZoSQRFh55EVqKRQn1nC31Hu6/cyBWpvjKULYxz0CbQFQm1IxALqQABE7JRUrZCOZyQTvxXdZ2IcYOfRsgGuGVRElnvsx4ipzqiMvETEPk9N+iiWTC1Wxm5TGV/8lIzUfHQFKqk08pTy0FWz0AtYiXkS9jn8SPjn1mwhhjpu1vKJ5R8zxTISzmBLOWChl+NH4NtZdRGuHbm4znSBH5XWcEy0637I9U/+dNtazXW8cgiiQOVNQfC7Dq5GscTEMj6djSl6oiywGpq8RjPBYRAR1vfDyAMa/XK8EDSnayK0WCKbtWJEjYpscz29BNZM78U51sMTwmzvndahsjMzKiGC2rqGautAdrO+83C2nz8z6KJtCmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0xlbmd0aCAyNDkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTVFJigMwDLvnFfpAIV6TvKdDmUPn/9fKDoU5BAmvkpOWmFgLDzGEHyw9+JEhczf9G36i2btZepLJ2f+Y5yJTUfhSqC5iQl2IG8+hEfA9oWsSWbG98Tkso5lzvgcfhbgEM6EBY31JMrmo5pUhE04MdRwOWqTCuGtiw+Ja0TyN3G77RmZlJoQNj2RC3BiAiCDrArIYLJQ2NhMyWc4D7Q3JDVpg16kbUYuCK5TWCXSiVsSqzOCz5tZ2N0Mt8uCoffH6aFaXYIXRS/VYeF+FPpipmXbukkJ64U07IsweCqQyOy0rtXvE6m6B+j/LUvD9yff4Ha8PzfxcnAplbmRzdHJlYW0KZW5kb2JqCjE5IDAgb2JqCjw8IC9MZW5ndGggOTQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRY3BEcAgCAT/VEEJCgraTyaTh/b/jRAyfGDnDu6EBQu2eUYfBZUmXhVYB0pj3FCPQL3hci3J3AUPcCd/2tBUnJbTd2mRSVUp3KQSef8OZyaQqHnRY533C2P7IzwKZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvVHlwZSAvWE9iamVjdCAvU3VidHlwZSAvRm9ybSAvQkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0xlbmd0aCAzOQovRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJzjMjQwUzA2NVXI5TI3NgKzcsAsI3MjIAski2BBZDO40gAV8wp8CmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0xlbmd0aCAzMjIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicNVG7bcUwDOw1BRcwIH4lzeMgSJG3f5s72qlI07wfVV4ypVwudckqWWHypUN1iqZ8nmam/A71kOOYHtkhulPWlnsYFpaJeUodsZos93ALNr4AmhJzC/H3CPArgFHARKBu8fcPulkSQBoU/BTomquWWGICDYuFrdkV4lbdKVi4q/h2JLkHCXIxWehTDkWKKbfAfBks2ZFanOtyWQr/bn0CGmGFOOyzi0TgecADTCT+ZIBszz5b7OrqRTZ2hjjp0ICLgJvNJAFBUzirPrhh+2q75ueZKCc4OdavojG+DU7mS1LeV7nHz6BB3vgzPGd3jlAOmlAI9N0CIIfdwEaEPrXPwC4Dtkm7d2NK+ZxkKb4ENgr2qFMdyvBi7MxWb9j8x+jKZlFskJX10ekOytygE2Ieb2ShW7K2+zcPs33/AV8Ze2QKZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvTGVuZ3RoIDgzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4m9j5RlMLevw0QJW64J909XB0JmSluM8NDBp4MLIZdcYH0ljALXEdQjp3so2HVvuoEjfWmUvPvD5Se7KzihusBAkIaZgplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMzIwIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVSS24FMQjbzym4QKXwT87zqqqLvvtvaxO9FUwwYOMpL1nSS77UJdulw+RbH/clsULej+2azFLF9xazFM8tr0fPEbctCgRREz1YmS8VItTP9Og6qHBKn4FXCLcUG7yDSQCDavgHHqUzIFDnQMa7YjJSA4Ik2HNpcQiJciaJf6S8nt8nraSh9D1Zmcvfk0ul0B1NTugBxcrFSaBdSfmgmZhKRJKX632xQvSGwJI8PkcxyYDsNoltogUm5x6lJczEFDqwxwK8ZprVVehgwh6HKYxXC7OoHmzyWxOVpB2t4xnZMN7LMFNioeGwBdTmYmWC7uXjNa/CiO1Rk13DcO6WzXcI0Wj+GxbK4GMVkoBHp7ESDWk4wIjAnl44xV7zEzkOwIhjnZosDGNoJqd6jonA0J6zpWHGxx5a9fMPVOl8hwplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9MZW5ndGggMzQwIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVSOW4EMQzr/Qp9IIBu2+/ZIEiR/L8NqdkUA3F0UpQ7WlR2y4eFVLXsdPm0ldoSN+R3ZYXECcmrEu1ShkiovFYh1e+ZMq+3NWcEyFKlwuSk5HHJgj/DpacLx/m2sa/lyB2PHlgVI6FEwDLFxOgals7usGZbfpZpwI94hJwr1i3HWAVSG9047Yr3oXktsgaIvZmWigodVokWfkHxoEeNffYYVFgg0e0cSXCMiVCRgHaB2kgMOXssdlEf9DMoMRPo2htF3EGBJZKYOcW6dPTf+NCxoP7YjDe/OirpW1pZY9I+G+2Uxiwy6XpY9HTz1seDCzTvovzn1QwSNGWNksYHrdo5hqKZUVZ4t0OTDc0xxyHzDp7DGQlK+jwUv48lEx2UyN8ODaF/Xx6jjJw23gLmoj9tFQcO4rPDXrmBFUoXa5L3AalM6IHp/6/xtb7X1x8d7YDGCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0xlbmd0aCAyNTEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicLVFJcgNBCLvPK/SEZqffY5crh+T/1wjKBwYNi0B0WuKgjJ8gLFe85ZGraMPfMzGC3wWHfivXbVjkQFQgSWNQNaF28Xr0HthxmAnMk9awDGasD/yMKdzoxeExGWe312XUEOxdrz2ZQcmsXMQlExdM1WEjZw4/mTIutHM9NyDnRliXYZBuVhozEo40hUghhaqbpM4EQRKMrkaNNnIU+6Uvj3SGVY2oMexzLW1fz004a9DsWKzy5JQeXXEuJxcvrBz09TYDF1FprPJASMD9bg/1c7KT33hL584W0+N7zcnywlRgxZvXbkA21eLfvIjj+4yv5+f5/ANfYFuICmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0xlbmd0aCAyMTUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicNVE5DgMhDOz3Ff5AJIwveE+iKM3+v82M0VYewVyGtJQhmfJSk6gh5VM+epkunLrc18xqNOeWtC1zgLi2vC+tksCJZoiDwWmYuAGaPAFD19GoUUMXHtDUpVMosNwEPoq3bg/dY7WBl7Yh54kgYigZLEHNqUUTFm3PJ6Q1v16LG96X7d3IU6XGlhiBBgFWOBzX6NfwlT1PJtF0FTLUqzXLGAkTRSI8+Y6m1RPrWjTSMhLUxhGsagO8O/0wTgAAE3HLAmSfSpSz5MRvsfSzBlf6/gGfR1SWCmVuZHN0cmVhbQplbmRvYmoKMTUgMCBvYmoKPDwgL1R5cGUgL0ZvbnQgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucyAvRmlyc3RDaGFyIDAgL0xhc3RDaGFyIDI1NQovRm9udERlc2NyaXB0b3IgMTQgMCBSIC9TdWJ0eXBlIC9UeXBlMyAvTmFtZSAvQk1RUURWK0RlamFWdVNhbnMKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXQovQ2hhclByb2NzIDE2IDAgUgovRW5jb2RpbmcgPDwgL1R5cGUgL0VuY29kaW5nCi9EaWZmZXJlbmNlcyBbIDQ4IC96ZXJvIC9vbmUgL3R3byAvdGhyZWUgL2ZvdXIgL2ZpdmUgL3NpeCA1NiAvZWlnaHQgL25pbmUgXQo+PgovV2lkdGhzIDEzIDAgUiA+PgplbmRvYmoKMTQgMCBvYmoKPDwgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9Gb250TmFtZSAvQk1RUURWK0RlamFWdVNhbnMgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0FzY2VudCA5MjkgL0Rlc2NlbnQgLTIzNiAvQ2FwSGVpZ2h0IDAKL1hIZWlnaHQgMCAvSXRhbGljQW5nbGUgMCAvU3RlbVYgMCAvTWF4V2lkdGggMTM0MiA+PgplbmRvYmoKMTMgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTYgMCBvYmoKPDwgL2VpZ2h0IDE3IDAgUiAvZml2ZSAxOCAwIFIgL2ZvdXIgMTkgMCBSIC9uaW5lIDIxIDAgUiAvb25lIDIyIDAgUgovc2l4IDIzIDAgUiAvdGhyZWUgMjQgMCBSIC90d28gMjUgMCBSIC96ZXJvIDI2IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTUgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAwIC9jYSAxID4+Ci9BMiA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAxID4+Ci9BMyA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAwLjUgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0YxLURlamFWdVNhbnMtbWludXMgMjAgMCBSID4+CmVuZG9iagoyIDAgb2JqCjw8IC9UeXBlIC9QYWdlcyAvS2lkcyBbIDExIDAgUiBdIC9Db3VudCAxID4+CmVuZG9iagoyNyAwIG9iago8PCAvQ3JlYXRvciAoTWF0cGxvdGxpYiB2My43LjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My43LjEpIC9DcmVhdGlvbkRhdGUgKEQ6MjAyMzAzMTQxNjE0MTVaKQo+PgplbmRvYmoKeHJlZgowIDI4CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDExNzA4IDAwMDAwIG4gCjAwMDAwMTE0NDUgMDAwMDAgbiAKMDAwMDAxMTQ3NyAwMDAwMCBuIAowMDAwMDExNjE3IDAwMDAwIG4gCjAwMDAwMTE2MzggMDAwMDAgbiAKMDAwMDAxMTY1OSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzNDAgMDAwMDAgbiAKMDAwMDAwNjU2NSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDY1NDQgMDAwMDAgbiAKMDAwMDAxMDI1NCAwMDAwMCBuIAowMDAwMDEwMDQ3IDAwMDAwIG4gCjAwMDAwMDk2ODEgMDAwMDAgbiAKMDAwMDAxMTMwNyAwMDAwMCBuIAowMDAwMDA2NTg1IDAwMDAwIG4gCjAwMDAwMDcwNTMgMDAwMDAgbiAKMDAwMDAwNzM3NSAwMDAwMCBuIAowMDAwMDA3NTQxIDAwMDAwIG4gCjAwMDAwMDc3MTMgMDAwMDAgbiAKMDAwMDAwODEwOCAwMDAwMCBuIAowMDAwMDA4MjYzIDAwMDAwIG4gCjAwMDAwMDg2NTYgMDAwMDAgbiAKMDAwMDAwOTA2OSAwMDAwMCBuIAowMDAwMDA5MzkzIDAwMDAwIG4gCjAwMDAwMTE3NjggMDAwMDAgbiAKdHJhaWxlcgo8PCAvU2l6ZSAyOCAvUm9vdCAxIDAgUiAvSW5mbyAyNyAwIFIgPj4Kc3RhcnR4cmVmCjExOTE5CiUlRU9GCg==", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["sns.set()\n", "plot_args = {\"color\": to_rgb(\"C0\") + (0.5,), \"edgecolor\": \"C0\", \"linewidth\": 0.5, \"width\": 1.0}\n", "plt.hist(imgs.view(-1).cpu().numpy(), bins=256, density=True, **plot_args)\n", "plt.yscale(\"log\")\n", "plt.xticks([0, 64, 128, 192, 256])\n", "plt.show()\n", "plt.close()"]}, {"cell_type": "markdown", "id": "ce5603e4", "metadata": {"papermill": {"duration": 0.021009, "end_time": "2023-03-14T16:14:15.731581", "exception": false, "start_time": "2023-03-14T16:14:15.710572", "status": "completed"}, "tags": []}, "source": ["As we would expect from the seen images, the pixel value 0 (black) is the dominant value, followed by a batch of values between 250 and 255.\n", "Note that we use a log scale on the y-axis due to the big imbalance in the dataset.\n", "Interestingly, the pixel values 64, 128 and 191 also stand out which is likely due to the quantization used during the creation of the dataset.\n", "For RGB images, we would also see two peaks around 0 and 255,\n", "but the values in between would be much more frequent than in MNIST\n", "(see Figure 1 in the [PixelCNN++](https://arxiv.org/pdf/1701.05517.pdf) for a visualization on CIFAR10).\n", "\n", "Next, we can visualize the distribution our model predicts (in average):"]}, {"cell_type": "code", "execution_count": 26, "id": "2c42b03c", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:14:15.775112Z", "iopub.status.busy": "2023-03-14T16:14:15.774748Z", "iopub.status.idle": "2023-03-14T16:14:17.315574Z", "shell.execute_reply": "2023-03-14T16:14:17.314907Z"}, "papermill": {"duration": 1.567075, "end_time": "2023-03-14T16:14:17.319534", "exception": false, "start_time": "2023-03-14T16:14:15.752459", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNDA3LjAyIDMwMC42NTI2MjUgXSAvQ29udGVudHMgOSAwIFIgL0Fubm90cyAxMCAwIFIgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0xlbmd0aCAxMiAwIFIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnic1Z1LryTHcYX3/St6KS0Yk6/Ix5I07QG0o0jYC8MLgaZkERwZFCH57zsi605HVNWhQYuijeDFSH3P6c7u+G51VkZGZda7z7/56x+//ua37z97/sOXj3f229c/PPLzW/n3h2d6fiv//uuZn+/l3x8eSX778GhpUCry8LuPD2tK1Ln0wqKl86//8Xj8/vHuU3n5D89EK4/eRuI5b7+0lfLqacznn/Wd35+e8Pifnv14tELjWQbVlvRdPzzqWjSLk757SatS+/jhjtd5ZX/c75+XBisPyvLa3inL///5m+e/PP/0fPdp0ajy8zfy71v5d5Aylg9hyYu495nn6fM51b/748vHF8/vPzacKLP8FT62vX99/6Y+vpe/UHp+ksTiTmk3pj/PRW1mae3rD4/Pvnq8+6f8zPn51e/3H++rf3/86/NX6dfPf3t+9ZvHP371+GK/398v2twSjZoa8ylcL//8eHOtR2v6wz8h4N5+uYhLyTRXSyufIvbyz4+45ExVW9Of/BMizmX+ciHXVCnxGOXynXPy3yHkVYh3a/rzU0Je5RcMeTaSP2nt7Ryyk39+yHXIca2t6U/7CSEX7r9cyLulKf8z+uzT+lST/tY4B5XnmNTyaKtmOWTeQpTApCObgzU8i/dX+dfPr759dFprcuXMvRwEtCMTI1c5UFi6Uw35KX+QnFa1RoaEoa3U/WQB9b2+TRrjqSDfHuZGOZXa9HXSwlj7Y3z94SktfPL5N9/+7p//8uXv/vTDJx/++Ke//PD8/D+fX/wyvDNnSlVe6oA77ecQz61TH3lwrcz/J8RLBOIlLyoS5Vxu2GDazyFeJMZVepJvdGrVIc9jf7F/Ceb5pzOX8DbB/wfmcxDX1ZtnbtrfPi5Q6LNSnqmMKhj7zzvOJcpUZ5ospH8cefqI/H/Fqh6RlTVTXSVLJ5ioHePbUlnGu/3jCLftATHxDv1Hnv88Pf+37x8/8Zk6eOZJq+6T7CddvoZdzkGpyx+mSxtJBxwn/TunlynDEjkk9znPmjnJMqD+LCYXHL9wydT6HodduLx0ZuIp7zFPtLwamAqMXqhIIFX++vVK5aV3yXlkDJvHiZVXA1OB0QuVJj3vzlIuVF76kO9K59nHiZVXA1OB0QsVlhO+DnKvVF66joS7vAufWTk1MBUYvVDpxG2n8BcqL13ib7Nwu7F6qYGpwOiFyqB562m3NnTYWOrRyxojpwamcYtaSCwSP8tY6ULjpUvsiTmXE6OXEpgGjPzDYySJYefKZyKmCwWJf0hYnpNX41LB0QuVTD3r1NiVykvXo6LUleaJlVcDU4HRC5VC8orrKPc7p0v83Kv8d2X1UgNTgdELFckIGYzxTZdHbfBa68rqpQamAqMXKkwtgxG+6aK11coYJ1ZeDUwFRi9UOkkKfR/hm64kuuSA68TKq4GpwOiFyqDFYIRveq8kGFI+s/JqYCoweqGyqBQwwjddxq8951TWmZVTA1OB0X94THnZAiN802VkwqnXnQ07Vk6NSwVHL1SyZHVXIluTzK9U6VHbmZFTA9O4RS0kpE8oYIRvei8ST52rGiOnBKYBIxci8rdeYIRvep80Wkq5nTk5NTAVGL1QYRmTghG+6dJjVDm55DMqEwMzgbELk07zPov9ndNl9CqJTuITKCcGZgJjFyaTMprBN11GIav2sc+31opXA1OB0QuVRRXN35vOOh6R8NuJlVcDU4HRf3gseSaavzddZ6elzd5OrLwalwqOXqhkWmj+3nSdPxm17fkBa8WrganA6IVKpYzm703nKW1xTfXEyquBqcDohUqTl4HRveldq9AsneyJlVcDU4HRCxWmce1pD03i5plWy1dGLzUwjVvUQqLTQvP3ps9CpfXE/GLklcA0YORCZGp099G96RpazrxzPWvFq4GpwOiFyiJG8/emSwbMPLmMEyuvBqYCo//wkEGYDjTuA3xnSNZXp3Sk5UzLqXG5/Ej8CqZQQnP4zmiDJBnmVc68vByZDASgZKokvGCk7wzmXTwfF2BejkwGAlAy0leguXxn6Ik48crlDMzLkclAAEqGaaL5fGfwolTySPkMzMuRyUAASkY/Ohj1O6OznKLL2AN815CXI5OBAJSMnGHQvL4zury2S7PjAszJkclAAEpmUb8fL1tser1or1dQXo5M5Ba40MiJFprfdwZ36rmP1AyUlwITwcErlUIZzfE7o+vpeOV8geXlyGQgACVTqaF5fmdIdDK4nTtBdA15OTIZCEDJNBpott8ZcjqebeQLrpcYmQoMXql0Gc+jTMAMGdjWKp3JvMBycmQyEICSGVTQnL8zpJvNJdW3b5IBc3JkMhCAkpnEaN7fGVylW9HJygswJ0cmAwEoGekr0Ny/M3qhUhKPeQbm5chkIAAhUzIlNP/vDOlo5R3SkQlYQ14OTAYDUDJFokCZgBlcZICb9vJZz8vUyFxg+Mqlyhj2xmSLckgUzq3UCyYnRyZyC1xpNJqoEuCMXmlKBr0XAL014qXIRGDwSkU+M6oGOKOxPFrHxIu149TIXGD4ymVQQ/UAZ8jwrUtOtBdGuYa8HJkMBKBkJnVYEjCDM63M0vYFmJMjk4EAlMyiBWsCZnDS1c3lGO86YE6OTAYCEDI1U4E1ATP6pFZGruMMzMuByWAASqZQgzUBM/RC7bmOy1pcQ16OTAYCUDKVBqwJmNGaLnnncubl1MhcYPjKhXXFGMgDzNBF72322s64vByZDASgZLq0gvIAMyR9bpIl7stxPTAnRyYDASiZQXw/XrYop57etVR/AeXkyERugSuNSRNWBMzgTlW6lGPG7g2UkyITgcEfu8UlWBEwo2fKq869xYhryMuByWAASkb3hUC5gBlzyhulNNsZmJcjk4EAlEyhDisCZkhHkkrLvZ6BeTkyGQhAyVRasCpgBjcRB68rMCdHJgMBKBmmDKsCZnDdX6B54WVqZC4wfOXSqcGagBlNkqFVM9czLi9HJgMBKJlBA9YEzGBdEDGlRzkD83JkMhCAkpm0YE3ADJaDI7W0F4J7YE6OTAYCEDJyfimwJmCG9CdDzkWLL8CcHJgMBqBkMvGt7z1E+boUaeDIBRwoJ0cmcgtcaejGqigXMEPS5sK1782NPoJyUmQiMHil0ijBqoAZcl5udXG+wPJyZDIQgJJhyQJRLmCGfnGyDHXnGZiXI5OBAJRMJ4Z1ATP0Au+W81ExccCcHJkMBKBkBk1YFzBDR7m8yixnYF6OTAYCUDJLPjPKBcyQr81M69gV2DXk5chkIAAh0xNVWBcwQzrdntPgCy9TA3PB4SuXTB1WBcyQoVwaKU8+4/JyZDIQgJIptGBdwAwZ/tda8l6s5oE5OTIZCEDJNL1mDuQCZuwLDjuvMy+nRuYCw1cuct69Hy1bbIsG17IXe7oGvByZyC1wpaEPUSZgRl+6uGYe8y9HI16KTAQGr1QGLVgVMEM6kqV7bOcLLCdHJgMBKJlFBVYFzBgyfJvpbYbBGvJyZDIQgJAZ8q6wKmBGXZImvm1W7xrycmAyGICSyTJqRZmAGTJM6a3Umc/AvByZDASgZColWBUwg3V72PVWY7OGvByZDASgZJqMzlAmYEaT5ri243o7a8jLkclAAEqGqcO6gBlcdYlNy2deTo3MBYavXDpNWBUwQ9PnNkq54nJyZDIQgJKZlGFVwAzed7J5m+11wJwcmQwEoGTkzHvve7coUbWaas8XUE6OTOQWuNCY8jxYFTBDTsg15eOObG+NeCkwERy8UpERPawKmMFNh3KjXGB5OTIZCEDJVCqwKmCGRCsdyUjlDMzLkclAAEqm6Z4uIBcwo40933JcxuuAOTkyGQhAyTANWBUwQ4e2JbW0zsC8HJkMBKBkBiVYFTBDEOTUx7GO4gTsJUcmAwEomUkFVgXM0KvfV3/LBawhL0cmAwEomaV3agS5gBmsxbTMx5WI1pCXI5OBAITMkjE9rAuYIQfHWtL1tjMwLwcmgwEomaJ3gwDZgBlyEupDvjYXXqZG5gLDVy6SHt+Pli3uqzF737eg8ZicHJnILXCl0eRsi3IBM/Qij5nmcfXh0YiXIhOBwSsVpgnrAmY0rZ6tclTurSEvRyYDASgZ3V0M5QJm6ObDOfNxhZA15OXIZCAAJTOpwbqAGaxbwEhqNM7AvByZDASgZBYNWBcwo61jIWw+A/NyZDIQwIdHSYkWqgs4o00Ji0e7AnNyXDI/AkDJFCqoLuAMGdpyW2uceTk1MhcYvnKpulfqPRNwhkQ3WuF9+wSPy8mRyUAASqbRQHUBZ3S9xUTmNs/AvByZDASgZOT8guoCzmh6uUfu4wLMy5HJQABKRm9fdKOyRb01y0h1tDMoL0cmcgtcaUxiVBdwhqRCuivDGAbKS5GJwOCViu7vCHIBZ+jymqlXMl9gOTkyGQhAyOQsnxnkAs5gvcSj1rdz0ashLwcmgwEomUIV1QWcIUPbvXRinoF5OTIZCEDJVOqoLuAMQcBJmm43YC85MhkIQMk0Wqgu4IymAxVpO9+AveTIZCAAJSOfGdUFnMHadptHD2wNeTkyGQhAyQxqqC7gDDk9F0kTeZyBeTkyGQhAyUx5iLIBM7QEu2YbF16mRuYCw1cua1c97rmAGdz0pkjH5jiuIS9HJgMBCJmSqdyOlkPcW1m3Y9MKD8rJgYncA1cahRjVBZyh97bJo659pfdbKyctMhMYvnKpNFBlwBlyOi5tlSNHsoa8HJkMBKBkmBKqDDhDt8gck/dFda4hL0cmAwEoma4r0EA2YIZ0JfLVObbI9MCcHJkMBKBkBnVUGXCGrvjknDpfgDk5MhkIQMlMmrAyYIaMVFqqc/IZmJcjk4EAhExNlGFlwAwZ9tdUji3hXUNeDkwGA1AyWf7yKBswo01dWMIln4F5OTIZCEDJSG8BawNmyOm5r1zfyBgwJ0cmAwEomUoL1gbMYL25o4zo8hmYlyOTgQCUDFO59b6HyHqPhVx2wd6DcnJkIrfAlUanBmsDZuztyHjM4UA5KTIRGLxS0ZoqygbMEASpprdphhOslxyZDASgZBYlWBswY+xtlY7NPFxDXo5MBgIQMk1agbUBM3TYn5vk0WdgXg5MBgNQMpkY1gbMkNNxLmntjf1cQ16OTAYCUDKFJqwNmKGbtqV0bDPlGvJyZDIQgJLRK1FRNmAG6xRM4bQuwJwcmQwEoGSYKqwNmNF2WOuY6bWGvByZDASgZDp1WBswQ0/Pc99szTVjYmQqMHilMmjCyoAZurI81bxve+9hOTkyGQhAycjZ5X6sbFESZ+lBxjEHYw14OTKRW+BCQ2uqsDJghnateRwbE7814qXARHDwSiXLc1EmYEbPpFO5exmJa8jLkclAAEqm0IJ1ATN0U2KdZGlnYF6OTAYCUDKNCqwLmKGXeZcy+QLMy5HJQABKhvVaH5AJmME6kEu9ljMwL0cmAwEoGXkI6wJm6KRL6eOYyTwBe8mRyUAASmZSgnUBM6RDkSMjc7kAc3JkMhCAklkyPkOZgBkyyJ191Xzm5dTIXGD4wqXLu8KqgBn6pdG7aK0zLi8HJoMBKJlME1YFzNhFe+lN+ALMyZHJQABKpspnvlHZog5XuLWj33WgnByZyC1wpdGowqqAGTrQHZyPK8DfQDkpMhEYvFJh6rAqYIaecnj24xpnB8vJkclAAEqm04JVATN0u/y86t7KwwNzcmQyEICSkRE9rAqYIYn01J3s8hmYlyOTgQCUzKIGqwJmNL2h7jo2Vj0De8mRyUAAQmYkGrAqYEbTmzvOVK7AnByYDAagZGQUAqsCZrR9xfdxWjrx+qhG5gLDVy6VCqwJmCFimfrfGZeXI5OBAJRMI4Y1ATNk8LJyn6tcgDk5MhkIQMmwzsKBXMCMIRnjLO0YzVhDXo5MBgJQMoPS/XjZol4ctWbO8wzKy5GJ3AJXGpMqrAuYofe4TKstD8pJkYnA4JXKIoZ1ATN0im7J+8wLLCdHJgMBCJmZZNyKcgEzdJlw55TyBZiTA5PBAJRMoQzrAmbopczj7Yt04vVRjcwFhq9cKjVYFTCDdR8YOTnPG66XHJkMBKBkGnVYFTBj3zlr8t6YzANzcmQyEICSYVqwKmCGdLQ953qMXhwwJ0cmAwEomUEFVgXM4CZfoLfLgxwvUyNzgeErl0kNVgXM0HLjTLzqDddLjkwGAlAyiwasCpih281Kz3IUkhwwJ0cmAwEImZUp3freQ9Qt7FrK5QLKy4GJ3ANXGvuOyiATMEM619Fmbg6UlyITgcErFd2YDmUCZujtl2d5WznsYDk5MhkIQMk0mrAqYIaWFbUDqTdgLzkyGQhAyXSdQQGZgBkytM19lFYuwJwcmQwEoGQkClgVMEMnW1KZx2rzE7CXHJkMBKBk5r5j2D0XMEO38mCe5cLL1MhcYPjKZdGENQEz9I6oa/JRR3K4nByZDATw4VFTpoyqAs5geW3mmtsFmJPjkvkRAEqmUENVAWfI4IXl1Ly3y3cNeTkyGQhAyVR5LsgFnCFfm55kRFfOwLwcmQwEoGSkG70fL1tkvdCjpn2bSw/KyZGJ3AJXGp0Kqgo4Q28+lxJndqCcFJkIDF6pDGqoKuAMia7kt+VpHpaTI5OBAJTMpIGqAs7gqZdLlb0Frwfm5MhkIAAhkxMlVBVwRm/yqB3bqrqGvByYDAagZDJVVBdwRl/Ua5s7UfTAnByZDASgZAoxqgs4Q5fbjHRcJOUa8nJkMhCAkqk0UV3AGVyopDLWFZiTI5OBAJSM7jmGsgEzNIVu820c44A5OTIZCEDJdB2ngWzADF1UkkeqfAHm5MhkIAAlM6ijyoAz9NJvHmUvzHINeTkyGQhAyUwCve8Wj2H/2lvAuAa8HJnILXChURJlVBlwhoQ/R8nzDOolBSaCg1cqWV6IsgEz9HbUnNe+yNvDcnJkMhCAkpFQUGXAGZIqtrJ432LYA3NyZDIQgJKptFBlwBm68EhSxTpuwF5yZDIQgJJhKqgy4AxdTNJa2Tf88cCcHJkMBKBkOjGqDDhDopPBHK95AebkyGQgACUzaKDagDMUwRql1Duwj3JkMhCAklmUYG3ADF0Wq3eiuAJzcmQyEICQqYkqrA2Y0ZPeyLCOeQbm5cBkMAAlk4lhbcAMXR4r77Q3bPPAnByZDASgZArN+/Gyxa7X1820NyHwoJwcmcgtcKXR5GOjbMAM6UR0ZdbIBspLkYnA4JUK673sQTZgRi/UpJM9ZjOtIS9HJgMBKJlOHdYGzJDopCtZ68zLqZG5wPCVy5A+AuUCZui1HVUSgHLB5eTIZCAAJbN090uQC5iht0WdtR1ZkgPm5MhkIAAh0+SFsDJgRpeOpY1jw0PXkJcDk8EAlEymASsDZgiCkWerd2AvOTIZCEDJ6Fo9lAuYIcOVIaOXdeFlamQuMHzl0qjAuoAZXRdSS5Y4LricHJkMBKBkmBjWBczoukFZn/sSMg/MyZHJQABKptO8971bnIlSTnLyOYPycmQit8CVxqQE6wJm9Ep9ctobwLw14qXIRGDwSmVJFCgTMKNPKqXLYXGB5eTIZCAAIcPyXFgXMGNfmlr7KGdgXg5MBgNQMpkmrAuYIWliLX3lfAbm5chkIAAlo5VmlAuYISKPOle+AHNyZDIQgJJpurcWyAXM0A4lreOU5HiZGpkLDF+5sJxfUCZgxtCNZ1vd2+K4hrwcmQwEoGS6LhUBmYAZfVEePeV5BublyGQgACWjZ16UC5ihm9r1XPZtXDwwJ0cmAwEomUUNVgXMkINjpZKOK3sdMCdHJgMBCJmeaNyOl0NUYSaJ8ALKyYGJ3ANXGoUSrAqYIY/6mqN6UE6KTAQGr1SqtIJyATN0+r8M3rdEPcN6yZHJQABKphHDqoAZI8vpuCfmMzAvRyYDASgZpgnrAmYIgtlH2kvLz8BecmQyEICSGbq+COQCZshr2/i4YsAa8nJkMhCAkplUYV3ADDk4OvNo9QzMy5HJQABKRteKoGzAjKEF2Fn2EmoPzMmRyUAAQmYkGZ+hbMAMedRGbeUKzMmByWAASqZItoOyATPktYnTcauoM7CXHJkMBKBkqrwQZQNm6Nem6fzuBZiTI5OBAJSMVhFvVLYoX5xZ2zquEbIGvByZyC1wpcG0YGXADDkgSu7jqAwcjXgpMhEYvFIZVGBlwAx5NDuveYXl5MhkIAAlM4lhZcCMnLVOkjuPM7GTHpkNRKBsFg1YGzAjSw6wkujtwszrkdlABMJmZj3fgozAjDwT5TZ7vjLzemA2GIGyKVRhfcCM3I97I5VxYeb1yGwgAmVTiWGNwAw5NvSdynFVjGPm9Rebd58WDbs8f/PMVDSi/Hz/fPf5N3/949ff/Pb9Z8+vf3hc+H14E5aMCLgca7YfXz6q7tNTTs/7KF2eeWvv9lL3NHvtj7b3xeO/AVABDMUKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iago2Mjg2CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE3IDAgb2JqCjw8IC9MZW5ndGggMzk1IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD1SS27FQAjb5xRcoNLwm895UlXdvPtva0NSqSq8iTHGMH3KkLnlS10ScYXJt16uWzymfC5bWpl5iLuLjSU+ttyX7iG2XXQusTgdR/ILMp0qRKjNqtGh+EKWhQeQTvChC8J9Of7jL4DB17ANuOE9MkGwJOYpQsZuURmaEkERYeeRFaikUJ9Zwt9R7uv3MgVqb4ylC2Mc9Am0BUJtSMQC6kAAROyUVK2QjmckE78V3WdiHGDn0bIBrhlURJZ77MeIqc6ojLxExD5PTfoolkwtVsZuUxlf/JSM1Hx0BSqpNPKU8tBVs9ALWIl5EvY5/Ej459ZsIYY6btbyieUfM8UyEs5gSzlgoZfjR+DbWXURrh25uM50gR+V1nBMtOt+yPVP/nTbWs11vHIIokDlTUHwuw6uRrHExDI+nY0peqIssBqavEYzwWEQEdb3w8gDGv1yvBA0p2sitFgim7ViRI2KbHM9vQTWTO/FOdbDE8Js753WobIzMyohgtq6hmrrQHazvvNwtp8/M+iibQplbmRzdHJlYW0KZW5kb2JqCjE4IDAgb2JqCjw8IC9MZW5ndGggMjQ5IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nE1RSYoDMAy75xX6QCFek7ynQ5lD5//Xyg6FOQQJr5KTlphYCw8xhB8sPfiRIXM3/Rt+otm7WXqSydn/mOciU1H4UqguYkJdiBvPoRHwPaFrElmxvfE5LKOZc74HH4W4BDOhAWN9STK5qOaVIRNODHUcDlqkwrhrYsPiWtE8jdxu+0ZmZSaEDY9kQtwYgIgg6wKyGCyUNjYTMlnOA+0NyQ1aYNepG1GLgiuU1gl0olbEqszgs+bWdjdDLfLgqH3x+mhWl2CF0Uv1WHhfhT6YqZl27pJCeuFNOyLMHgqkMjstK7V7xOpugfo/y1Lw/cn3+B2vD838XJwKZW5kc3RyZWFtCmVuZG9iagoxOSAwIG9iago8PCAvTGVuZ3RoIDk0IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWNwRHAIAgE/1RBCQoK2k8mk4f2/40QMnxg5w7uhAULtnlGHwWVJl4VWAdKY9xQj0C94XItydwFD3Anf9rQVJyW03dpkUlVKdykEnn/DmcmkKh50WOd9wtj+yM8CmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL1R5cGUgL1hPYmplY3QgL1N1YnR5cGUgL0Zvcm0gL0JCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9MZW5ndGggMzkKL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnic4zI0MFMwNjVVyOUyNzYCs3LALCNzIyALJItgQWQzuNIAFfMKfAplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9MZW5ndGggMzIyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVRu23FMAzsNQUXMCB+Jc3jIEiRt3+bO9qpSNO8H1VeMqVcLnXJKllh8qVDdYqmfJ5mpvwO9ZDjmB7ZIbpT1pZ7GBaWiXlKHbGaLPdwCza+AJoScwvx9wjwK4BRwESgbvH3D7pZEkAaFPwU6JqrllhiAg2Lha3ZFeJW3SlYuKv4diS5BwlyMVnoUw5Fiim3wHwZLNmRWpzrclkK/259AhphhTjss4tE4HnAA0wk/mSAbM8+W+zq6kU2doY46dCAi4CbzSQBQVM4qz64Yftqu+bnmSgnODnWr6Ixvg1O5ktS3le5x8+gQd74Mzxnd45QDppQCPTdAiCH3cBGhD61z8AuA7ZJu3djSvmcZCm+BDYK9qhTHcrwYuzMVm/Y/MfoymZRbJCV9dHpDsrcoBNiHm9koVuytvs3D7N9/wFfGXtkCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0xlbmd0aCA4MyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagoyMyAwIG9iago8PCAvTGVuZ3RoIDMyMCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UktuBTEI288puECl8E/O86qqi777b2sTvRVMMGDjKS9Z0ku+1CXbpcPkWx/3JbFC3o/tmsxSxfcWsxTPLa9HzxG3LQoEURM9WJkvFSLUz/ToOqhwSp+BVwi3FBu8g0kAg2r4Bx6lMyBQ50DGu2IyUgOCJNhzaXEIiXImiX+kvJ7fJ62kofQ9WZnL35NLpdAdTU7oAcXKxUmgXUn5oJmYSkSSl+t9sUL0hsCSPD5HMcmA7DaJbaIFJucepSXMxBQ6sMcCvGaa1VXoYMIehymMVwuzqB5s8lsTlaQdreMZ2TDeyzBTYqHhsAXU5mJlgu7l4zWvwojtUZNdw3Duls13CNFo/hsWyuBjFZKAR6exEg1pOMCIwJ5eOMVe8xM5DsCIY52aLAxjaCaneo6JwNCes6VhxsceWvXzD1TpfIcKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDM0MCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UjluBDEM6/0KfSCAbtvv2SBIkfy/DanZFANxdFKUO1pUdsuHhVS17HT5tJXaEjfkd2WFxAnJqxLtUoZIqLxWIdXvmTKvtzVnBMhSpcLkpORxyYI/w6WnC8f5trGv5cgdjx5YFSOhRMAyxcToGpbO7rBmW36WacCPeIScK9Ytx1gFUhvdOO2K96F5LbIGiL2ZlooKHVaJFn5B8aBHjX32GFRYINHtHElwjIlQkYB2gdpIDDl7LHZRH/QzKDET6NobRdxBgSWSmDnFunT03/jQsaD+2Iw3vzoq6VtaWWPSPhvtlMYsMul6WPR089bHgws076L859UMEjRljZLGB63aOYaimVFWeLdDkw3NMcch8w6ewxkJSvo8FL+PJRMdlMjfDg2hf18eo4ycNt4C5qI/bRUHDuKzw165gRVKF2uS9wGpTOiB6f+v8bW+19cfHe2AxgplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9MZW5ndGggMjUxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nC1RSXIDQQi7zyv0hGan32OXK4fk/9cIygcGDYtAdFrioIyfICxXvOWRq2jD3zMxgt8Fh34r121Y5EBUIEljUDWhdvF69B7YcZgJzJPWsAxmrA/8jCnc6MXhMRlnt9dl1BDsXa89mUHJrFzEJRMXTNVhI2cOP5kyLrRzPTcg50ZYl2GQblYaMxKONIVIIYWqm6TOBEESjK5GjTZyFPulL490hlWNqDHscy1tX89NOGvQ7Fis8uSUHl1xLicXL6wc9PU2AxdRaazyQEjA/W4P9XOyk994S+fOFtPje83J8sJUYMWb125ANtXi37yI4/uMr+fn+fwDX2BbiAplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9MZW5ndGggMjE1IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVROQ4DIQzs9xX+QCSML3hPoijN/r/NjNFWHsFchrSUIZnyUpOoIeVTPnqZLpy63NfMajTnlrQtc4C4trwvrZLAiWaIg8FpmLgBmjwBQ9fRqFFDFx7Q1KVTKLDcBD6Kt24P3WO1gZe2IeeJIGIoGSxBzalFExZtzyekNb9eixvel+3dyFOlxpYYgQYBVjgc1+jX8JU9TybRdBUy1Ks1yxgJE0UiPPmOptUT61o00jIS1MYRrGoDvDv9ME4AABNxywJkn0qUs+TEb7H0swZX+v4Bn0dUlgplbmRzdHJlYW0KZW5kb2JqCjE1IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9CYXNlRm9udCAvQk1RUURWK0RlamFWdVNhbnMgL0ZpcnN0Q2hhciAwIC9MYXN0Q2hhciAyNTUKL0ZvbnREZXNjcmlwdG9yIDE0IDAgUiAvU3VidHlwZSAvVHlwZTMgL05hbWUgL0JNUVFEVitEZWphVnVTYW5zCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0KL0NoYXJQcm9jcyAxNiAwIFIKL0VuY29kaW5nIDw8IC9UeXBlIC9FbmNvZGluZwovRGlmZmVyZW5jZXMgWyA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggNTYgL2VpZ2h0IC9uaW5lIF0KPj4KL1dpZHRocyAxMyAwIFIgPj4KZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9Gb250RGVzY3JpcHRvciAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zIC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Bc2NlbnQgOTI5IC9EZXNjZW50IC0yMzYgL0NhcEhlaWdodCAwCi9YSGVpZ2h0IDAgL0l0YWxpY0FuZ2xlIDAgL1N0ZW1WIDAgL01heFdpZHRoIDEzNDIgPj4KZW5kb2JqCjEzIDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE2IDAgb2JqCjw8IC9laWdodCAxNyAwIFIgL2ZpdmUgMTggMCBSIC9mb3VyIDE5IDAgUiAvbmluZSAyMSAwIFIgL29uZSAyMiAwIFIKL3NpeCAyMyAwIFIgL3RocmVlIDI0IDAgUiAvdHdvIDI1IDAgUiAvemVybyAyNiAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE1IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMCAvY2EgMSA+PgovQTIgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMSA+PgovQTMgPDwgL1R5cGUgL0V4dEdTdGF0ZSAvQ0EgMSAvY2EgMC41ID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9GMS1EZWphVnVTYW5zLW1pbnVzIDIwIDAgUiA+PgplbmRvYmoKMiAwIG9iago8PCAvVHlwZSAvUGFnZXMgL0tpZHMgWyAxMSAwIFIgXSAvQ291bnQgMSA+PgplbmRvYmoKMjcgMCBvYmoKPDwgL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNy4xLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNy4xKSAvQ3JlYXRpb25EYXRlIChEOjIwMjMwMzE0MTYxNDE3WikKPj4KZW5kb2JqCnhyZWYKMCAyOAowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxMTg2NSAwMDAwMCBuIAowMDAwMDExNjAyIDAwMDAwIG4gCjAwMDAwMTE2MzQgMDAwMDAgbiAKMDAwMDAxMTc3NCAwMDAwMCBuIAowMDAwMDExNzk1IDAwMDAwIG4gCjAwMDAwMTE4MTYgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQwIDAwMDAwIG4gCjAwMDAwMDY3MjIgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDA2NzAxIDAwMDAwIG4gCjAwMDAwMTA0MTEgMDAwMDAgbiAKMDAwMDAxMDIwNCAwMDAwMCBuIAowMDAwMDA5ODM4IDAwMDAwIG4gCjAwMDAwMTE0NjQgMDAwMDAgbiAKMDAwMDAwNjc0MiAwMDAwMCBuIAowMDAwMDA3MjEwIDAwMDAwIG4gCjAwMDAwMDc1MzIgMDAwMDAgbiAKMDAwMDAwNzY5OCAwMDAwMCBuIAowMDAwMDA3ODcwIDAwMDAwIG4gCjAwMDAwMDgyNjUgMDAwMDAgbiAKMDAwMDAwODQyMCAwMDAwMCBuIAowMDAwMDA4ODEzIDAwMDAwIG4gCjAwMDAwMDkyMjYgMDAwMDAgbiAKMDAwMDAwOTU1MCAwMDAwMCBuIAowMDAwMDExOTI1IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgMjggL1Jvb3QgMSAwIFIgL0luZm8gMjcgMCBSID4+CnN0YXJ0eHJlZgoxMjA3NgolJUVPRgo=", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["plt.bar(np.arange(mean_out.shape[0]), mean_out, **plot_args)\n", "plt.yscale(\"log\")\n", "plt.xticks([0, 64, 128, 192, 256])\n", "plt.show()\n", "plt.close()"]}, {"cell_type": "markdown", "id": "2b27a02f", "metadata": {"papermill": {"duration": 0.023024, "end_time": "2023-03-14T16:14:17.369147", "exception": false, "start_time": "2023-03-14T16:14:17.346123", "status": "completed"}, "tags": []}, "source": ["This distribution is very close to the actual dataset distribution.\n", "This is in general a good sign, but we can see a slightly smoother histogram than above.\n", "\n", "Finally, to take a closer look at learned value relations, we can\n", "visualize the distribution for individual pixel predictions to get a\n", "better intuition. For this, we pick 4 random images and pixels, and\n", "visualize their distribution below:"]}, {"cell_type": "code", "execution_count": 27, "id": "f81e2fbf", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:14:17.417398Z", "iopub.status.busy": "2023-03-14T16:14:17.417029Z", "iopub.status.idle": "2023-03-14T16:14:21.350130Z", "shell.execute_reply": "2023-03-14T16:14:21.349431Z"}, "papermill": {"duration": 3.965661, "end_time": "2023-03-14T16:14:21.358031", "exception": false, "start_time": "2023-03-14T16:14:17.392370", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNjEyLjkgMzY3LjE4MDYyNSBdIC9Db250ZW50cyA5IDAgUiAvQW5ub3RzIDEwIDAgUiA+PgplbmRvYmoKOSAwIG9iago8PCAvTGVuZ3RoIDEyIDAgUiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJzc3U+TLLd1Puh9fYpaSguC+A+cpTi0GaGdpBuehcMLBU1pxOhLB38Meb7+nBdZnThIHEi0dHnHCTkk335Pd3bnU1UoIAsJfPn1d//9l2+/+/03Xz3/rz88vuxfffvTwz2/5//++Wmf3/N//9+ne37D//3zw/JXHx/ZeUP8r7fXv0IuxlWbfeLIjl/+P4/Hnx5f/oZ/+KenNeRKjsWmWqcvIllH2Zb6/D/4vd8M3/D4W9/9eMRiytPbasrxaz8+gnUmhBxylPmbzEMiQ+95O8KQtD/8x+f10D4Fk8Pr/+LTJWf88/989/y/nz88v/yNx2m652/5v9/zfxvchZZ/Ry7kSxz/4B4Pf8XjD4/fPX98P7I1LvGj8n7w9uU3r/TxIz9i9vmF5VLKxrWD+ZLp6YjBvOPjffvx8dWHx5f/6p7OPT/8qT2cH/7z8e/PX9lfP//j+eG3j3/58Phd+42f/LydKyb4QNaPJy7yT3DmztnX0ULJP+vMc/zlT71EQ7mmcDnzM/4UJ56jiTiYjSX5n3Xmztdf/NR98Ca74BKN5y7yT3Dy3nvjcLSaSow/7+TJ//Inz3+HT6XUS+Mk8k9x8oVMwdFyLuHnPed9yr/4yR8HDMlQKZTLpU3u+T962oX/Hh8cP4tqIpdriK8TtvjmUiHAZ9tP/1fu188P3z+yIeKXSXIp+wMEjR8XXDCJ/5NzZIBnNN5ZCuU8CE6nHyWaSpVKzJbi6yixFRj0R/4DrC3lCfDXP100jtukiANGEyo15G8/PvnQX3z93fd//Le//uGPP/z0xce//PDXn55f/9fzd7/s45LJxEShXh6WM/6nHpWc+GT5PzHZ6MSjYnMtn/lBsT//QeHXUfv7/n97UILlzlS0/MY9Pioi/4ffHTweF6omVRddCT7Wz/K45Dvxh2wiv5PYS/9R5P8Uf+C3PP5zKx86pvhZ+P3/Av5wQPGTzwZ+X0qF+/Lx6MjzuwBX8ntXPraev0lNsn//c/n9j99/8zO/s40S+K3W+0o+P7+olnud/E7puaHg8Qw/Yp6yD2PhTRQCn6PjIxGaxn6gIeYRw1e3Blo4YMBnbLDBTz7vua884kgxl1FtiO/Poyo0HQrOuqrwvAo+V2Mjv0ZpUJPpDjwaA/t4U4Mr3Chefc4CesTczDnnRzYZ3x9Id2CgYPCW7ObX11nw5EzlF5L3FzcR3x9Id2CgyH9/4LfGq897zl+alPmpUi5qIr4/j6rAOsmkEKotE89ZCJaPVpPlHxZqQ3p/Hp2Bfbj/FjAWmXzOAvfK+HXkbHUD2xDfH0h3YKDCfd3k7fz6Ogue+J81l5xGNxnfH0h3YKBq+Acq0QR0Fnzl9/Liaq2jm4zvD6Q7MBAZF3KiPAGdBe4wc1eZG2O6uIn4/kC6w8dH4UFXKJ6mIUYv+OJMrHzoMLrJ+PZAC4cGxCNYoqkN6gVfi/E2u3Jxk/EOQJoDA3FPj5vaOo0yegHDLc/DrZAvbiK+P5DuwECeu30U6jTM6AVfvbE28rld3ER8fyDdgYH474/W1rkNOgs8LDU+51LT4DbE9wfSHRiIRwsRnylffd7zYIvh7qCd1ER8fx5VgXUS/5iLZRpm9AI3x/hnqlmqDen9eXQG9skmRO/K3P6cBV+432Ott2Fkk/H9gXQHBuL36cjv1tNljl7AgItRPI1sIr0/j67APNW4GFKeBhm9gK5OKbF9WCzRenp/Hl2BecjYGH2em5+z4IkwrGhXewSaSO/Poys0HoqR8nSRoxc8Jcw3CcfTR6iJeAcgzeHjo1pTY8p5GmD0gs/RENkQwugm49sDLRwYyJnCg4Q0dX96IfDpRsv/Wwe3Ib4/kO7AQN60GYjTRY5e8Lg0z+9Urg5uQ3x/IN2BgQL3+EpJ0wCjF3DB0Lp8fCIo3GR8fyDdgYEi/xy/aCaf9xxzuRIfp3V1hJqM78+jKrBO4g4fuTg3QGfB8z+9db5dyuhqMr0/j87APpl7fKQ9fc4Cv0eZlF09fM4DDfH9gXQHBuJ/J5viNMDoBXyiXIuLeXQb4vsD6Q4MVI1Nzk99xDP3xZsUso00qsn4/jyqQtOh5ChMI4xeCI5PMcdMWaoN6Q48GgP7kKnJ5zA/fc4CjydM5qM5GtiG+P5AusPHBzci/G4d5nlSvYDBls/JBze6yfj2QAsHBnImp8hfTEBnAdfCkqVa6ugm4/sD6Q4MxO0s9/muUxTfRAGXM0r0x2BLuMn4/kC6AwMFE7kyv4X1Aj4w5U5PiW50k/H9gXQHBoomcFfPT5c5egFzm9EbbJ/m9AMN8f2BdAcG4pY25TrP1uwFtMax8i9Kg9sQ3x9Id2AgzNktaZ6u2QttVgvuL7y4yfj+QLoDAxXu91XvpsscvRBswtVDmwc2md6fR1doPJTqPJHjzPkJYviX+DyaDfEOPIoC61Tu8+H7Jp6z0GatOkvZDWoyvT+PzsA+3I3Jll8pk89ZwBMlUYjlwibj+wPpDh8fDpN2M//vNMoQFV+SSbUcd/JJORHfnmglAaPWlcnzhE1R8SUYl21KV7oeb0CkQ4DIm5h9oKkhEhV8sJMdH96PdEO+AZJOAaRg+I/HxJ4J6awEl9nFHxfnJZ7MN0DSKYAUjc+hzO/4ooJbnZKtfOIj3pBvgKRTAClxqxJjVVqks4Ibp3Oq0aYrnsg3QNIpgJSNzcnV6fKHqHjuTydXHKULnsw3QNIpDiTiZ8M8jVNU0Fkkyu9tksAT+RZIGgWQiqmZS0qbdFZ8bQsGtFl30k7EGxDpECDik8nFz9NdRQUzo0Og6uhKJ/INkHQKIBH3CwvNkzpFpV16DTWHesGT+QZIOgVWcOJfm2suc4vUK/iMnrhL9OpudzyZ3x9pQQEkxx1D5cPFXgiBB7AWl4pGuiHfgEiFgBC3KcXaeYanqOANzOUS20yGfqgh3oBIl4BRML7YMk/zFJXWODtPzl3sZL4Bkk4BJH4yYDLHfIWkV7DUER88pXzBk/kGSDoFkJKxxWMBtgnprODOsUrcWrsLnsw3QNIpDiQq/BY1D0h6BRfVLP8ef8WT+RZIGgWQ+MlQQkpKm3RWcI8CRUoTnsw3QNIpjlUu2z0a84CkVwL/s5ZU2kx8cawh3wBJpwBSNfwmTvMMUFHxpeB+aGfdiDfkGyDpFEAiw33BHJU26az4ao2NPqZ6wZP5Bkg6BSN5/smSQ5yHJL2CWdUZ84vcBU/m90daUADJcdewzOui9ELggRq/5ftjSNKPNOQbEKkQEPLcLyw8iJ+JzopPuPu32ugGuiHegEiXgFHAcqsxzJdIegU3kJdyXI3sRxLpBkA6A4AidwnJzdNCRQVrfPKrybd7zAY4kW+ApFMcSFRIuah9FnAXMHNQm58mjjTkWxApEBBKplab/Hx5pFcwB8JiSnEY6WS8AZEuAaNssPLzPD1UVPj3mBBsarP1pZ3MN0DSKYBUTK6O5imiouJD5Pd6jssFT+YbIOkUQMJayT67+fJIr2DtqljeB7UCT+YbIOkUQOLzrNxPnocivYIlKr0LMYcLnsw3QNIpGClYE2q083RRUfFYUs8HF/MFT+b3R1pQAIn/5BrLvMSnqOAmYR500DFeE3gy3wBJpwCSN66maOfRSK8El0whfnHlEW/IN0DSKYAUjK3ZzRNIRQUf8dsYKfoRb8g3QNIpDiSqGXMcFKRXBbP52SaRv+KJfAskjQJIkbuGyif/73n7FMSmHOsVTuQbAGkM4OEGpXKHZx6P9ErrCdkUjg+QOpxINwDSIUCUuUtYKynPobOCd6+UQ4zhQifzDZB0CiDxeXJO8+WRXuHOkLEu2Uwj3pBvgKRTAKmayF8o00d75XhZBevziDfkGyDpFEAiEwg3Cc9IZ8UXawoPQFy94Ml8AySdgpEiDuNynS+R9IrPgfuN72M2gSfz+yMtKIDk+AsflOmjvYJZ2dFGFyc8kW+ApFMAyRtLwZa5TeqVYKNJybX14cWhZLwBkQ5xEBGFUuYLJL2CuWuYVHvMse3HGvItkDQKIAVTiX92Hoz0CqaJViJ8JDLiyXwDJJ0CSJGH71jEcjJ6L4CCSYqb5d7jDYBUBvgk/t7EA7AZ6KygI4R3+AlOxBsQ6RIwyiZRTnkejvQK1rLOmXJ1o92Qb4CkUwCpcJeQe4LzcKRXsCJowIxsP+IN+QZIOgWQKncJy7zdRy/gxj4exhLRlU7kGxCpEBAiHCUnpcE+K5hrXHOg45NIQSfiDYh0CTZKlr+goEwc7RU0znzCtuSLnczvj7SgAJJrm+mmuT3qFWDwUN8dk9lHvDPfAEmnABI28LAlzkORXsESYiUnX8oFT+YbIOkUBxJ/b1QmjvZKe8ZQDYUueDLfAkmjABJ3nK3H4H1COiv46Jrf5UPbllLiyXwDJJ0CSNx1tr7G+fJIr2BJVbbhztAVT+QbIOkUQOK+sw1JmTvaK558++j6GPtLPJFvgKRTAIn7zjZi5eYJ6ax4bAjL3cdaLngy3wBJpwBSwQVFCvMFkl7BzH4sO2KveDLfAEmnAFLF0CsrRu8FTBT1nn88X+lEvgGRCgEhwj55QZlA2itoePjZElMZ6WS8AZEuwUbYu9xyIzxfIukV3B9iQyYqFzuZ3x9pQQEk7ubYgk2nJ6Sz0nbdyZaOj5AEnsw3QNIpDiSy/F4+D0l6BadZyL4+sRV4Mt8CSaMAkuduITnlsnavBJ+5FXLH0s/iWEO+AZJOASR+Mliqbh6S9Ar22Cs+kc0j3pBvgKRTACkazE5TJpD2Cj4KyVgMIV3wZL4Bkk4BpGQ49lZpk86KT9ZYZ+vR2xZ4Mt8ASacAUjbRObxiJqSzgqU0svPluAIg8GS+AZJOAaSCpUSyMoG0V4DBP1uPC24j3plvgKRTAKniDodw3U/+TVbadezgXpPaJZ7IN0DSKYBExrlANF8m6ZU25ZgPZic8kW+ApFMwEnY7dzErE0l7BTccl5xdveLJ/P5IC4oDiVwK173l32QFmxwUflkdE0n7sYZ8CySNAkjOVB6OzT2As9BWQGIU70e6Id+ASIWAEC695qLMIu0VNDw12pLrSCfjDYh0CRgFrP8cr3vMv8kKboOIMbwWj5J2It8ASacAEhbsr67Ol0l6BW/zCatElyueyDdA0imAlLBKRlUmkfYK9ldLOEM/4g35Bkg6BZAydwspKdNIewV7rPG7mD3mAPRjDfkGSDoFkIrx3voyXybpFSzuH0rwx5q/Ak/mGyDpFEDiDjMPVMs8JOkV7HOUQj0+JhF2It6ASIcAERnrebg6D0h6xWNvZx7yH7OSBJ3MN0DSKQ4k7giGrLRIZ4XHsfyiwg7YFzyZb4GkUTASdkj3wSqfSfaKx+fZMdKr2e54Mr8/0oICSPyK8UF5azsL+FyND5d9uNDJfAMiFQJC3mQf43VD+jdZwXV+HuMXd6GT8QZEugSMuCfok0vzgKRXfKJ2CfIY+ws7mW+ApFMAKZroU1WmkvYKZkNmxxzugifzDZB0CiAlE3zWnkjvBVwrioxSaaQb8g2IVAgIZe4SFh/nCyS9gttpo0OXeqAb4g2IdAkY8Rf8/qRMI+2V4Ishitm70W7IN0DSKYBUuUtY83Wf+jdZwYdFtRLuZBvwhnwDJJ3iQCJ+Gw/zBZJeaVv3lpL8BW/It0DSKICED8usVaaR9grWY+c+kX012+exhnwDJJ2CkbCberBFmUbaK1iPPbgUj3X/BJ7M74+0oACS4zNwMcwXSHoFy0S5GMMx+pd4It8ASacAkjcpeOfnEUmv4LOQgt0i8xVP5Bsg6RRACiYGfpuae5K9guuOJVT+zwVP5hsg6RRAiiaEkLzSJp0V3DFCzrt8xZP5Bkg6BZCS8SEq023PAtYcoeLCsaqNoJP5BkQqBISycSGSMo+0V9A6Bx7AxjTSyXgDIl0CRsXYkPJ1P/s3WUG7U337BHK0k/kGSDrFgUT81zvltXZWgBGDjUdHcsQ78y2QNAogVe4WFqtMIukV3HcUsOnIFU/mGyDpFEDC5o+lXHe4f5MV7FlLlbAuwogn8w2QdIqPD49t17GK73SZRFTa0KNiT+0Lnsxvj7SiABIP4gO5eRqpqODCWg0x2nLBk/kGSDoFkDy3vVhXZEY6K/g0xPFwtq2FJPFkvgGSTgGkwF0dG+d1W0QF645aF4/pWQOeyDdA0imAFI3Hx0HTkERU2s0QMeWcL3gy3wBJpwBSMi66eTnSXji2GuOfThc6mW9ApEJAKGOnNf65meis8K/h11SwbpQT6QZAusMhRBjKTwMSUQl4K+OjuzDKDfkWSBoFkIqp3ALPk0hFBQMPbqNdW5xFHGvIN0DSKYDEJxPjPGOrF3zyJnk6ZkFKOplvQKRCQIhMjok7zjPRWfElG+5Pu+JHOhlvQKRLsBF2XI/8nfNgpFcwdy2mEny62Mn8/kgLCiA5/slcytwe9QqeMSnlY+l6iSfzDZB0CiB57hDiXpkZ6ay0kVmidhPkYNfjDYh0CBAF7hBWN08gFZXgg4m4GzKPdEO+AZJOASR8nljrdYv7N1nB52n8wsrWj3hDvgGSTgGkxJ1CSnm+PNIrvvB58ZtZu5tW4sl8AySd4kCiZP08h1RUMKXGW37uuAuezLdA0iiAxB2dZOm6xf2brLQ7/PmXHX3tEe/MN0DSKYDEp4SV++fLI72C5aL40InSBU/mGyDpFECqJif80GT0XsD7PP+W9Hpv63Qy34BIhYAQ7m/Eh2Uz0VnBHcc5c5/6QifjDYh0CTbCfus8uojz06hXMHmdf5N1+WIn8/sjLSiA5ExIMc6TSEUFzxgfQrV0wZP5Bkg6BZCwi3hy8yRSUcEQluL7hX+JJ/INkHQKIAXjUqphvkbSK7jAXwMX3AVP5hsg6RRAwk7i/EY1D0l6BX0hbKrlygVP5hsg6RQHEqXi50mkotKmHVdr22f8Ek/mWyBpFEBK3C3ElrQz0lnB5dmSsRnCBU/mGyDpFEDK3DGs2c9Dkl7BWhHFpdymHUs8mW+ApFMAqXDPkMI8iVRUsJN24WfPqwvQ8WS+AZJOAaRqUrb2utX9m6xgfz9HpcQJT+QbIOkUQCKDS9VuHpX0io/VWEvvDXfHk/kGSDoFI2HnddwKOg9LeqWtOhL4yUMXPJnfH2lBAST+k7N3163u32QFN7DHWGPbC0niyXwDJJ0CSN647OddNnshWD7JVPIxi6Qfacg3IFIhIBSMzSHNs0hFxZe265gNZaAb4g2IdInDiHL0123u32QF245xbzGkcrGT+RZIGgWQoqkZex3PSGcFg9ic67FXhMST+QZIOgWQEnaCSsok0l5ps/wJ27Rc8GS+AZJOAaTMT4bslUmkvYKtWSz56OsFT+YbIOkUQCrcLcxE82WSXvHFmxQDtiAZ8WS+AZJOAaTK3cKSSXn/PyvBVUOhluOCWz/WkG+ApFMAibhbWIMyj7RXfMqGu4vRuRFvyDdA0ikYCbuv83fXuU3qFfSMqqdj5xGJJ/P7Iy0ogOS4Z0ilzkOSXsG7GFaOauuOSjyZb4CkUwDJG1u0jyXPAi6r8VFTChc6mW9ApEIcQlScKyrRq4KxRwnRhTTSyXgLIk0CRsHU4mqZhyS9gm3HsuP+dbnYyXwDJJ0CSPxkKD4pU0l7xfN5UU3Z1guezDdA0imAlEwuyp5tvYA5bJEh8pVO5hsQqRAQyiaVQFkhOitYc7xkb0sd6WS8AZEuAaNiMHVWmUjaK1hOw5fsXL3YyXwDJJ0CSNWEksJ1u/s3WcEwzVrK9WIn4g2IdAgQEZ4KNs0XSHoFM9iyzeEY1go6mW+ApFMwEvZfL7ko00h7Bc8dF3yuVzyZ3x9pQQEkx33CEq8b3r/JClbTjhToePMXeDLfAEmnOJB4aMF/uYb0quDdjN/D8nHnv8CT+RZIGgWQePhealWGbL3SPlHztR63/wk8mW+ApFMAKXC3kJIylbRXMNEv1ba19ogn8w2QdAogRZOr5VHGjHRWfPameHJtJaQBT+QbIOkUQEr8ZLDKhe2zgNv8qOJ+tgudzDcgUiEglA1uwVLmkfYKGh7uWLu2dY2kE/EGRLoEjLB2jw9BaY/OChb4sRm7117sZL4Bkk4BpGp8DTbMA5JewYA/YFKtv+DJfAMknQJIZFwNRZlG2ittnM+/6RjaCjyZb4CkUzASv4NbbnmVy0i9gqv9tlI6rkcKPJnfH2lBcSBRTc7Pb229grtqU06v+UgCT+ZbIGkUQHKGu8tVmUbaK1h1NCdffLngyXwDJJ0CSNwXrDldN7x/k5V2OcSXdFySFHgy3wBJpwASnwE+v58vk/RKcNxA8+GP6239WEO+AZJOAaTIPcNCyjTSXsF4lgchr3FbP9aQb4CkUwApGdzOf93w/k1W2ngWK0ZNeCLfAEmnAFLmriEFO18m6RXMhoyuvnrcAk/mGyDpFEAqGH5ZZSppr+CflQsUrngi3wBJpwBSxRdFmUraK5jD5nKsJVzwZL4Bkk4BJDKWnNKVfOXoFjkehsRygZP5BkAaw8FDhD0zNaBXBVuO8S953bV9wsl0CyANgomwEBRhs6OJqFfwcVEq1rornczvj7SgABK6OCHS3Bb1StsEweYy4cl8AySdAkiev1ldrbVXMDWLh/j1WI9M4Ml8AySdAkjBJEJ/Z0Y6Ky+MeAxHJry4yXBkQQGkaCKPKKrSJp0VjGUT9xqPe7YFnsw3QNIpgJRMIPW2tl7BlcfqnD0+2hZ4Mt8ASacAUubDYOLsjHRWMDZzJcS25bjEk/kGSDoFkPBFyfPmyKKCK7Vkkz0WJJN4It8ASacAUuWOYQ1lvkTSK8fdNe8LAAk8mW+ApFMcSESk3EZ6Fvj4JidsG3Whk/kWRAoEhIj7hfysUBqks4KzTB6Ls4x0Mt6ASJdgI+y9bm3M8+WRXsFgn9/xbUgXO5nfH2lBASTuO1vnlGmkvYLptZn4DOmCJ/MNkHQKIHHf2Tqls30WcP8xn1k9VrURdDLfgEiFgBC3KdbjU9eJ6Kzg7qNY3LF1raCT8QZEugSMuN9s20KQk9FZwcSRYjET8mIn8w2QdAogcb+ZD6JMIu0V3DOaEtljJRKBJ/MNkHQKIHG/mVvgOF8i6RU8Y8iVVP0FT+YbIOkUQOIvbApRabXPCuayY7Hfki54Mt8ASacAElaBylaZRNor+GQ2cK/oWGlb4Ml8AySd4kDCldeotElnpV2EtDWkesGT+RZIGgWQsIJviWEekfRKWwbBBe9oxBvyE+nL33icuH/+9skcOCf3/Ob55dff/fdfvv3u99989fz2p8cV8uORBFxOOJK3x+MPj2D5qXrYDt8s8uuPzEfWD9K/WRzhbx75y9+49oDyeWFL6JIP8OmL+G7fHlQ8pM9vhm96/K2feDx4EIyFg/j/xvPINo2n9vaKhj9U/PSQ88Pzp/bw9G+4/oq//WxuD6rDg/r8/uke2qPKP4Wtswgf5g+PQM+vqOIJY7nH/fzzcfj2QuAvv3mljx8fjh2/sFwKuFMAR7M52vR02KDbOz7gtx8fX314fPmv7sm/6MOfHpZ/4sN/Pv79+Sv76+d/PD/89vEvHx6/+6URoneYG5OwXu/wBO/5J0CILh5Hq4Xb8p+FkOPnVOAxeeCWIU8v8zP/FAoZu9ny0Y5FNn6OgvP1MzKkwL22kgulkUHkn4Ah+cxDMXzGSOFnKpD/nArET1bvIhb5HBR6/ikUeETh2tFsKP5nMXCn+jMyyIY98Hs0v3eX2Bv2Hv3DLaPDh1/9b+enhY8xUc2Whw0HA588/xFY3YAJusmv3K+fH75/8LOSagrJpewPJbSdXHA8pOX/ZG5yuRKNd5ZCPwjeR/tRoqlU+UQy7jY5juJbgZV/xO+3hXssT/v+T2yVY32IOCAfulD7+779+ORDf/H1d9//8d/++oc//vDTFx//8sNff3p+/V/P333eBytj/x6byfcHq0f/+IOVhwcLqyhVx2kN8bM8VnTnh4ScqQV90PMROZNP9YDwiCk57mAny6+gz/KI5Bs/IsF75or89no+JCL6RI8JBhwuRe72+MqDns/xmIQ7PyYJV7FtjaU/Jj36hx8TPKziMYn8t5OLhf9Ul/65xwTXKEK1NfEjsX5I7PtD8s9ZYnj++NmD6L83PH8M3/8/G57zn2cSeWh8wTBkLG4W96+RE3fqMCd6qLzJSntM+WBUW34ea8jP4fkOXDoKuDB9IxCu4l+5zgreaVPyrl21EMca8q24dBRwYSJHzLief+U6K2DhM/RtlueF8cy34tJRwIXLY2i3Zq6z0tYwxHrY8cIo8624dBRwYXpHtlFpu84KVluLqXgqF0aZb8WloxxcRLngav/M9apgYjFRtDZcGGW+GZeGAi5M+ijY5nniOittNmh0RFdGmW/FpaMwFz9BClU3t/Rnoc2i5be/ekWU+U5YOgmsHH9vrfhM5Ip1VvDxCL/eSvsUTSDKeCss3QRaPBwgtNqz1lnBvCybgm0T/KWizLfi0lHAhckhPGj0M9dZaYvYpGjbp4+SUeZbceko4MI0EdzuOGm9F3CNoFbnywVxyLfCUklghdkiLvt59NMrwWKH++ja1r8CUcZbYekm0MK0EX6iKG+JZwXTAJJ7dbf6oWS8FZZOAqx2acZ6pdU6K/jsPPlw7HInjjXkW3HpKODCRBJ+n5uHPr2CK9Y8OEzhwjjkW3HpKAcXWW595qFPr+C9D9vgWndhlPlmXBoKuDAzJzmtM39WMCk+2WKPgbVkFPlWXDoKc1XunPPPufmyTa+0+XGB4yujzHfiWqCAy2EF5WTn0U+v4LbUaGM6XoyCUeZbceko4PJ4PXk7t129gt5VKfw0ciPjkG/FpaOAK/ATpJCdO/S9ErC9cM0+XhRFvBWWTgIsLK9YlWfWK8fUZ+5j4cbnAXDIt6LSQACVuLNJfJyZ6qxgdgyRtRQl4ZBuRaWTACtzV5OI5gs2vdLmwVZXi7sgynwrLh0FXMVY16b6TFxnBbeOkauYlDAwDvlWXDrKwUXOhToPfXolWHyMSJXSyDjkm3FpKOCqpjo+pfmaTa94yqZgtcYL45BvxaWjgItMcR7fOHGdFfTZ+VkUQrkwynwrLh2FuYi/2YVY56FPr6BJJ1x6oJFxyHfiWqCAy7UJa2Vuu3ql3d0ZY3AXxiHfiktHARcP9lzETdIT11nx1FZ28MldGGW+FZeOAq5ggkupzEOfXuHfwy86bGVwZRT5Vlw6CrgwATD7orRdZ6Vd/ON3wJwujDLfiktHAVcyzmXsDjZxnRUsSpO5ZxrilVHkW3HpKODiEbIrOc+DoF7xRMbXGsldGGW+FZeOcnDxOIb/fo3rVQm4w427V69n13msId+MS0MB17FKr/LG+F4IznJHyx37r0tEmW+FpZLAis/FUVHm7PYKXnGuFIpxQBzirbB0E2iRyd7GpLRbZ8Vzy86D6Nr2QBoURb4Vl47y8RGtbXPX0nTxRlQweTKFGtpyiJJR5htxrVDA5Uz0rs5TdkUFn09HVokXRRFvhaWTAMub4H2aJ+yKCj95TKrp2GhDHGvIt+LSUcAVjMd8kOnSjaighaqYx5xHxiHfiktHARfuZuY++jT8ERW85nLmQaG7Mop8Ky4dBVzJ8Pnmec6uqLQPL2yJtY6MQ74Vl45ycBGfRlDarrOCPblt9L5eGId8My4NBVzZVJ9tmIY/ooLLyzyYLldFEW+FpZMAq5ji83yZqxfw7sc9eLJpJBzyrbBUElhV7MMVvdJsnRX0Q32ouZYBcYi3wtJNoEVYAhdrekxaZwULCxKV4zZOqSjzrbh0FOZytu07PU/aFRV85FpCrtaNjEO+E9cCBVzcC/CkDBXPArqhyVXXljsXRxryrbBUElh5bnusd9NFG1HB1ZnMh24bn0pEEW+FpZtAKxgXji2KrlpnBdfgY+F3wHpRlPlWXDoKuKKxwWWnvBLPiqeA98HYVmiWjDLfiktHObgo8LNlvmjTK7gvkXwN7W59ySjzzbg0FHAlU0Ow88V5UfFUcLf+sUW6ZJT5Vlw6Crgy9wNCsUrbdVYwIgzZ+zQxinwrLh0FXIXPIEY7X7bpFUz7To478W5kHPKtuHQUcFWTQrLzxF1RweeHxfMZxgujzLfi0lHARSaGVOapu6IScE0+uERXRplvxaWjYMlVi8V843z3sKiAxdWAbXonxjPfiWuBcqxQ67WbFnuB3wNNSKm2T6cHRJFvhaWSwMpzj7PUed6uqOCzfBe4bacRUcZbYekm0Arc4aypzu1Wr+Cmakr1WABVHGvIt+LSUQ4u4mPUeQDUK8cGGu+fwwpGmW/GpaGAK3KHk2ietisq2LimxpxzuDKKfCsuHQVcyZRoc1GarrOCnnvlf7pyYZT5Vlw6CriyydFh4baJ66zgE8Q2N/DCOORbceko4CpYnd/O03ZFBdNzKz4UKyPjkG/FpaOAq/JP8nkrbddZweVlmxO1+aaSUeZbceko4OJ+ZgzcXs9cZwUXmCv3HEK8Mop8Ky4dhbmCxQrwbp62KypYzxebKLya+vNYQ74T1wIFXNxax1jz3Hb1Cnbk5CdRbXtMiWMN+VZcOgq4+Nxi4mHfzHVWWne0BuzxMDLKfCsuHeXgosjnMg+CegX9q8L/c3wcKxhlvhmXhgKugI659nlsr2AzKu99bXfvS0aZb8Wlo4ArcqezzPez9AIPCU12HNcRcci3wlJJYJW4x8mNzzwC6hX0FkJgFDcgDvFWWLoJtDJ3OLF81Kx1VnAruiNHbf3AQVHkW3HpKOAq/JNUotJunZU20Mn+uCVKMsp8Ky4dBVzVhGRjmAdAvdKG0SnklEfGId+KS0cBFxmfnFMm7vYKuqIl2eDLyDjkW3HpKMwVbdvHO8xtV6+0mYHJxeNqhGCU+U5cCxRwOWMTn8c8AOoVzLDJlEu7i0wyynwrLh3l4KLE4+R5ANQreAcMhA1zrowi34xLQwGXNzXxQea2q1fabs7F2XbTnWSU+VZcOgq4ArfWPJ6ZL970ChaYajv8hSujyLfi0lHAxaOYlOZ11XsB82sqpoCHEXHIt8JSSWCVTErZOqXhOiuttxBfT6z3A4lwKyjdA1KZX0e5uHlc3SuYK18jv+zCKDjkW3HpKOAq3NksUZm62yvBYl8yzD+9Mop8Ky4dBVyVO5vKysS9wL+F45zcaCjjrahUEEgR9zNrtfMlm17BVb/Kr7/jZutOKOOtsHQT1kqWu5mUlEm7vYL+OqZAHFfmhaLMd+JaoBxc/I1embTbKzxqNpFHzyWMjEO+GZeGAi5nsKU0zZdsegXzangQbeOFcci34tJRwOVNyS7RPOzpFV8TblTMla6MIt+KS0cBF59B9l6ZtNsrbQUu7pDGK6PMt+LSUcAVDf8Lc0AmrrOCAU4sfEA3Mg75Vlw6CriS4eFLrvPIp1dwy35x3KbTyDjkW3HpKODCJq8xKBN3eyVY/qfjt8MyMg75Vlw6CriK8djZQWm7zgoWio21uGPxg36sId+KS0cBVzUuo18+ab0XsJJNKpRzviDKfCsslQRW/EXOUZm12yvBVcNPpXJ8wn8eaoi3wtJNDi3KaKs1rVelTUHihj2lUXHIN+PSUJgrW+5wllrmyza90ppz7sQfH2T0Yw35TlwLFHA57nDyF/MAqFd8tcaRp7ZJoDjWkG/FpaOAy3OHk3yem65ewbsfFkwiujKKfCsuHQVcgTucRHm+eNMruA4fqrWvZ1dnlPlWXDoKuLi1Ltx0zwOgXsGzKMRkj3vLBKPMt+LSUcCVTCguKJN2ewVtFLatOVaLkIwi34pLRwEXPnb2Ns0Xb3qlrQoRbPDuwijzrbh0FHAV47AlzTwA6pVjucXkj49h+7GGfCsuHQVc1djCDdCs9V7A1XiXsjsGQP1IQ74VlkpyWFHBdhga1qvi+dnETVU5Fn/riDLeDEszgRaZWmLVnlpnpQ2iQ6zHJ0BCUeZbcekozFWsKSUlZdJur2DGFoaFga6MIt+Ja4ECLmdyyfPW6b2A9ik6f/QgpGGPt6JSQSDlTSqZwjz26ZX2xsfnl8JAOMRbYekm0OJnRylZma7bK+hVOevD0Zfvxxryrbh0FHBF7mrWoMx56xXPP26jp+OaoGCU+VZcOgq4Enc1CRvaTVxnpU1CQl8hjYxDvhWXjgKuzGdMRZmu2yvHFpSRfBwZh3wrLh0FXIW/4FeU0nadFSwqFbiFf7X057GGfCsuHeXgouq4K6BxvSqY9J2tP3rz/VAy3gxLIwEW7i13VZmv2yvotYeaS6gXRJlvxaWjgIv4ZHxSJu32CqaKELZ/oCujyLfi0lGYq/I3Y6WH+aJNr6C7ULgrGtOFUeY7cS1QwMWjYywMOGu9F9BAWX7BUbggynwrLJUEVt7wGDkr03Z7pY2fg8OvGRBlvBWWbgKtgM/pg53brV5pI2g8hdxVUeRbceko4IqGX1HWzsOfXvHchXchvD7KEIwy34pLRwFXMq7mrEzb7ZV2R3WwPtWRcci34tJRwMVtdS2BlKbrrLRFf7hrVUZFGW+FpZMcWFSrpXnw0ytYyNNy+35Mq5SIIt+MS0MBV2krkWl9iLPieTRNjhv3q2KPt8LSSYCFk6FYlXbrrOC2Asu/5VjETCDKfCsuHQVc1L65zpdtegUrB2YK9dVwdUaZb8WlozAX8a8l/nIe/PSKx8TTmmO5Msp8J64FCric4f+mOrdcvYKee46uHutVCkaZb8Wlo4DLGz6EV6Y49wp6C/zj9bg7QzDKfCsuHQVcgQ/jSZm22ysYRCfHDPXCKPOtuHQUcPEYhoK21WKvtNs5XcAN1iOjzLfi0lHAlQz3oJSPYs8Cugs+JTqucglEmW+FpZIcVkTJKnN2e8XntjxLzmlElPFmWJoJtLKplIoyZbdXsJFb5jc/ny+KMt+KS0cBVzGFcszzhZtewc12WHzLXxllvhWXjgKuyt/MDfU8AOoVsESfc0gz45lvxaWjgIu4w1mqMmW3V8ASuCN6LBw7Mp75Vlw6ysdHsvyTHKWp7RKVtvqWdTbFK6PIN+JaoYDLcYeTfJoGQKKCvlXgJr7t2SkZZb4Vl44CLo+5kTTP2hUVn7mNchnrS42MMt+KS0cBF/fPreWWe+Y6K7heih1r4pVR5ltx6Sjg4h66dSFOAyBRCQ6f9NhjXUpxrCHfiktHARd30a2fVwXqhbYUniNfywVR5lthqSSHFVmPWTMK1quCrnsKNrSbMPqhhngzLM0EWtw/t/xEmS7ciIovnPN4un3kOiiKfCsuHQVc3D+30c0Td0UF91Vj2bI4Ksp4KyydBFjcO7dxvkDfC1glyVnMPB0Jh3wrLJUEVtw1tyn56bKNqPjiTQw84Ckjooy3wtJNWMvxT/K5zFN2RQWNeaQY2909g6LId+JaoIAL92dmmqfsigpmMlePTsPIOORbceko4OKeOXcv5/10RcVn7o+66Nr9+uJYQ74Vl44CLqz+WoObhz694jPnPuUyMYp8Ky4dBVyRe5v83fPbYq9gJRtXrG2rB0pGmW/FpaMcXGSxxrfG9apgak3hZ5ELF0aZb8aloYAr8XDPRjuPfnqlfXCRU2iLEA+MIt+KS0cBVzZ85m6euCsqrX9lXUkTo8i34tJRwIVPvBwfZ+Y6K+1aaQi2xgujzLfi0lHAVQ2/1ynPrVeO9ikyUbs3URLKfCsqDQRQZHjc5+Ypu6LiM3ZStDHWgVCmW1HpJIzlrQkuVJqfV73iK3E31Nbj2qlAlPlOXAsUcDl0yBPNl2x6BR+HEVFqd70OjCLfiktHAZc3ziU/T9oVlbaZfE3W5QujzLfi0lHAFTBNmerccvUKJiHFFHKbJy8ZZb4Vl45ycJHLuc4XbnoFd/jwj1t3ZZT5ZlwaCrgidzZLmCftigpu5OQ26liXcmAU+VZcOgq4eLjHHaiitF1nBZNzbXaZ8pVR5Ftx6Sjgypi/Vsp84aZXwEIlvr8zDoxnvhWXjgIuzF+jOE/aFZX2DhhyoXhhlPlWXDoKuKqJnofHs9Z7ASi2pnB8rjginvlWWCoJrMgEPqE8X7XplbZwYM7p1enqiCLeCks3Ya3Ah+EjzFumi4pP+FCMDx0vijLfiWuBAi5nnPd+nrQrKr5YQ3xwmy+MMt+KS0cBF5+b9/OmP72AyzO4Bu8nRJFvhaWSHFbkQ05zI98ruEKD5fnDBVHGm2FpJtAKpvoYlAm7veIT96u4W5UviCLeCksnAVbERzc2Ka3WWTkG0HzC4YIo8624dBRwYbuxVOI89OkVX7KpOdu2UY1klPlWXDoKuLLhF1NUpuv2Slt/i1+DxwVnwSjzrbh0FHDhXpTiotJynRV8HJa5R3p8Zi0YZb4Vl44CrsrdTf6++bJNr2Bz00g5uCujzLfi0lHARdzd5PHePPrpFYwJo+ceVrkwynwrLh2FuaLlDif6BRNXr7Q2qpCv+coo8p24FijgctzjJArzZZtewUzmmEo57vsRjDLfiktHObgo8NlpWkfBp2wK/6vUC6LMN8NSSGDlTQ0uKJN2e+W4Sxh3o4+IMt4KSzeBVuCnh7fKfIhe8dyyE/8mWy+KMt+KS0cBV+Qz8LhdeuI6K57zkFq/YWSU+VZcOgq4kuERYFQm7fYKLtC44p1zF0aZb8Wlo4Arm8gF5fJ8r2BRqcRN1MQo8624dBRwFcPnjkUnJ66zgrnMxdaU6MIo8624dBRwVcM/kpRJu73iK/ccQqCJUeZbceko4CLjQvZWabvOis9Y68CXWC6MMt+KS0dhrmT5CcLfPg+AesXnyL+r2LaWzcAo8p24FigHF4WSlUm7veJTMtVyG18ujDLfjEtDAZfjTmcNdm67egXPItcWs7kwynwrLh0FXN5gYXSaL970SrvlLoV6fKohGGW+FZeOAi6cAWVl8m6v4CIN5dJ2cZOKIt4KSycBFvczIx9FabnOiud+qXe+Hr0ugSjzrbh0FHAl/kmn9CLOQruelfll5y6IMt8KSyWBVTbcOy/KvN1ewXp4OQR3zH3riDLeCks3gVYxPvpYlVbrrLRlPMmm49KNUJT5Vlw6CriqcTG4Ol+66RWPOUkhpraMp2SU+VZcOgq4+IsYqjJtt1fwGbXPiY4PNASjzLfi0lEOLorcI1eGP2cFswIt/3TMF0aZb8aloTBXtqbGxAPkiatXcM2hxujLlVHmO3EtUMDlsC0UlXn40yuYX+P4lXd8piEZRb4Vl44CLm8y1m6bhz+94iN3tHK2x/BHMMp8Ky4dBVyBu5wFC2xNXGcF9yiGaF9TdwWjzLfi0lHAFfknq/ZpbK/gAmAs9djITTLKfCsuHQVciZ8gVXljPAvcVTDZ41l0RRT5VlgqCay43xQpJqXhOiu44y66mC6GIt2KSheBVTG47zDNw59ewagwkLXHZ4vCUOZbceko4KoGwz1l6m6v4EOx6Gs65qJKRpFvxaWjHFyU1CfXewG3cvKvSG0TN4ko882wFBJYkeET93G+bNMruAhPgX9VHhFlvBWWbsJaxZqSPCnTdnsFrXkJ9bW4p1CU+U5cCxRwOZNTwDLME9dZ8ZgcWHOMV0aZb8Wlo4DLm5RiCPNlm15Bnz1XH4/PMgSjzLfi0lHAFUxMySrTdnsFdydWxzjlwijzrbh0FHBFExLX56FPr+BjixqpZrowynwrLh0FXMl4Hs+E+bJNr2D1QNxwdzT1glHmW3HpKODK6DjxK2rmOittAJ1cPFaKEIwy34pLRwFX4S4nNo6fuc6Kj8yCdSrLhVHmW3HpKAcXJe5HKW3XWTnuf6XyenZ1RplvxqWhgKtyp5OUic5noe3LXF05PuQXiDLfCkslgRVxj5NImbfbK5j9nbAVZR4RZbwVlm7CWpW/Odvs5narV/CCq/ZY+EAY9nQnqgUIqJzBQrlufhX2CrqhjPBa900QynwrLh0FXB4TZqwy2aZXMIT23Fs4PiYTjDLfiktHAVcwIfti52arV7B0erSvyzZCUcRbYekkwOKnRw5R2e6nVzz/uOMW/VhoSiDKfCsuHQVcybjMPcx56NMrfExjsQJqvjDKfCsuHQVcmc8tFlJarrPiI5nMT6djhXDBKPOtuHSUg4tyisp6QL3iA7EQhWMdEsEo8824NBRwFSwl4pQpu73iMwtx32pilPlWXDoKuPhkclYW2z0LPhTsTuazuyDKfCsslQRW3PbkwvUZ66zwL8Al5pAuiDLeCks3YS3iX5urV6bs9go+tPAU6qs/3xVlvhPXAgVc2IKzkjJlt1fwcWu7pTNfGGW+FZeOAi4e8GVSZr2dBXTbs7P2+MxaIMp8KyyVBFbBcNc8lHns0ytYJInK++5bHVHGW2HpJtDCDkfOKtN1e8VjI2t+vVG6KMr85PryN55P/Omfv30yB87JPb95fvn1d//9l2+/+/03Xz2//enRSSXnR5kHTAs+8rfH4w+PbJMp43cf0fUbV8eeDnD9bnGg1bG//I3DuVk+N2uIW6ADffoivvu3BxYP6/Ob4Zsef+snHo9Y8KcWzL98wWC35+NZ1uM3GbtScXvRkbefHxJ+eP7UHp7Lof/2U7g9kg6P5PN7/m97KB/jQ5kwRKE2HVn8vT0d/gomFE8Ra1x6/vk8dvvym1f6+PHhWO0Ly6WU+ZmIg/mS6UkmVmxF8u3Hx1cfHl/+q3s69/zwp4flb//wn49/f/7K/vr5H88Pv338y4fH736Rc+bnvAk82rB+OGkRf4Kzds6+jhb4ne7vn3WOv/hpl8hjh5rCeNZn+ilOGnNEcTAbsdHT3z9r5+svfdo+eIMriYmG8xbxJzhx3/ap4KPVVGL8OSdO/hc/ce5zeSzHNbZFIv4UJ94+BOCjcWc4/JznuU/5lz7xdjx8IpGcrZfmt8f/8HPc85/DbVokvL8Gcv51tnyO/F5US8KZ9lP/lfv188P3D2yNU7FbHN6fGwYaOi64YBL/J2ec/TNiwyEK/SA4FxyF2jez2Y/4NZbfv2H6+if2eMdUf/wcH6FQ+zO+/fjkI3zx9Xff//Hf/vqHP/7w0xcf//LDX396fv1fz9/9kvRUDFl81jHS9/ifoifM2edxKovxoT4Hfb4PPfd9uTV3uP4/2Iv8n8J3wWJ50pwtxc+DH26Ez+MPtsHk9RG/5/8wfgV+LiYzWMje5X8Ov7BbqNwMMvna3r7b/wNo4Ti9Psh5/DPDp+H7H//T4RPW8ySfn184zz1Rk6OPLrZRApauzJhkMVTeRKUmkwoPk46rGO9Hkuk5dLqpz0qBfbC2lMVe9Vee90LAc+d1pbAfR6a319ENGg4FZ13VdF6ViKs3sXqSZjLcQEdFYJ62GmXB3Ikrz1mJ1WDRg0ojmkhv77NQYJ+2/GR0yovrrPBJRldrSIOaTG/vs1Bgn9iWDXEzz3shVRbxqX26KsxEensd3YBx2uqSFbMhrjpnBU+TECiVwUyEt9dZIDBPW00yYfbDleesYFEMH6mt4SPQRHp7n4UC+2D5SMxAm33OCka48dVnFWoivb3PQoF92nqRFfMbrj5nBX3BnH27eC7URHp7n4UC+7QFIhNde9RvosK9HB50hOP5cx5Jprf3WSh8fBSsCFk8zeOKXuH+juPT9XVQk+ndfVYKzYeHm0Rz+9MrCSu++5hHNZlu4KMqsE9b8zHXeWjRK+jxZO/qqCbT2/ssFNgHqzxSqPPYolf4NWVtKmFE6+HtdRYGrMMnEK2tSutzVvh9nGqybaeOfiSZ3t5nocA+bSHHory43gspGkp8ojSaifT2OroB47R1G2OZRxa9wo2w5YO7OpiJ8PY6CwTmwUKNHCktz1nJFle/XLu+048k09v7LBTYp63MWMt8VaNXuDeIj0pdGNVEenufhQL7tKUYU55HFr3CY9AcSslhUJPp7X0WCuyDtRejz0rzc1awuSfezcNV7Uxv77NQaD4UI+X5ykav4A6gdFzC6AcS4QY6qsHHR21rK+Jj4atOr/BYvVRbKIxmIr27z0qBfbCYYg5p7vr0SrvOXKh9Si/URHp7n4UC+2D1RP7B+bpGr+SE++tiDKOaSG/vs1Bgn7ZcYknzyKJXsMGKi671koWaSG/vs1Bgn7ZCYpw7h2cBM5R9PZbZ78eR6e11dAPGwYqI5KLS+JwVbmawNG5bzeE8kAxvr7NAYJ62BqL65DkrPBb1lIqjAU2mt/dZKLBPW/QwxXlk0Sv4nMLb2taf7keS6e19Fgrsg3UOscvCxPNeKFhzjciPZjK9vY5u0HAoOcx1UnReFX6a5MDtsZdmMtxAR0VgnraOIZ/ozHNWeABRC8VcRzSR3t5nofDxQVi4MARlIlSv8Agichew3bws1ER6d5+VAvtgpcJo/XxVo1eKM8FXT35Qk+ntfRYK7NOWJizTDMQ3UcnRFDTJaVCT6e19Fgrs09YijMqbV6/gmRJz++imH0iEt9dZGLAOlh7MHM46Z6Xw+1SyttBoJtLb+ywU2KetNViViZi9knGdueA6s1ST6e19Fgrs0xYXTMpMzF7hXmCgVNu8lX4kmd7eZ6HAPlhNsHo3X9XoFR6he+4GUh3VRHp7n4VC88GTQGl+3guY32P5fTwMZjLdQEczYJy2XmBWZmL2SsX0FRcjSTMZ3l5ngcA8ZLDVq1XanrPS+jgutfUg+pFkenufhcLHh7P8zVjgYR5aiFLBptU1tvUV+7FkenehpQOI2lKAWZmNKUqMUXzwoU5yZ3x/owUEjNoCgIHmZkiUuLdcashtRVNJJ+L7Gy0gYIRl/4KleZQhSrgUn2os9UIn4vsbLSBg1Nb6K8qbvSjxE8b54tqgXdKJ+P5GCwgYtQX+YtXao7OEy6rZ2ramvqQT8f2NFhAwwqp+ydX5iocoVW9sCSG4kU7G9zdaQBxGlFNV5mmKUsEufi4lGulkvIORCgGjtoJfqlp7dJZKMtkeq09LuZ7eX2jBACGs2le8Mp1VlEowiYdi9gon4vsbLSBg1NbqI2XWpihhModPNdWRTsb3N1pAYPGjtkAfrqVORr2UEw/uyWU30sn49kYrCBhhXT7tU8Re4ZdU5TO2NMLJ+P5COgOAvAmF21xN6CyVzK1O9EfXqMOJ9P5CCwcQtfX3ijKDU5RqMCVSIhrlZHx/owUEjNrahTz0UozOEg/tLbZY9SOdjO9vtICAEbe5xWP5stnoLOEDIIuN4y50Ir6/0QLiMKLiqzKfU5TQT+Tx66s56nQi3sFIhYBRNrWElLT26Cxl4k4QD2PTSCfj+xstII6FIUuJPimjkF5qd0LFtkuJOJZI7y+0YIBQNZnfmpTJnaLEzQ72KLHpAifi+xstIGBEWDMqR601OksYboR4XBLpxxLp/YUWDCzk+UdL5hOdhXoJHcUc2z2q4lgivb3QigFCjvuCRVnYpFcq/3SOqS0hIA4l4/sL6QwA8twRLCUoDVEv8WiM/79r65P1Y8n0/kILBxAFrK8Vg3I1pJdwaxT/87juKOREfH+jBQSMIvcDySkzPkWJW2XrmeNCJ+P7Gy0gDiMqpF2+Pivc+7GB38LCCCfjHYQ0BgAlU6tNXrka0ksYr9ZiQxjhRHp/oYUDiLIpldsX7Q3tLHHsXQ5Hv1HIifj+RgsIGBWTqyNl/qcoYSqaa7dISbgzvL/PAgE+1aTqs1OuhPQSxy5iWfoLm4jvb7SAgBEWZg/BKWOPXiq4KYhHZOVCJ+L7Gy0g2ChYE2q0yjxQUWIM7kTbkia6M7690QoCRvw311iUZTlFCRPTKg/t84VOxPc3WkDAyBtXU7TKEKSXKjpDORxv+/1gMr6/0QICRoH7N9kp80JFicf0xP1q60c6Gd/faAFxGGFFE6u1R2cJC+gUi00ornRnvIORCgGjyL1B7XP990LFTk22pjKyyfj+PioCdBL3BLmjowxAeqngYyEf/MAmw/v7LBgglPk9qeIOl1noLGGhhppsu81DHEzG9zdaQMCoGNwDRMq1kF4quBJL/vU6Ow8m4/sbLSBgVE0k67U5ob1UEg/HYi1XOhHf32gBASMygfjlorVGZwkfTcdEIV3oRHx/owUEG7WtZB13k2ejXsKtQ6UcGymJg8n49kYrCBg5w2P3oM0J7SWOI7+btduHxMFkfH+jBQSMvLEUcOfvbHSWajXVY23TkU7G9zdaQBxGRFhSUTV6lXhAxm/36ZgVKuhEvIORCgGjYCrxDytjkF7CUjvO15wvdCK+v9ECAkb8XCBcHpuJ3ivVGX6nj/ni1tP7++gI4En8zalm5XJIL/FzxVk+ZhjZRHp/oYUDiDI/EXLKyiikl/jZgo0T2lIgUk7E9zdaQMCocF+weG1OaC9hSGZzbttLSDoR399oAQGjyn3BomzL0Svk+elC/ug79kPJ+P5COgOACIfJSWurzxK3OpQCRvgSTqb3F1o4MFGy3A/kgZYyAumlym/ylnK+yon49kYrCBhhlxZrk9IW9VLFZp0pHG9pgk7E9zdaQMCI+8rWlqiMQHqpLRDL47J6oRPx/Y0WEIcRf3PUZoT2EjpBIZVjSqigE/EORioEjLi7bD3XFKOzVBM2xq3H/HRBJ+L7Gy0gYMQdZutrVK6I9FLFYufHeipSrqf3F1owQAg3A4ekTQvtJSyp6xwld4ET8f2NFhAwOu7AC1prdJYYw2ESX57ozvj+RgsIGHGP2UYKyvWQXqrYytbXYxQi6ER8f6MFBIy4y2xT1ojeKzUazo6bZIVbT+/voyOAh7vLNgdtVmgv8esphlg8jWwivb/QwoGJsJO45dZXuR7SS9yRrrgrNlzkRHx7oxUEjBz3BEvxyhikl/gVlbIvxxV+QSfi+xstIA4jsvwWroxBegkY/OPHxdmR7j3ewUiFgJHn1ws57fJ1L5HFdJDcFgIVB5Px/Y0WEDAK3Bek6pQxSC9xw8Nv7q87ZvrBZHx/owUEjKLBzDNtZmgv8aA1+ZyPe/YFnYjvb7SAgFEynHurtUdnqfK7vY+e3IVOxPc3WkDAKJvIoy2rjEJ6CR9Rl3zcsj/Ivaf3F1owQIjPxvmszQvtJcK065qP6/z9YDK+v9ECAkYVu20GZYd7UapkUkzR+5FOxvc3WkDAiLDjJpFyTaSX+O0rlHqsrSbkenp/oQUDC2HfcRezNju0l7jV4X6ibRtzSjgR395oBXEYkUtB2etelPgNPtbC/ekLnYh3MFIhYORM5eGX8rZ/VohPi0pOFzgZ319IZwCQN4VbYG1qaC9hXwssFUYjnEjvL7RwABGfgitR2fNelHjQ6ip/XUY5Gd/faAEBIx6z84lW5ZpIL7WLaCXVK52I72+0gIARn6erVZsZ2kv8oiopvjrY/WAyvr/RAgJGmfuClLSZob3Eb/A+WvJppJPx/Y0WEDAqxnvri3JNpJf4CWNzrseK8/1gMr6/0QICRtVgYFqUUUgvceOcC72urfWDyfj+RgsIGJGxnkeoyiikl7BdZUmvuxwFnYjvb7SAOIzI+5C19ugsccPDw9Zjq0pJJ+IdjFQINsJe5T5Y7WPHXqJsuBdUvB/pZHx7oxUEjLCWddDe1s4KoUPkq0sXOBHfX0hnAJA32cc4bQ3/Jkts4Zy1x1YzEu5M7y+0cABRMFgMJCnjkF7CwL547NA4yMn4/kYLCBhFE7FmvDIO6aVK/FYfqG2PJulEfH+jBQSMkgk+q0+j9wrxALa+Nj8Vh5Lx/YV0BgBlfiIUH5VrIr1EzrgQYrjAifT+QgsHEBXsOETa3NBeosQttHW5XuREfH+jBQSMKvcEa552jH+TJW50cqihppFOxvc3WkAcRuQpBOWaSC+BhWJw9UIn4h2MVAgYkamB+zda5/Es4V+cHyv1CToR399oAcFG2Nc8YO+q2aiX2kewPtQLnYxvb7SCgJHjU3BYy3o2OkuYlc5N87HDXD+YjO9vtICAkTcp4N5pxegsMUtNruQ00sn4/kYLCBgFE4Ov2hzRXuKBfeW+tb/I9fT+QgsGCEUTQkhea43OEvEp+2RrucCJ+P5GCwgYJeND1CbRnhXscBlTeHUg+7GG/P5GOgSIcKNZJG2CaC8xC/elj1Uxz0OJ8P4+CwUAFWNDytPO8m+yRFh6rtIxy1i4ifj+RguIw4j4z3faC+0scQtdjylYI52IdzBSIWBUuS9YrDZNpJcYg19gdNzGN9Cd8f2NFhAwonalftpp/k2W0DJ7f0yC6McS6f2FFgwfHx57n2PB3fmaiCi1XZsT/6Iwy5353ZWWFFBy3Bkkp8wPFSXsI19qzm118EFP5PdXWlBAyXN/kMq04fybLPGzJ/ucrBvxZHx/owUEjALOJiqLrogSg5jCL65cRrshv7/SggJK7dZOp8wSFSW8rkLk11i46on8/koLCigl46JTFhHtFTxjeJhf6pWux/cX0hkAlI3Fp0DzhRFRwtPFuWDbFCwpJ+L7Gy0kDiTCMH4ej4gS3ukp1JSudj3ewUiFgFEx3JyQMlNUlAhrYzg+3ZFOxvc3WkDAqJoSozIxq1faa4pHZn6Ek/H9hXQGAOGLxD1nRegswaJihd5Z7ozvb7SQYCTsgR75W5XxSC9BgyqV5Ge8M7+90ooCSvz2HTNm6c1KZ4mwFl2t9YIn4/sbLSBg5PlsSsxK96iXGMOF2vYqGuXe0/sLLRggFLg/WJ0yT1SU0FXM/KJqn1iLow35/ZUWFFCK3COsddp4/k2W0PJ4bqjb/XuDnsjvr7SggFLiPiGlrFwh6SV+XaHbWN2IJ+P7Gy0gDiNKPKBQuki91J4xPHg9RrX9aEO+g5JKAaVssE3KtPn8myzhGmROsbgRT8b3N1pAwIhPKbmclOsjveSsNzmH1O6bEUcb8vsrLSigVE1O+KkZ6b3SrvJnPuN8tRP5/Y10CBCRweqy0/7zb7LUXlU2tZXW+rFken+hhQMTYU90Hl1E5WnUS+gNJRezp5FuyG+vtKKAkjMhxajMGxWlppGKa/eFXPXe8/srLSig5Lmnk5wyc1SU0D7HHG1bL2PQE/n9lRYUUArGpVSDcqGkl5zN+Ozo6HBLvB7f32gBAaNobOJ3KmVU0kvOFsPD2lj91U7k91daUBxKlIpXZo+KEr+R8QiNohvxZLyDkQoBo8Qdw0Jea5POEkYf3AiFVEa7Ib+/0oICSpm7hhX9wVnpLKHXWPgpc/QmpZ7I76+0oIASdiugoMwhFSW8jxHVdLU70/sLLRggVE3K1k4b07/JUnsTizyQzVc5kd9faUEBJbS9tjhlWNJL6DUSv9PTFa/H9zdaQLARdkrPLirzSEXpGMfyCCRc7UR+e6UVBZT4b87c11FapF46LkAS0Ygn4/sbLSBg5I3LXtkgs1ec5X5RdceQpB9KxvcX0hkAhAVoQlKmkYoSLFwp9DKScmd8f6OFxIFEx117GtKrBA3uFMVj2DbinfkOSioFlCLubSSrXCfpJWerCYEbnnLVE/n9lRYUUEq4jp+0maS9xANYzqtrO6pLPZnfX2lBAaVscs5em0vaSxjrO19KyBc9md9faUEBpcI9w0ykXCnppTa/hvh9v1z1RH5/pQUFlCp3DUsmrRNwlpzFkhH8G+iqJ/L7Ky0ooETcN6xBm1DaSxil4Tptu6d20BP5/ZUWFKyE/dP526vSLvUS3vFjLdT2Nxr0RH57pRUFlBz3EKlUZWTSS2ili+PRWrrqifz+SgsKKHlji/r55FlpFtXScdXtYvee399IhziIiJ8LRTd6ldB15ANaf7UT8Q5GqgSQgqnF1aKMTnoJGsVR8Qremd9faUEBpYhV6JM2r7SX2rUj7hMdl0uknsjvr7SggFIyuWg7svVKe68vgUq62on8/kY6BIiySXyaWTM6S7jk7yi8t0ndTsT3N1pIAKkYzKXVZpb2knO4SBKPKRPCTsT3N1pAwKiaUFKYtq9/kyVnyVRbXbtbW9rJ/P5KCwookfGYjKVcL+kl5xJ2GUvHTVxCT+b3V1pQsBJ2VS+5aDNLewkvrJSinfVEfnulFQWUHPcNS5w2sn+TJWjwsY81kC96Z35/pQXFocSDDP7TVaVXqbU/iY6dj6SezHdQUimg5LlzWKs2fOslaGBuxHFbwKh35vdXWlBAKXD3kJI2w7SXnOOBrY32mDch9GR+f6UFBZSiyZU7hcr1kl5yuEGJ+9ptla1BT+T3V1pQQCmZVK12sfus4PMRfidLbV+WwU7k9zfSIUCEbcRc1qaX9pJzZFz0tS2sLe1EfH+jhQSQSltfNGht0llyuKkEH4+EK57I76+0oIBSNb4G3Lk+K50laPAghIqid+b3V1pQQIlfMzUUbYZpLzlvcX/b8dGSwBPx/Y0WEGyULT8XYtQuKvUS3sUqFdxYOtrJ/PZKK4pDiWpyXnl76yWH27iiD37SE/kOSioFlBy3vqlqc0x7qWnwM+b4/O2i957fX2lBASXsC5HTtMn9myy18SxXj/6k1BP5/ZUWFFDiU6jFO+WaSS/hnQy/7PVcEnoiv7/SggJK3HPGZ9bK2KSX8MqiGlPbo23QE/n9lRYUUEoGN/xPW92/yRKuRtpMdNyzJPVEfn+lBQWUMncQKVjlmkkvccXYQtG7i57M76+0oIBS4eNYq8027aU2WiNcTrroyfz+SgsKKFU+T1u02aa9hOdMCvF1f7fQk/n9lRYUUCJjyWl9ylcBLbTjI7f1kqWczO8vpDIcPkTYQlMVepXwmoo8rD2W5OhyIt1BSIVgo4J5otgGaTbqJbyifKme/MVO5rdXWlFAyZlCIZLSHvUSnjPcZXTHM0noyfz+SgsKKHn+bn1h115Cf6jkmLK/6on8/koLCigFkwjdHkXpLKGF5ldWbTuTDHoiv7/SggJK0URK2Kd+VjpLeGVhftLruST0RH5/pQUFlJIJpN8C10u4FEnFuuMTOKEn8/srLSiglPk4mFurKJ0lPGeI0mslHKEn8/srLSighB3+Sla2TRYlPGf4YFTooifz+ystKKBUuYNYQ1GumPQSNBwPRY4Fg0e9M7+/0oLiUCIi7b7Ts4LXlSX/Wp9T2Ml8ByMNAkTE/UMq2rrTvdQ62DxMKxc7Gd/faCHBSNiZ3dqYlaslveQcu5REr15Ax5P57ZVWFFDiTrR1Tptr2kvOex7zU/Lloifz+ystKKDEnWju7Chvb2cF7XOuJcx2Ir+/kQ4BIu5BW5+S0m73kvOZ82SPi0rCTsT3N1pIAIk70DZ4bZ5pLzmfDLmS8oQn8vsrLSigxB1oPoo2z7SX0Pb4TOG4x1vqifz+SgsKKHEHmlvhqFwx6SWHhZS4P5TdVU/k91daUECJO9DcaY5a032W0P5wWx1o0hP5/ZUWFFDiLrTNVptn2ksueJM8N0X1oifz+ystKA4lstxx1tqls+SKMzmXeqzQIfRkvoOSSgEl3H9UYlAGJ73kUsbSJfG4V1DoyfxU+vI3Hqfun799MgjOyj2/eX759Xf//Zdvv/v9N189v/3pcZH8eASOxz3pSN4ejz88gnUmHLjye0V8/YnpuOoh+reKn/+bx/3yN649nHxOFm/r+eCevojv8u0hxQP6/Gb4psff+onHIySPBYf4/4azyDYN5/X2SoY/U/zskPMD86f2wPRvuPyCv/1Ebo+mw6P5/P7pHtrDyT+FDbewJMKA3+MrqHiiWB6VPP98HL29BPjLb17p48eHY8MvLJdCxg4xfDSbo01PwlxePty3Hx9ffXh8+a+4gPf88KeH5e//8J+Pf3/+yv76+R/PD799/MuHx+9+YYDI43QeN6RA47O6x58AIGLZIByt4v7cnwGQ42cU4LYjBO/y9XV9xp9CIGP3Wz7asRLH3xdwvn4+ghSyweU/SgOBiD8BQfK4P5uPFin8LAHyn1GA+BnqXfRhFOjxpxDg4YNrR7MBWyH9XQKf8ucjEK1wxLtjidzTem/Be/IPN4Ium9T/8MidOVstPpjP9UXAJ85/AdY74NPvHr9yv35++P6R2xq22BM5+0MI7SQX2t2jKWWs/+Of0XhnKfSD4N0SR8ntmxnyR/waW7jz8bTv/8SOOdaHiJ/jIxRqf8a3H598hC++/u77P/7bX//wxx9++uLjX37460/Pr//r+bvP+XiUZHB3bOmPR08+0ePB3UZ+w2OD5IJ8PPhpj+foL/GApJ//gIRKDfV/ywNybM6ZIr+ZvT8iIvpEDwmRSSU7z3z2Mz0k8cYPScAWDpSceEh69IkeEn5XNs7xC89x+/B5mq1w32bLZW5UbOBuRH9IevSpHpKEi7r8gDhf4ud5SPz/pocEo/XHzx5T/73R+mP4/v/ZaJ3/PJPIA/ULXwO2DovVOv8aT3G3z2PpiKH0Jktts1obQzzy82hDfo7WN/BaoMALkzsC4eL+5HWWAp6hIaQLo4x30lqQQAuTPLB4v6J1lrCSBl6XcUSU8U5aCxJoYbIHmj9F6yyliri4eEEU8U5aCxJoYdpHtlFruc5S5l9mrT+uZvWDyXgnrQXJoUXEA8Wsar1K2Zvqiw3hgijivbRUEmhhIkjBvqOz1lkqETve5hxGRBnvpLUgYa1iTaHqlEb+rPCw2JacS7gQingjqwUIqNoG0hUfjkxWZ4l/PPtyrAEkCEW6k9VCBFieO5posBWss1Rw7xb3R8vFUMQ7aS1IoIWJIjzAuXby32SJ3wIr2dpuuxUHk/FOWgsSaGHCCA8iFaz3Cv8D9yq1iZLiUDLeyUoHARVmjbjslSFPL2F+abbHfBJBKNKdrBYiwMLkkbbexox1ljB3AqsGjYQi3clqAQIrTCEJ3A1XrM4S/4tfbvwLRkIZ76S1IIEWppLwO5wy3umlyl31SNaWEVHGO2ktSA4tsrj6qGq9SjUbm2L09YIo4r20VBJoYWJOcmoX/iwxC/8ruTQhnvFOWgsS1qrcJecfdMp1ml7CjFR+9R0jHoEo4o20ViTQ4j65zckqQ55e4icRxdfwUBj2dCerBQis8OF78VZptXqJ+Jdh4/g4Esp4J60FCbQCdzMLWaUT30vMgpkMbdGqEfGMd9JakEArckezas+sV4EwcT7VmC6AIt5JSuWAU+I+JvGBFKmzRNi4odgYBkAR7iS1AIFV5h4mESlXaHqJn0Al+1rThVDEO2ktSKBVjHU2kzLe6SVMVwkuHCsSi6MN+U5eC5TDi5wLVRnx9FLbQz07/nVXR5Hv5aWiwKua6rytyoWaXmqNVM10YZTxTloLEmiRKc7jO2ets4TtsG3xdDHs6U5WCxC24t5SdiFWZcTTS3jFpUDHp4f9YDLeSGtFAi1nkouuKO1WL4GlWutrnhXPfCevBQq8PI+JI1aUm73OEhVTQ6YwKop0J6sFCKzQC0ipKGOeXnIOc2dcDXk0HPKdvBYo8MKcq+yL1nKdJfStbLDHPfrSUeY7eS1Q4JWMc5myMvrppbaDJh/9uCgvHUW+k9cCBV6ZO5wlZ2UE1Et43XlvXbg6ynwnrwXK4UVogLT26yy1/ZAp2LYKq3SU+V5eKgq8+A3OkXZB8KzgVRf5P56uiiLfSUsnAdaxZa02ObeXsOVP9hHraw+KMt5Ja2ECLuIztjFpbddZwnr2HvsA5SujyHfyWqB8fETLv9c7l+brN6KEfUnIJe7LXx1Fvo/XEgVezkTubSozdEWp7QiEKSLp6ijynbwWKPDyJniflDm6ooSVybjFIpocRb6T1wIFXoF7T8HH+QqOKGFtYAqV2qeIg6PId/JaoMALdzlzd30eC4kS7qTjdj37fHGU+U5eCxR4JWN9zMpkXVFCvyHGZGu8OMp8J68FyuFFfB5Ba7/OEtqpyH2IV3svHEW+l5eKAq9sqs+4r2L2OkttX0E+x1f7JRxFvpPXAgVexRSflUtfvYIVYGtJ5K6IPd7JSgcBVcXNApjCNludJSxvRg53TI+GMt5Ja2ECLuJuZ7t5f+Y6S1hZsLiCO6IujCLfyWuBwl6Of9TzzykjoV7CepWOfKUrY4830lqRQMtxp5O0YeNZwXOIuHd6PLekoch30tJJgOWND9a7+RqOKKEzWrKL7YMMqSjinbQWJuAKxoVjC6OJ6yyhRfeltEWCBsUe76S1IIFWNDZgRQpF6yzhWRRqjZmuiiLfyWuBcnhR4GeLcgWnl9raK/z2ZxXHM9/LS0WBVzI1BKtcrhclvOxq4n+Vq6PId/JaoMArmxICP0cUr7PU+lfluIlsYOzxTloLEmgVPoUYrXL9ppe4sTKRn1DHEEgoynwnrwUKvKpJIVllJq8ooZWqlQKVi6PMd/JaoMAL+xamoszmFSV031OKOZWro8h38lqgYClWi8V9o3ITsSjhdZdKIEsXR5lv5LVCOZau9eoNjL3iPBnrfXb5oijznbR0EmB57neWqkzmFSXnownOllAuiiLeSWthAi7M1KqpKm1XL6GHFbFMS70yinwnrwXK4cXvbb4qI6FecsEZPmJ9vRq7o8z38lJR4BW530mkzOkVJbRSyeNOz4ujzHfyWqDAK5nCZ1O05uss4T2QSnqNHIWjzHfyWqDAC4tlcnOkjIR6yWEFQSybGq+OIt/Ja4ECr4KV+60yr1eU0E65EnikeHUU+U5eCxR4YS6gL1lrv84S1uUlRyHXq6PId/JaoMCLDNaezMqVnF6CiysltHUEL45nvpPXAoW9uMX2WA5CGQv10uFSXxeir451uyvRKxR4cQsUI65ezV5nqfVLg7dH9144ynwnrwUKvPjcYsIeU7PXWcKuZjG3xZQujiLfyWuBcnhR5JNRxkO91N4HU8o2XhxlvpeXigKvYPDJjvYhbS+5kDHCRpfr4ijynbwWKPCK3Pksym0vveL4x2Phvmm6Kop8Jy2dBFiJe541RGUw1EsY9Hg+4Kux74oi3klrYQKuzB1PrC6lcJ2l1icN9TW9RDKKfCevBQq8+N0tUola23WWeODDJ2q9TVdHke/ktUCBVzUh2RiUsVAvoZXywbq2l/zgKPKdvBYo8CLjk3PavN5ewntgsjkeY0fpKPKdvBYo7BWxN4rj1mj26iVcs8EV59frUTiKfCOvFQq8nLGJT0QZC/WSw1MquEr56ijynbwWKIcXpeC1yb291PqkGftEXhh7vJeWSgItbyqfr1dar15y/AKMMfl4RezxTloLEmgFUxKPapTrOL2E11yqhYK7Kop8J68FCryiySkpi673Cp5EyXOLTldFke+kpZMAC1NEsnVaw3WW8CQim0qoF0UR76S1MAFXNjHl4pRBdi+hv5BKoTwxinwnrwUKvPhsUona7N5echgq1hhsvjqKfCevBQq8Knc6tWWMewWjneqYKF4VRb6Tlk4CLOIeZ61WuYbTS+hdEY97qrsoingnrYUJcyXLPU6c88zVS4678SUl+3pn7Iwy38hrhXJ48Xd6bW5vL/ELjn8bDnF1FPleXioKvJyp2VZSruH0Ep5HgYo9vU5Hke/ktUCBF1bbdYmUUVAv4VogH9K2NV2ko8x38lqgwItPIXuvze3tJTyPfKF6fN4oHGW+k9cCBV7c28yeSGu/zpKL7X2wtPUaB0eR7+S1QIFXMpH/+qoMhXrJ8dCaX3U1hKujyHfyWqDAKxv+24M2v7eX4GJDIltmxzPfyWuBAq+Cy3y2au3XWWpj6sqHqRdHme/ktUCBVzUup6K9HN8raKVSKv64U0goynwnLZ0EWMSnlqM2ubeXjkY9l+M2x0HxjHfSWpgcXP9fd3eTK0tvmwF4fldxVyCL+tcwRgADmdlZguFBBh4FyPrDt3Ruka0mN8DZ55d9+7geVKsltUraY9K02q63hD485dF++qqKUeWxvEwU9hqZO55zTWMeR0rHpfw8y3E7lnDPcngo8CLueK5ure2VErWVJqd93o4qj+TloMCrcMdz8y1ieL0l3Ee5rNLm7ajySF4OCrwqdzz3HsZcjpSozURlzjOvqh1VHsnLQYFXS/wvh7W2V0rP545qPc8KaUeVR/JyUODVU+VLttb2SgmfO36/n3l77ajySF4OCrywbUvJ3ZjLkRJc+sp59m/HN4/k5aDAi1ugWWY3xkJSosZdiL3npNtR5ZG8HBR4rZQnlud+c/2poJXiBguHSlyKKo+kZZMcrD1xvoap9VMCS6NKNL8U3ziWlmkCLmzE2JZ5c70ljBEx8/yc3fXBqPJIXg4Ke82c5uzdWtsrJVzZYIJGl6POA3l5KPDiK57DOH5dKsSXyZ+/s9ZLI0ocycoGAVVJfY5djWGQlB6Vzp++9m34J46k5ZiAC0+tzGGt6pUSDWxZssaZsVeMOo/k5aDAq3GXc1VraZyU+E8kfAFSvRx1HsnLQYFXx2+GuRizOFKi3tLs46ftUowSR9JySKA1uMu5p7WqV0q4MrDsdSuqPJKXgwKvmfLKzfxifEvPpw6TgXQ7qjySl4NyvPYiXLHl9VOCy6a6zzMJn45vHsvLRIEXdsujZS3tldLjwt33szfh5fgnj+TloMBrp7lKt9b2SolGS/JxFEYVR9JySFhr8atXLWTM4EiJ+k7UyjibQitFnQfy8lDgRamvai3GeSv4zA3C0xq3osojadkkwCqJR8vDWtkrJRr8n3udnV4EUaWRrBwRYNVUV6/ZaLekhNEODw7z6DeiyiN5OSjwaqmskbMxCpISbqPacC7V7ajySF4OCrx6ojWGta5XSoTpearlnImjHVUeyctBgdfgTufELsXfXm8Jn7u815hfjiqP5OWgHK+9cGyl6fVTenoMtc1zAoBy1HksLxMFXpN7ndwZsLoSbwkd0jLyz1NCylHnkbwcFHgt7nby/7Tar7dEWA03qYxxOeo8kpeDAq/9vHoZ8zhSotET/6l9ziRUjjqP5OWgsNfmv7vzstb1SolGTZSxA+HtqPJAXh4KvIg76Py5MtovKeF7cBLNM3LUjiqP5OWgwIt7nLsUayG0lNBOLRbKt6POI3k5KPCq/D5lW2t7pYR+fOFm6stR55G8HBR4tYSNZK1DHKWEdn330c9oWzuqPJKXgwIv/obbzfqN9q3Qs55k0Fm5pBVVHknLJjlYe/dsLeyV0jOorjTP4hKlqOJYWqYJuEZau09rXa+UnsnmnfdZZ68ZVR7Jy0GB1+QrHm0YczlSokGJm6hRvhxVHsnLQYHX4ldPHjQbXm+JP3Gp9N7Pb7TKUeeRvBwUeG3ueM5lreuVEua4eh3rrJhQjjqP5OWg/PtXzxmT771/t1+qhG9Bqqufffa0o8rjeLko8KKEwXL/HgupEvWR5uhn/0HtqPNIXg4KvJ7e+TaW9qoS4TFj2pvodlR5JC8HBV7cUc/Y6tPwekvoZ3XCrxqXo84jeTko8GqYjanteyykSvQ8elbXMwfx4ajySF4OCry4r56LsaWQVAiLnymv2/BNI0nZHAdq54Kt8iypnxImA+vI/TlaXAuqOJaWaQIu7qRnvk++53BUiQZ3TXNrtG5GlUfyclDg9VwxGSt7VekZH/Z65mo+HFUeyctBgRd30rFbi8H1p0JYWT/m2utS1HkkLZsEWNxDz72X7zkcVXq+/Gqfo38q6jiSlmPCXMT/lC/GWNerShgb0lrjeVrjg1Hlgbw8FHhxBz2PbazrVaXnO3A9p1F9KL5pJCsHBFbcOccE8vf8jSqhPe+7tOfZgw9DlUfyclDghV9XOTPGP1LCbxiEE+y/HFUeyctBgVfjPie/3PhilBLGhXs0etbcfDiqPJKXg3K8MOFubNirSvjcrT97zX44qjyWl4kCr45RX8vGMEhK+A5c/AHc7XZUeSQvBwVeA6sdyFjbq0roY1HeddPtqPJIXg4KvLBSnviNDK+3hHadfWis21HlkbwcFHitxF941t31U4BJHWM8+zRehm8eycoEgdR+9oUwVvWqElr0Vha1T0IJI0k5IGxVcqpU1zbuKyk9Tx4s/q9+G6o8kJeHAi/+/0yNu+aG11vCSuc26VmIqhlVHEnLIYFWSUS9GOt5VQkshbDX/7fim0fyclDgVVPmy15GyyWlZzeEtffzjJl21HkkLwfleG3+PC1j/kZK+P6bvc+1b0eVx/IyUeDFoxia1VjPq0roi5Y2RqXbUeWRvBwUePEdwn2oabVfbwkzW6tjA/bbUeWRvBwUeA3udK45jTkcKWHuYa5Cz47PH44qj+TloMBrcr9zN2M9ryphexLCqq52Oeo8kpeDAq/F3fNM1tfjnwr/gVTL+vlhViuqPJKWTQKszRec1zAmcKSEhUmj79bGpajiSFqOCXPhpHB+C+NQdlVCn7TPTmc9iWZUeSAvDwVelKiUYqznVaXnPqI5+rwdVR7Jy0GBF19bKcYRQlLBdfH7PZvxfiBKHMnKBjlUu1TcHJbVTwkNOvcVzo/YilDSWFamCLC4l1latdbxSgnrKfntZrsNJY6k5ZBAq6VZeu5Wq/WW6DkQoZ3TUj8UVR7Jy0GBF98gpc9mjICkhCvrmYc97XZUeSQvBwVeI3EXvVmreKWEK6uNoQzHN4/k5aDAa3KPc2In1G+vt4SRTiU6AyDNKHEkLYcEWos7nPxCY/5GStgXu4xcxq2o80heDgq8Nnc4V7eW8krp7PPM4+h1Oeo8kpeDwl4tc58TfYNvLynhPlq99zPfpRx1HsjLQ4EXd8zL3tWYv5ES4ejU1dqzb4R21HkkLwfleO3KV2dynQpUZqvtOWHjUnzzWFoWCbBK4qRaa3ml9Bx4M7DX86eijiNpOSbgqmnWkq0VElLC2bJ5teeAjQ9FiSNpOSTQanwJZRZjHCQlnBixWl+134oqj+TloMCrJx4INms1r5SIiXob9Jy8/uGo8kheDgq8RuLGmqypeimhTa9rbaqXo84jeTko8OKrqQ0rj7693hJO4uVL3mtdjjqP5OWgwIvHMrV3azWvlNCsE/cfzi/Z2lHlkbwcFHhh78VRstV+vSUeJabF99Hut6PKI3k5KOzFLVCuA3PuX15Swn00eh1n3ks7qjyQl4dyvHadw1rNKyWqPZXWfyZylKPOY3mZKPAi7nuumo32S0rUnhOomOZy1HkkLwcFXoU7n2tvYyZHSlR32s9ZepejziN5OSjwwiXsYa3rlRJOs2yllOe0VO2o80heDgq8GvfQ+W2s9ustcfOeqO2ztEQxqjiSlkMCLb4/GlmdibdC+JWRqJ+VEtpQ5ZG0bBJgDb4YmtaiXimhCz83j6nnp6KOI2k5JuCaCc+9Lqvlekto0bl7ReeJBMWo80heDgq8Fjc+lZYxkyMl9LCIxqr7dlR5JC8HBV475VaXtaZXSlRxQiO/5ZejyiN5OSjHa7fWrTW9UqLKDTxRGfV2VHksLxOFvUbG3GiZxkyOlNBl2KX8fBwVo8SBtDwSaFGare9pjIOkBJZKc555r0/FN4/k5aDAq6TRxrAW9UrpaaXK4O/F21HlkbwcFHhh5n3WYbReUsK34K61ZLodVR7Jy0GBV+N/usyfaKWEVr0M2pNuR5VH8nJQ4NX5apb15fhWMOKZc9Iet6LKI2nZJMAa3O/EaMbQekuEVeKTSt+XooojaTkm4JqJesZY5pvrLVHB49g4ofFi1HkkLwcFXov75nlZa3ulxH3RxE36OVXpw1HlkbwclOO1u317/alAJW/qZX8rvnksLYsEWDvxhZdmzOJICR2stip/HX4q6jiSlmPCXDOnyXeHta5XSsTXPPgbcI2LUeeBvDwUeNHzi2Ez2i4p4VO3uEO6+u2o8kheDgq8Suq91WrM4kiJysJpvAxxO6o8kpeDAi9ugXrP1speKdGz+QaOMr4dVR7Jy0GBV0u18wuMcZCUCJtJzF3OjtnaUeWRvBwUePWEvlM15nGkhPuIBtXzI4d2VHkkLwcFXoM7njycMYZCUnpGiNxxWO1y1HkkLwcFXpO7njiS3vB6S4RNz7Az1b4cdR7Jy0E5Xrtzd8pqv94Scb7wlThvR5XH8jJR4IVnV7a1HPqtUN5p5tLnhajiSFY2CKj4evve1tJeKRHlRLuNcxCOGOo4kpZjwlyLXz0ydze/uaRERKnVfc5K/WBUeSAvDwVeOOmNKhkfRSkRtn6ufZ+VS9pR5ZG8HBR4ldRGydZiHCnhPsrYVo9uR5VH8nJQ4IUTWsrMRvMlJcoNT1IR7ctR55G8HBR44Tcdvh5jJkdKlDHiKfm0X9pR5ZG8HBR4dezKRdbKXintnvg92tkhVN5Mx5G0HBJoDb62NrfVer0l/pc49Wad3xvl3T7ySF4OyvHaozdrGyEp8W00uIn6abzeN9NxLC2TBFozrTHIWtUrpb3RJd17X4gqjqTlkEALzwwPa7Pet0I5p1rKPL8HyXt95JG0bBJgbRwtxS8wtN4Sf+Iqf/LOzrPve+k0kpUjwlib/+5YxVrTKyUeUq/Kb10/DXUcSMsjgRaP+fg+sVb0SolZWsORxV+IbxxJyyGBVuHO5rYWxL2VhSex1zmcXr2VjiNZ2SCgqol75XUaQx8p8fde5v7VmaN/30unkawcEWC1RJOytY5XStyUc4eqnq1dlKGKX62//EfhC/9dfv/Xb+bANdHvv/3+y3/+6//+55//+sff/vr7n//7S0SV5r91THPhUZrxnAD5679/jdzT/HjxSe7XOe98//P7teptvHf++6//B28bTO0KZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyNzIzNgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagoxNyAwIG9iago8PCAvTGVuZ3RoIDM5NSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UktuxUAI2+cUXKDS8JvPeVJV3bz7b2tDUqkqvIkxxjB9ypC55UtdEnGFybderls8pnwuW1qZeYi7i40lPrbcl+4htl10LrE4HUfyCzKdKkSozarRofhCloUHkE7woQvCfTn+4y+AwdewDbjhPTJBsCTmKULGblEZmhJBEWHnkRWopFCfWcLfUe7r9zIFam+MpQtjHPQJtAVCbUjEAupAAETslFStkI5nJBO/Fd1nYhxg59GyAa4ZVESWe+zHiKnOqIy8RMQ+T036KJZMLVbGblMZX/yUjNR8dAUqqTTylPLQVbPQC1iJeRL2OfxI+OfWbCGGOm7W8onlHzPFMhLOYEs5YKGX40fg21l1Ea4dubjOdIEfldZwTLTrfsj1T/5021rNdbxyCKJA5U1B8LsOrkaxxMQyPp2NKXqiLLAamrxGM8FhEBHW98PIAxr9crwQNKdrIrRYIpu1YkSNimxzPb0E1kzvxTnWwxPCbO+d1qGyMzMqIYLauoZq60B2s77zcLafPzPoom0KZW5kc3RyZWFtCmVuZG9iagoxOCAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxNUUmKAzAMu+cV+kAhXpO8p0OZQ+f/18oOhTkECa+Sk5aYWAsPMYQfLD34kSFzN/0bfqLZu1l6ksnZ/5jnIlNR+FKoLmJCXYgbz6ER8D2haxJZsb3xOSyjmXO+Bx+FuAQzoQFjfUkyuajmlSETTgx1HA5apMK4a2LD4lrRPI3cbvtGZmUmhA2PZELcGICIIOsCshgslDY2EzJZzgPtDckNWmDXqRtRi4IrlNYJdKJWxKrM4LPm1nY3Qy3y4Kh98fpoVpdghdFL9Vh4X4U+mKmZdu6SQnrhTTsizB4KpDI7LSu1e8TqboH6P8tS8P3J9/gdrw/N/FycCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCA5NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFjcERwCAIBP9UQQkKCtpPJpOH9v+NEDJ8YOcO7oQFC7Z5Rh8FlSZeFVgHSmPcUI9AveFyLcncBQ9wJ3/a0FScltN3aZFJVSncpBJ5/w5nJpCoedFjnfcLY/sjPAplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9UeXBlIC9YT2JqZWN0IC9TdWJ0eXBlIC9Gb3JtIC9CQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvTGVuZ3RoIDM5Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nOMyNDBTMDY1VcjlMjc2ArNywCwjcyMgCySLYEFkM7jSABXzCnwKZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvTGVuZ3RoIDMyMiAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UbttxTAM7DUFFzAgfiXN4yBIkbd/mzvaqUjTvB9VXjKlXC51ySpZYfKlQ3WKpnyeZqb8DvWQ45ge2SG6U9aWexgWlol5Sh2xmiz3cAs2vgCaEnML8fcI8CuAUcBEoG7x9w+6WRJAGhT8FOiaq5ZYYgINi4Wt2RXiVt0pWLir+HYkuQcJcjFZ6FMORYopt8B8GSzZkVqc63JZCv9ufQIaYYU47LOLROB5wANMJP5kgGzPPlvs6upFNnaGOOnQgIuAm80kAUFTOKs+uGH7arvm55koJzg51q+iMb4NTuZLUt5XucfPoEHe+DM8Z3eOUA6aUAj03QIgh93ARoQ+tc/ALgO2Sbt3Y0r5nGQpvgQ2CvaoUx3K8GLszFZv2PzH6MpmUWyQlfXR6Q7K3KATYh5vZKFbsrb7Nw+zff8BXxl7ZAplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9MZW5ndGggODMgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRYy7DcAwCER7pmAEfib2PlGUwt6/DRAlbrgn3T1cHQmZKW4zw0MGngwshl1xgfSWMAtcR1COneyjYdW+6gSN9aZS8+8PlJ7srOKG6wECQhpmCmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0xlbmd0aCAzMjAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicNVJLbgUxCNvPKbhApfBPzvOqqou++29rE70VTDBg4ykvWdJLvtQl26XD5Fsf9yWxQt6P7ZrMUsX3FrMUzy2vR88Rty0KBFETPViZLxUi1M/06DqocEqfgVcItxQbvINJAINq+AcepTMgUOdAxrtiMlIDgiTYc2lxCIlyJol/pLye3yetpKH0PVmZy9+TS6XQHU1O6AHFysVJoF1J+aCZmEpEkpfrfbFC9IbAkjw+RzHJgOw2iW2iBSbnHqUlzMQUOrDHArxmmtVV6GDCHocpjFcLs6gebPJbE5WkHa3jGdkw3sswU2Kh4bAF1OZiZYLu5eM1r8KI7VGTXcNw7pbNdwjRaP4bFsrgYxWSgEensRINaTjAiMCeXjjFXvMTOQ7AiGOdmiwMY2gmp3qOicDQnrOlYcbHHlr18w9U6XyHCmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0xlbmd0aCAzNDAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicNVI5bgQxDOv9Cn0ggG7b79kgSJH8vw2p2RQDcXRSlDtaVHbLh4VUtex0+bSV2hI35HdlhcQJyasS7VKGSKi8ViHV75kyr7c1ZwTIUqXC5KTkccmCP8OlpwvH+baxr+XIHY8eWBUjoUTAMsXE6BqWzu6wZlt+lmnAj3iEnCvWLcdYBVIb3TjtiveheS2yBoi9mZaKCh1WiRZ+QfGgR4199hhUWCDR7RxJcIyJUJGAdoHaSAw5eyx2UR/0MygxE+jaG0XcQYElkpg5xbp09N/40LGg/tiMN786KulbWllj0j4b7ZTGLDLpelj0dPPWx4MLNO+i/OfVDBI0ZY2Sxget2jmGoplRVni3Q5MNzTHHIfMOnsMZCUr6PBS/jyUTHZTI3w4NoX9fHqOMnDbeAuaiP20VBw7is8NeuYEVShdrkvcBqUzogen/r/G1vtfXHx3tgMYKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvTGVuZ3RoIDI1MSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJwtUUlyA0EIu88r9IRmp99jlyuH5P/XCMoHBg2LQHRa4qCMnyAsV7zlkatow98zMYLfBYd+K9dtWORAVCBJY1A1oXbxevQe2HGYCcyT1rAMZqwP/Iwp3OjF4TEZZ7fXZdQQ7F2vPZlByaxcxCUTF0zVYSNnDj+ZMi60cz03IOdGWJdhkG5WGjMSjjSFSCGFqpukzgRBEoyuRo02chT7pS+PdIZVjagx7HMtbV/PTThr0OxYrPLklB5dcS4nFy+sHPT1NgMXUWms8kBIwP1uD/VzspPfeEvnzhbT43vNyfLCVGDFm9duQDbV4t+8iOP7jK/n5/n8A19gW4gKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvTGVuZ3RoIDIxNSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UTkOAyEM7PcV/kAkjC94T6Iozf6/zYzRVh7BXIa0lCGZ8lKTqCHlUz56mS6cutzXzGo055a0LXOAuLa8L62SwIlmiIPBaZi4AZo8AUPX0ahRQxce0NSlUyiw3AQ+irduD91jtYGXtiHniSBiKBksQc2pRRMWbc8npDW/Xosb3pft3chTpcaWGIEGAVY4HNfo1/CVPU8m0XQVMtSrNcsYCRNFIjz5jqbVE+taNNIyEtTGEaxqA7w7/TBOAAATccsCZJ9KlLPkxG+x9LMGV/r+AZ9HVJYKZW5kc3RyZWFtCmVuZG9iagoxNSAwIG9iago8PCAvVHlwZSAvRm9udCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9GaXJzdENoYXIgMCAvTGFzdENoYXIgMjU1Ci9Gb250RGVzY3JpcHRvciAxNCAwIFIgL1N1YnR5cGUgL1R5cGUzIC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdCi9DaGFyUHJvY3MgMTYgMCBSCi9FbmNvZGluZyA8PCAvVHlwZSAvRW5jb2RpbmcKL0RpZmZlcmVuY2VzIFsgNDggL3plcm8gL29uZSAvdHdvIC90aHJlZSAvZm91ciAvZml2ZSAvc2l4IDU2IC9laWdodCAvbmluZSBdCj4+Ci9XaWR0aHMgMTMgMCBSID4+CmVuZG9iagoxNCAwIG9iago8PCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL0ZvbnROYW1lIC9CTVFRRFYrRGVqYVZ1U2FucyAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvQXNjZW50IDkyOSAvRGVzY2VudCAtMjM2IC9DYXBIZWlnaHQgMAovWEhlaWdodCAwIC9JdGFsaWNBbmdsZSAwIC9TdGVtViAwIC9NYXhXaWR0aCAxMzQyID4+CmVuZG9iagoxMyAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNiAwIG9iago8PCAvZWlnaHQgMTcgMCBSIC9maXZlIDE4IDAgUiAvZm91ciAxOSAwIFIgL25pbmUgMjEgMCBSIC9vbmUgMjIgMCBSCi9zaXggMjMgMCBSIC90aHJlZSAyNCAwIFIgL3R3byAyNSAwIFIgL3plcm8gMjYgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNSAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDAgL2NhIDEgPj4KL0EyIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDEgL2NhIDEgPj4KL0EzIDw8IC9UeXBlIC9FeHRHU3RhdGUgL0NBIDEgL2NhIDAuNSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvRjEtRGVqYVZ1U2Fucy1taW51cyAyMCAwIFIgPj4KZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjI3IDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjcuMSkgL0NyZWF0aW9uRGF0ZSAoRDoyMDIzMDMxNDE2MTQyMFopCj4+CmVuZG9iagp4cmVmCjAgMjgKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMzI4MTUgMDAwMDAgbiAKMDAwMDAzMjU1MiAwMDAwMCBuIAowMDAwMDMyNTg0IDAwMDAwIG4gCjAwMDAwMzI3MjQgMDAwMDAgbiAKMDAwMDAzMjc0NSAwMDAwMCBuIAowMDAwMDMyNzY2IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDMzOSAwMDAwMCBuIAowMDAwMDI3NjcyIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAyNzY1MCAwMDAwMCBuIAowMDAwMDMxMzYxIDAwMDAwIG4gCjAwMDAwMzExNTQgMDAwMDAgbiAKMDAwMDAzMDc4OCAwMDAwMCBuIAowMDAwMDMyNDE0IDAwMDAwIG4gCjAwMDAwMjc2OTIgMDAwMDAgbiAKMDAwMDAyODE2MCAwMDAwMCBuIAowMDAwMDI4NDgyIDAwMDAwIG4gCjAwMDAwMjg2NDggMDAwMDAgbiAKMDAwMDAyODgyMCAwMDAwMCBuIAowMDAwMDI5MjE1IDAwMDAwIG4gCjAwMDAwMjkzNzAgMDAwMDAgbiAKMDAwMDAyOTc2MyAwMDAwMCBuIAowMDAwMDMwMTc2IDAwMDAwIG4gCjAwMDAwMzA1MDAgMDAwMDAgbiAKMDAwMDAzMjg3NSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9TaXplIDI4IC9Sb290IDEgMCBSIC9JbmZvIDI3IDAgUiA+PgpzdGFydHhyZWYKMzMwMjYKJSVFT0YK", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["fig, ax = plt.subplots(2, 2, figsize=(10, 6))\n", "for i in range(4):\n", " ax_sub = ax[i // 2][i % 2]\n", " ax_sub.bar(np.arange(out.shape[1], dtype=np.int32), out[i + 4, :, 0, 14, 14], **plot_args)\n", " ax_sub.set_yscale(\"log\")\n", " ax_sub.set_xticks([0, 64, 128, 192, 256])\n", "plt.show()\n", "plt.close()"]}, {"cell_type": "markdown", "id": "54244225", "metadata": {"papermill": {"duration": 0.03147, "end_time": "2023-03-14T16:14:21.422816", "exception": false, "start_time": "2023-03-14T16:14:21.391346", "status": "completed"}, "tags": []}, "source": ["Overall we see a very diverse set of distributions, with a usual peak\n", "for 0 and close to 1. However, the distributions in the first row show a\n", "potentially undesirable behavior. For instance, the value 242 has a\n", "1000x lower likelihood than 243 although they are extremely close and\n", "can often not be distinguished. This shows that the model might have not\n", "generlized well over pixel values. The better solution to this problem\n", "is to use discrete logitics mixtures instead of a softmax distribution.\n", "A discrete logistic distribution can be imagined as discretized, binned\n", "Gaussians. Using a mixture of discrete logistics instead of a softmax\n", "introduces an inductive bias to the model to assign close-by values\n", "similar likelihoods. We can visualize a discrete logistic below:"]}, {"cell_type": "code", "execution_count": 28, "id": "495873dd", "metadata": {"execution": {"iopub.execute_input": "2023-03-14T16:14:21.486686Z", "iopub.status.busy": "2023-03-14T16:14:21.486297Z", "iopub.status.idle": "2023-03-14T16:14:22.170511Z", "shell.execute_reply": "2023-03-14T16:14:22.169883Z"}, "papermill": {"duration": 0.72124, "end_time": "2023-03-14T16:14:22.174755", "exception": false, "start_time": "2023-03-14T16:14:21.453515", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1R5cGUgL0NhdGFsb2cgL1BhZ2VzIDIgMCBSID4+CmVuZG9iago4IDAgb2JqCjw8IC9Gb250IDMgMCBSIC9YT2JqZWN0IDcgMCBSIC9FeHRHU3RhdGUgNCAwIFIgL1BhdHRlcm4gNSAwIFIKL1NoYWRpbmcgNiAwIFIgL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9UeXBlIC9QYWdlIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovTWVkaWFCb3ggWyAwIDAgNDA5LjMyNSAyMzEuNjEwNjI1IF0gL0NvbnRlbnRzIDkgMCBSIC9Bbm5vdHMgMTAgMCBSID4+CmVuZG9iago5IDAgb2JqCjw8IC9MZW5ndGggMTIgMCBSIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nNXdTXMcVbIG4H39ilqaBenz/bHElxlHsLqAIu6CmMVgDCPCPQQ2AzP//mZWd0vVlW96HNhaJDMOrBep1O8jqfucVHfV8y9f/37/6vU3L1+s//Pt8vzxrVfvlrj+zH9+WsP6M//5Y43rS/7z0xL4rdNSwqScKv/9zcPfU47UYmj81zf8bjdv/mNZflyef8EHeLcGmrG30kMdQ71RZoizhT7Wt/K5X968w/K+916W2mjwZyuJZqvyeU9LnnwjbrI3D1ni253GJbx87E223ehfV3XYnAuNNbbGtde3r9f/W/+5Pv8iSbe4fsV/fuY/Z7FH04VNe6d+e/vOyc1nXb5dvl5/vR4wUKz8Vbgec3vz5SVdfuWvUFg/D/yfWie5+ZXO4q9Oy4u75flf4xrjevfj9lW7+2H5bn0WQ/hs/dt699Xyl7vl6+1TfbqCMQcK7fA1eAg/Qc0YJ20H+JCm8SmbjkT58MW8Zp+iZ480+ofVTE9YM/EHtnHo+RB+gqIpFYof2jQ/ZdPRb0vy25+iX2/0YeXKE5bLme+i4/G+8Rp+gpo5DRrxA38u61M2VXf4J3yH/yd7jkgfWrPd1Nz/cM9EQe4oJvX9AdLtAf73M77FNPL1n/XZ/b85mvyw1yfTtNbXZ6/frL///c2/Xj+d6Ac+qv7ZH//t4HlQH3WKiQ3KjyJP+I1zOVIbVEoso+xqPmYfWbMVSi322eJo/6VpevKmM8tiKpe0a/qYfWTTwd/aubdaRs/vL1qevKisGfLMKcRd0134kVVjrJRirvz/Ov/LV7U9fdnCH99maPuuD9nHVi2BH/j5x5Qftdv7m46nb9oD5VJmHPuqj+HHduUbFltNOcz83vt5PsLT3y3FWfnfofe+L/sYfmxZOWKJYfBd03x/13T7oCYH+VwOF9u2CZyF+K5yWx2+95GN332mHsblke2teqz75fu/f3//5v63//xJ23xmSHOEPFOsnd8o551jypV3ku26dyzbVpNXaX/Y77/evP83L5cPfE/Zln7O/5HmVPsizhPv/ubtNuLN+3JwnG2b+sKtDGgqMrlTTSOmo4CVYzHPMqjpJpP4KIct2Zv35VDMtQxoKjKp8wNsPf502DkW8yyDmm4yke9ai74/MXMo5loGNBUZfhhqIev7EyvHYp5lUNNNJlLuUd+fmDkUcy0DmopM4IVhUY/Cdo7FPMugpptMoBnAusXMoZhrGdCUZTKv1FvX6xYrN8Qcy8Cmm4x8oF632DkUcy0DmorMkLmCXreYORbzLIOaikyfNIdet5g5FvMsg5puMoX3QXrdYudQzLUMaCoybfI+SK9bzByLeZZBTTeZwkfR6xY7h2KuZUBTkamTApjDmDkW8yyDmm4ymauA9YyZQzHXMqCpyJRBFcxhzByLeZZBTTeZTBnMYewcirmWAU1FJg+KYA5j5ljMswxqusnwW2AOY+dQzLUMaCoyqfM+CKxbrByLeZZBTTeZxB8I1i1mDsVcy4CmIhM7JTCHMXMs5lkGNd1kIk00nzFzKOZaBjQVmdCoozmMlWMxzzKo6SYTqaI5jJlDMdcyoCnLpNkogTmMlRtijmVg000m8j5Ir1vsHIq5lgFNRWbIc1n0usXMsZhnGdR0k+HPCuYwdg7FXMuApiLTK2UwhzFzLOZZBjXdZOSZWnrdYudQzLUMaCoyjW84mMOYORbzLIOaikyd1MAcxsyxmGcZ1HSTKVTAHMbOoZhrGdBUZMrkfRBYt1g5FvMsg5puMpn3QWDdYuZQzLUMaCoy8mIPMIcxcyzmWQY13WQyVTCHsXMo5loGNBWZNCiBOYyZYzHPMqjpJpMpgDmMnUMx1zKgqcjETgPMYcwci3mWQU03mUQVzWfMHIq5lgFNRSZ03geBdYuVYzHPMqjpJpN4HwTWLWYOxVzLgKYsw+/E+yC9brFyQ8yxDGy6yURqYA5j51DMtQxoKjKjUQFzGDPHYp5lUNNNJlIEcxg7h2KuZUBTken1+tatgJVjMc8yqOkmE6iDOYydQzHXMqCpyLTK+yC9bjFzLOZZBjXdZPgoYA5j51DMtQxoKjK18j4IrGesHIt5lkFNRaZMGmAOY+ZYzLMMarrJFKpgDmPnUMy1DGgqMnKSJTCHMXMs5lkGNd1kCkUwh7FzKOZaBjQVGTmZFpjDmDkW8yyDmm4ymfdBYN1i5lDMtQxoKjJx8D4IrFusHIt5lkFNN5nM+yCwbjFzKOZaBjQVmdBpojmMlWMxzzKo6SaTqKM5jJlDMdcyoCnLzE4FjGGM2PBy7IKKCkuiBGYwVgy1PKuAnqwyOgUwfzFijOVYBfUUlch7H71WsWKI5VkF9GQVOU8umLsYMcZyrIJ6ikrkPY9eo1gxxPKsAnqyipyaEcxbjBhjOVZBPUUl0ACzFiuGWJ5VQE9WqZUamLMYMcZyrIJ6igp/GJixWDHE8qwCesoJrCpFMF8xYozlWAX13E7RxHsb8FprHGMsxyqo53Y6A97ZgNdZGzHE8qwCem4vcOOdDXjOC44xlmMV1HN7Mjgl9HokI4ZYnlVAz+3pQRTgc11gjLEcq6Ce269FqKNJihFDLM8qoKfsmamCOQpOMZVjE1BTHoAoo3PDwBRCORYBLU9LRC9HgyFW8usBSp6Wjl63CENo5FcDdORvjoRe4KrThpCaaw9UnUE6ei20TpvJ5FgEdD8t27Uw1GpUpx06ddciqLtcAw2dY0GnEzpN3yKgu1wzDZ2PQ6U5Iqct9SuCurPIQOduUWnGTr4faVD301IyOs+PSktETsX39wjqziIDnRNKpTUhpy11LAK6n5Za0PnDVNomcmq+71lRdxaZBM41p9LZkdP0/eiLup+WVtB5CY9pLwU4nVO/Iqg7i0x0DstjOlICTufUsQjozpu8is53ekwnYpq+H3tR89MyAjoz7iFtsWmkc+jXAzVnj4rOoXxIWw0dKJ1TxyKg+2mZAZ1v+5D2MDTTOfTrgZqzR0NnZj+kI6SslS6pYxHQ/bTEEAmcxv8Yz9qnlrqkfk1ge0Fp6KoPh7inuC08Dge5xp5VQH1WiRFdJeQQ95Z7BliX2LEKqi8qDV1V5hAPbgewrrFnFVBfJtAJXYXoEM8y8tRY19ixCqovKp3AVav2caYQcx/zgLWLPauA+qySE7rK2T6WkxXVgLCusWMVVF9UOroq3j7m+pzHpLGusWcVUJ9VuB24iuI+5r+NmFtWWA+xYxVUX1QGuurmPs7cKMY6FNZD7FkF1GeVmtFVWvdxplr5m6IprIfYsQqqLyoDXdV3H2eaOcSeFNZD7FkF1GeVVtBVoPdx4W+KxAc+YO1ixyqovqhMAlcN38eVas/tvA/aHWQXe1YB9eXJBAVdZX4fd0olpdAOWLvYsQqqLyqTdzZ6vbKLt6cF85bngLWLPauA+qwyCu9n9HrlMa6FcpzjvOJ/PMg+dqyC6rPK5GOA+cpj3CL/qOSZxy3WPnasguqLSqUC5iuPcY/8o9JGzbdY+9izCqh/WlLg99TzlV08Oo2R48Fql/o1geXFpNLU05VdHGPgXWBsudxa3eSeXQAAu8TIOxu1XtnFMcvzlPqc5eC1zx27IABx4RusJyy7ODY5p0NstRy89rlnFwAgT4KMvLtRa5ZdHGeiEFJSXvvcsQsCEBcuoqcsuziFeP396Y3XTe7ZBQCwS+avup6z7OLL9wX//dbrJnfsggDEpVPWk5ZdfL0f2X4HtPfa555dAAC78BtVz1p28eVxZ8x58Nrnjl0QgLh06nrasouv65SRDl773LMLAGCXKr/90uuXx/iyqm23WrvUsQkqLyaD9zl67fIYX7Y/8UC1jz2rgPqs0jJvdPTK5THetsolxHGLtY8dq6D6ojLkPdX9ymN8HqvMUG6x9rFnFVCfVXqmCV7rs4u3Edz5abT7g+xixyqovqhMiuD1Prt4G9fm0g9Yu9izCqgvL4LinwQwbdnF22i/biO4/UF2sWMVVF9UJjUwb9nF26+BWisHrF3sWQXUZ5XJNxtMW3bx9ivDtD2ZZ3+QXexYBdU/LTnwClXPWvbx+dfLfSish9ivCqwvKpW3Nmq9so/PT0XoXWE9xJ5VQH1W4d1M1XOWfbw9bSWVqbAeYscqqL6oVOp6yrKPz09x2oa1t1gPsWcVUJ9VUqSgZyz7+Px0uPNLLW+xrrFjFVRfVBolPWHZx+enTraosa6xZxVQX164Hano+cohnmV7rDkc45I6NkHlxaTxe6rVyiEeOc+oqa6xZxVQn1VK5H2NXq3cxr2V7Ukrh4NcY8cqqL6odN7X6NXKbdz5ExSAdYk9q4D6rFITZT1dOcazbsP840EusWMVVF9UOjU9XTnGI1SEdYk9q4D6rNISDT1dOcZtzgSwLrFjFVRfVAYFcEaVQ9xqbgjrHHtWAfVZpfMeD5xV5RC3lBPCOseOVVB9URm8r9HrlWM8t1MPqYNM52ckgvXlBDyZ9zV6vXKM+3mOcDxI9z5HQPVFZfK+Rq9XjvFFBWN5VgH1WWUWSmC6YsTY0LEK6ikqkwqYrhzjy/fK8SDuv1dQ/dNSQqGupytGbBj6VYE9RWXS1NMVM4ZYnlVAT1aJlaKerlgxxnKsgnqySgq8s9FngjNijOVYBfUUlco7G302OCuGWJ5VQE85aWDgnY16rLVijOVYBfUUlUZBz1fMGGJ5VgE9WaVE+Q2gugMxYozlWAX1FJVGVc9XzBhieVYBPVmlRup6vmLFGMuxCuopKp2Cnq+YMcTyrAJ6skpLvLPRCxMjxliOVVBPUem8s9ELEyuGWJ5VQE9W6Yl3NnphYsQYy7EK6ikqnaYepJgxxPKsAnrK6Z8zRT0bsGKM5VgF9RSVQQXMV6wYYnlWAT1ZZWZqYJBixBjLsQrqKSqDBhikWDHE8qwCep6WGoo8u+34M2HEBpZfFdhTVCbfYPVYa8YQy7MK6MkqUZ53rs+4b8QYy7EK6ikqk7qeDZgxxPKsAnqySqoU9GzAijGWYxXUk1UyH0PPBqwYYzlWQT1FpVLRswEzhlieVUBPubxJoK5nA1aMsRyroJ6iUnlno9crVgyxPKuAnqxSI+9s9MLEiDGWYxXUU1Qa72z0wsSKIZZnFdCTVVqkBq4SZMQYy7EK6ikqXETPBswYYnlWAT1ZpSeKejZgxRjLsQrqKSodXfvWjCGWZxXQk1VGQldItmKM5VgF9RSVjq6kbcYQy7MK6CkXqsvomutWjLEcq6CeojJ4Z6MXJlYMsTyrgJ6npYVMRc8GjNjA8qsCe4rKoK5nA2YMsTyrgJ6sErOcW0Zd3dCIMZZjFdRTVCZFPRswY4jlWQX0ZJVUqOjZgBVjLMcqqKeoTN7ZqMdaM4ZYnlVAT1aRm61nA1aMsRyroJ5ycd3AOxv1WGvFGMuxCuopKpWyng2YMcTyrAJ6skrlzwmuxGzEGMuxCuopKpUGuBqzFUMszyqgJ6u0SEHPBqwYYzlWQT1FpVHSswEzhlieVUBPVumRdzZ6YWLEGMuxCuopKo13NnphYsUQy7MK6MkqQ87dqxcmRoyxHKugnqLSKYL5ihVDLM8qoCerzEQFDFKMGGM5VkE9RaVTA4MUK4ZYnlVAz9PSQ6KhZwNGbGD5VYE9RWVQ1LMBM4ZYnlVAT1aJmXc26rHWijGWYxXUU1QG72zUY60ZQyzPKqAnq6TMOxv1WGvFGMuxCuopKpOCng2YMcTyrAJ6sgrf7KRnA1aMsRyroJ6iMqnq2YAZQyzPKqAnq5RCXc8GrBhjOVZBPUVlXt+4VTFiiOVZBfRklVp5Z6MXJkaMsRyroJ6s0vjD9GzAijGWYxXUU1Qq72z0wsSKIZZnFdCTVXqgoWcDVoyxHKugnqLSKOrZgBlDLM8qoCerjEhZzwasGGM5VkE9RaVRBfMVK4ZYnlVAT1aZkQYYpBgxxnKsgnqKSuedjV6YWDHE8qwCep6WERLvbNRjrREbWH5VYE9R6byzUY+1ZgyxPKuAnqwSE3U9G7BijOVYBfUUlU5TzwbMGGJ5VgE9WSVlSno2YMUYy7EK6ikqg4qeDZgxxPKsAnqySs7U9GzAijGWYxXUU1QG72zUY60ZQyzPKqAnq5TCOxu9XjFijOVYBfUUlck7G70wsWKI5VkF9GSVWqjq2YAVYyzHKqinqEwuohcmVgyxPKuAnqzSKgU9G7BijOVYBfVklc7H0LMBK8ZYjlVQT1GpVPVswIwhlmcV0JNVRuCdjV6YGDHGcqyCeopK5Z2NXphYMcTyrAJ6ssqMvLPRCxMjxlgPKs+/SFI4rV+t/PHSJa4v1+dfvv79/tXrb16+WF+9W5Tc6RrxJ+Sfz8sxl2+XzDfi+L19zY7vC46qP3z/nrsD2EcNXCRQrOtPXORn/vPH+c2Xl3T5dYlrWD8P/J+ivErobMNfjuthXp2WF3fL87/yf0/r3Y98xLDe/bB8tz778v7dq7ef8ack3jFc/lmfvf7t9frml5/u3/12/2r9gf/19v77f/12/8s/P1v/tt59tfzlbvma//f/oufBugplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjQ4NjgKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTcgMCBvYmoKPDwgL0xlbmd0aCAxNjQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZDBEUMhCETvVrElgIBAPclkcvi//2tAk1xkHWD3qTuBkFGHM8Nn4smD07E0cG8VjGsIryP0CE0Ck8DEwZp4DAsBp2GRYy7fVZZVp5Wumo2e171jQdVplzUNbdqB8q2PP8I13qPwGuweQgexKHRuZVoLmVg8a5w7zKPM535O23c9GK2m1Kw3ctnXPTrL1FBeWvuEzmi0/SfXL7sxXh+FFDkICmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0xlbmd0aCAxNzAgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZBLEsMgDEP3nEJHAP+A87TT6YLcf1vLmXSDFGPLL0RXdOyVh8fGlI33aGNPhC1c5XQaTlMZj4u7Zl2gy2Ey02+8mrnAVGGR1eyi+hi8ofOsZoevVTMxhDeZEhpgKndyD/X1pzjt25KQbFdh0J0apLMwzJH8PRBTc9BziJH8I19ya2HQmeYXFy2rGa1lTNHsYapsLQzqjUF3yvXUeq7zMBHv8wPfQT5kCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0xlbmd0aCAzMDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvTGVuZ3RoIDI0NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFkU1yBSEIhPeeoi/wquRXPc+kUllM7r8NzbwkK1qF5gPTAhNH8BJD7ImVEx8yfC/oMny3MjvwOtmZcE+4blzDZcMzYVvgOyrLO15Dd7ZSP52hqu8aOd4uUjV0ZWSfeqGaC8yQiK4RWXQrl3VA05TuUuEabFuCFPVKrCedoDToEcrwd5RrfHUTT6+x5FTNIVrNrRMairBseEHUySQRtQ2LJ5ZzIVH5qhurOi5gkyXi9IDcoJVmfHpSSREwg3ysyWjMAjbQk7tnF8aaSx5Fjlc0mLA7STXwgPfitr73NnGP8xf4hXff/ysOfdcCPn8AS/5dBgplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9MZW5ndGggMjMyIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVRSW7EMAy7+xX8wADW7rwnxaCH9v/XUsoUCEAltrglYmMjAi8x+DmI3PiSNaMmfmdyV/wsT4VHwq3gSRSBl+FedoLLG8ZlPw4zH7yXVs6kxpMMyEU2PTwRMtglEDowuwZ12Gbaib4h4bMjUs1GltPXEvTSKgTKU7bf6YISbav6c/usC2372hNOdnvqSeUTiOeWrMBl4xWTxVgGPVG5SzF9kOpsoSehvCifg2w+aohElyhn4InBwSjQDuy57WfiVSFoXd2nbWOoRkrH078NTU2SCPlECWe2NO4W/n/Pvb7X+w9OIVQRCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0xlbmd0aCAyMzEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicNU85kgQhDMt5hT4wVRjbQL+np7Y22Pl/upKZTpDwIcnTEx2ZeJkjI7Bmx9taZCBm4FNMxb/2tA8TqvfgHiKUiwthhpFw1qzjbp6OF/92lc9YB+82+IpZXhDYwkzWVxZnLtsFY2mcxDnJboxdE7GNda2nU1hHMKEMhHS2w5Qgc1Sk9MmOMuboOJEnnovv9tssdjl+DusLNo0hFef4KnqCNoOi7HnvAhpyQf9d3fgeRbvoJSAbCRbWUWLunOWEX712dB61KBJzQppBLhMhzekqphCaUKyzo6BSUXCpPqforJ9/5V9cLQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9MZW5ndGggMjQ5IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvTGVuZ3RoIDM5NSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9UktuxUAI2+cUXKDS8JvPeVJV3bz7b2tDUqkqvIkxxjB9ypC55UtdEnGFybderls8pnwuW1qZeYi7i40lPrbcl+4htl10LrE4HUfyCzKdKkSozarRofhCloUHkE7woQvCfTn+4y+AwdewDbjhPTJBsCTmKULGblEZmhJBEWHnkRWopFCfWcLfUe7r9zIFam+MpQtjHPQJtAVCbUjEAupAAETslFStkI5nJBO/Fd1nYhxg59GyAa4ZVESWe+zHiKnOqIy8RMQ+T036KJZMLVbGblMZX/yUjNR8dAUqqTTylPLQVbPQC1iJeRL2OfxI+OfWbCGGOm7W8onlHzPFMhLOYEs5YKGX40fg21l1Ea4dubjOdIEfldZwTLTrfsj1T/5021rNdbxyCKJA5U1B8LsOrkaxxMQyPp2NKXqiLLAamrxGM8FhEBHW98PIAxr9crwQNKdrIrRYIpu1YkSNimxzPb0E1kzvxTnWwxPCbO+d1qGyMzMqIYLauoZq60B2s77zcLafPzPoom0KZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvTGVuZ3RoIDI0OSAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxNUUmKAzAMu+cV+kAhXpO8p0OZQ+f/18oOhTkECa+Sk5aYWAsPMYQfLD34kSFzN/0bfqLZu1l6ksnZ/5jnIlNR+FKoLmJCXYgbz6ER8D2haxJZsb3xOSyjmXO+Bx+FuAQzoQFjfUkyuajmlSETTgx1HA5apMK4a2LD4lrRPI3cbvtGZmUmhA2PZELcGICIIOsCshgslDY2EzJZzgPtDckNWmDXqRtRi4IrlNYJdKJWxKrM4LPm1nY3Qy3y4Kh98fpoVpdghdFL9Vh4X4U+mKmZdu6SQnrhTTsizB4KpDI7LSu1e8TqboH6P8tS8P3J9/gdrw/N/FycCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0xlbmd0aCA5NCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFjcERwCAIBP9UQQkKCtpPJpOH9v+NEDJ8YOcO7oQFC7Z5Rh8FlSZeFVgHSmPcUI9AveFyLcncBQ9wJ3/a0FScltN3aZFJVSncpBJ5/w5nJpCoedFjnfcLY/sjPAplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9MZW5ndGggMzQxIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEVSS25EMQjbv1NwgUjhl5DztKq6mN5/W5tM1c3gCWBseMtTpmTKsLklIyTXlE99IkOspvw0ciQipvhJCQV2lY/Ha0usjeyRqBSf2vHjsfRGptkVWvXu0aXNolHNysg5yBChnhW6snvUDtnwelxIuu+UzSEcy/9QgSxl3XIKJUFb0HfsEd8PHa6CK4JhsGsug+1lMtT/+ocWXO9992LHLoAWrOe+wQ4AqKcTtAXIGdruNiloAFW6i0nCo/J6bnaibKNV6fkcADMOMHLAiCVbHb7R3gCWfV3oRY2K/StAUVlA/MjVdsHeMclIcBbmBo69cDzFmXBLOMYCQIq94hh68CXY5i9Xroia8Al1umQvvMKe2ubnQpMId60ADl5kw62ro6iW7ek8gvZnRXJGjNSLODohklrSOYLi0qAeWuNcN7HibSOxuVff7h/hnC9c9usXS+yExAplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9MZW5ndGggNzIgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZcQL6piblCLhdIDMTKAbMMgLQlnIKIZ4CYIG0QxSAWRLGZiRlEHZwBkcvgSgMAJdsWyQplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9MZW5ndGggNDcgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZclhBWLhdMLAfMAtGWcAoinsGVBgC5Zw0nCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0xlbmd0aCAxNjMgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicRZA7EgMhDEN7TqEj+CMDPs9mMik2929j2GxSwNNYIIO7E4LU2oKJ6IKHtiXdBe+tBGdj/Ok2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlDcPVf9b9i3TmbiYHJyh0IzepT3Pk2O6K6usn+pMfcrNd+K+xVYWlZS8sJt527ZkAJ3FM52qs9Px8KOvYKZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iago8PCAvTGVuZ3RoIDIxOCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw9ULmNBDEMy12FGljAeu2pZxaLS6b/9Ej59iLRFkVSKjWZkikvdZQlWVPeOnyWxA55huVuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rnKPn16LGz7D8UMUkZIO5jX/WP3ycw2vU48nkW5vvuJenKkOAxEckpq8I11YsS4SEWk1QU3PwFotgLu3Xv4btCO6DED2icRxmlKOob9rcKXPL+UnU9gKZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iago8PCAvTGVuZ3RoIDgzIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4m9j5RlMLevw0QJW64J909XB0JmSluM8NDBp4MLIZdcYH0ljALXEdQjp3so2HVvuoEjfWmUvPvD5Se7KzihusBAkIaZgplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjw8IC9MZW5ndGggNTEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicMza0UDBQMDQwB5JGhkCWkYlCiiEXSADEzOWCCeaAWQZAGqI4B64mhyuDKw0A4bQNmAplbmRzdHJlYW0KZW5kb2JqCjM0IDAgb2JqCjw8IC9MZW5ndGggMTYwIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nEWQORIDMQgEc72CJ0hcgvesy7XB+v+pB9ZHoukCNBy6Fk3KehRoPumxRqG60GvoLEqSRMEWkh1Qp2OIOyhITEhjkki2HoMjmlizXZiZVCqzUuG0acXCv9la1chEjXCN/InpBlT8T+pclPBNg6+SMfoYVLw7g4xJ+F5F3Fox7f5EMLEZ9glvRSYFhImxqdm+z2CGzPcK1zjH8w1MgjfrCmVuZHN0cmVhbQplbmRvYmoKMzUgMCBvYmoKPDwgL0xlbmd0aCAzMzQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicLVJLcsUgDNtzCl2gM/gH5DzpdLp4vf+2kpNFRg5g9DHlholKfFkgt6PWxLeNzECF4a+rzIXPSNvIOojLkIu4ki2Fe0Qs5DHEPMSC76vxHh75rMzJswfGL9l3Dyv21IRlIePFGdphFcdhFeRYsHUhqnt4U6TDqSTY44v/PsVzLQQtfEbQgF/kn6+O4PmSFmn3mG3TrnqwTDuqpLAcbE9zXiZfWme5Oh7PB8n2rtgRUrsCFIW5M85z4SjTVka0FnY2SGpcbG+O/VhK0IVuXEaKI5CfqSI8oKTJzCYK4o+cHnIqA2Hqmq50chtVcaeezDWbi7czSWbrvkixmcJ5XTiz/gxTZrV5J89yotSpCO+xZ0vQ0Dmunr2WWWh0mxO8pITPxk5PTr5XM+shORUJqWJaV8FpFJliCdsSX1NRU5p6Gf778u7xO37+ASxzfHMKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvTGVuZ3RoIDMyMCAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJw1UktuBTEI288puECl8E/O86qqi777b2sTvRVMMGDjKS9Z0ku+1CXbpcPkWx/3JbFC3o/tmsxSxfcWsxTPLa9HzxG3LQoEURM9WJkvFSLUz/ToOqhwSp+BVwi3FBu8g0kAg2r4Bx6lMyBQ50DGu2IyUgOCJNhzaXEIiXImiX+kvJ7fJ62kofQ9WZnL35NLpdAdTU7oAcXKxUmgXUn5oJmYSkSSl+t9sUL0hsCSPD5HMcmA7DaJbaIFJucepSXMxBQ6sMcCvGaa1VXoYMIehymMVwuzqB5s8lsTlaQdreMZ2TDeyzBTYqHhsAXU5mJlgu7l4zWvwojtUZNdw3Duls13CNFo/hsWyuBjFZKAR6exEg1pOMCIwJ5eOMVe8xM5DsCIY52aLAxjaCaneo6JwNCes6VhxsceWvXzD1TpfIcKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago8PCAvTGVuZ3RoIDE4IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iago8PCAvTGVuZ3RoIDEzMyAvRmlsdGVyIC9GbGF0ZURlY29kZSA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9MZW5ndGggMzQwIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVSOW4EMQzr/Qp9IIBu2+/ZIEiR/L8NqdkUA3F0UpQ7WlR2y4eFVLXsdPm0ldoSN+R3ZYXECcmrEu1ShkiovFYh1e+ZMq+3NWcEyFKlwuSk5HHJgj/DpacLx/m2sa/lyB2PHlgVI6FEwDLFxOgals7usGZbfpZpwI94hJwr1i3HWAVSG9047Yr3oXktsgaIvZmWigodVokWfkHxoEeNffYYVFgg0e0cSXCMiVCRgHaB2kgMOXssdlEf9DMoMRPo2htF3EGBJZKYOcW6dPTf+NCxoP7YjDe/OirpW1pZY9I+G+2Uxiwy6XpY9HTz1seDCzTvovzn1QwSNGWNksYHrdo5hqKZUVZ4t0OTDc0xxyHzDp7DGQlK+jwUv48lEx2UyN8ODaF/Xx6jjJw23gLmoj9tFQcO4rPDXrmBFUoXa5L3AalM6IHp/6/xtb7X1x8d7YDGCmVuZHN0cmVhbQplbmRvYmoKNDAgMCBvYmoKPDwgL0xlbmd0aCAyNTEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicLVFJcgNBCLvPK/SEZqffY5crh+T/1wjKBwYNi0B0WuKgjJ8gLFe85ZGraMPfMzGC3wWHfivXbVjkQFQgSWNQNaF28Xr0HthxmAnMk9awDGasD/yMKdzoxeExGWe312XUEOxdrz2ZQcmsXMQlExdM1WEjZw4/mTIutHM9NyDnRliXYZBuVhozEo40hUghhaqbpM4EQRKMrkaNNnIU+6Uvj3SGVY2oMexzLW1fz004a9DsWKzy5JQeXXEuJxcvrBz09TYDF1FprPJASMD9bg/1c7KT33hL584W0+N7zcnywlRgxZvXbkA21eLfvIjj+4yv5+f5/ANfYFuICmVuZHN0cmVhbQplbmRvYmoKNDEgMCBvYmoKPDwgL0xlbmd0aCAxNzQgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicTZBJDkMhDEP3nMIXqIQzwOc8v6q6aO+/rUMHdYH85CBwPDzQcSQudGTojI4rmxzjwLMgY+LROP/JuD7EMUHdoi1Yl3bH2cwSc8IyMQK2RsnZPKLAD8dcCBJklx++wCAiXY/5VvNZk/TPtzvdj7q0Zl89osCJ7AjFsAFXgP26x4FLwvle0+SXKiVjE4fygeoiUjY7oRC1VOxyqoqz3ZsrcBX0/NFD7u0FtSM83wplbmRzdHJlYW0KZW5kb2JqCjQyIDAgb2JqCjw8IC9MZW5ndGggNzUgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjQzIDAgb2JqCjw8IC9MZW5ndGggODkgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicNU25EYAwDOs9hUfAj0i8D8dRhP1b7IQ0lk6fEcoHa+QBguGNLyH4oi8ZhLULDyr7SHTYRA1nFSQTw68s8KqcFW1zJRPZWUyjs0HL9K3tb4Meuj/djhwKCmVuZHN0cmVhbQplbmRvYmoKNDQgMCBvYmoKPDwgL0xlbmd0aCAxNDEgL0ZpbHRlciAvRmxhdGVEZWNvZGUgPj4Kc3RyZWFtCnicPY/BDsMwCEPv+Qr/QKTYKaF8T6dqh+7/ryNLuwt6AmOMhdDQG6qaw4Zgm+PF0iVUa/gUxUAlN8iZYA6lpNIdR5F6YjgYXB60G47isej6EbuSZn3QxkK6JWiAe6xTadymcRPEHTUF6inqnKO8ELmfqWfYNJLdNLOSc7gNv3vPU9f/p6u8y/kFvXcu/gplbmRzdHJlYW0KZW5kb2JqCjQ1IDAgb2JqCjw8IC9MZW5ndGggMjE1IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlID4+CnN0cmVhbQp4nDVROQ4DIQzs9xX+QCSML3hPoijN/r/NjNFWHsFchrSUIZnyUpOoIeVTPnqZLpy63NfMajTnlrQtc4C4trwvrZLAiWaIg8FpmLgBmjwBQ9fRqFFDFx7Q1KVTKLDcBD6Kt24P3WO1gZe2IeeJIGIoGSxBzalFExZtzyekNb9eixvel+3dyFOlxpYYgQYBVjgc1+jX8JU9TybRdBUy1Ks1yxgJE0UiPPmOptUT61o00jIS1MYRrGoDvDv9ME4AABNxywJkn0qUs+TEb7H0swZX+v4Bn0dUlgplbmRzdHJlYW0KZW5kb2JqCjE1IDAgb2JqCjw8IC9UeXBlIC9Gb250IC9CYXNlRm9udCAvQk1RUURWK0RlamFWdVNhbnMgL0ZpcnN0Q2hhciAwIC9MYXN0Q2hhciAyNTUKL0ZvbnREZXNjcmlwdG9yIDE0IDAgUiAvU3VidHlwZSAvVHlwZTMgL05hbWUgL0JNUVFEVitEZWphVnVTYW5zCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0KL0NoYXJQcm9jcyAxNiAwIFIKL0VuY29kaW5nIDw8IC9UeXBlIC9FbmNvZGluZwovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNDYgL3BlcmlvZCA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggNTYKL2VpZ2h0IDY4IC9EIDgwIC9QIDk3IC9hIC9iIC9jIC9kIC9lIDEwMyAvZyAxMDUgL2kgMTA4IC9sIDExMCAvbiAvbyAxMTQgL3IKL3MgL3QgL3UgL3YgMTIwIC94IC95IF0KPj4KL1dpZHRocyAxMyAwIFIgPj4KZW5kb2JqCjE0IDAgb2JqCjw8IC9UeXBlIC9Gb250RGVzY3JpcHRvciAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zIC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Bc2NlbnQgOTI5IC9EZXNjZW50IC0yMzYgL0NhcEhlaWdodCAwCi9YSGVpZ2h0IDAgL0l0YWxpY0FuZ2xlIDAgL1N0ZW1WIDAgL01heFdpZHRoIDEzNDIgPj4KZW5kb2JqCjEzIDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE2IDAgb2JqCjw8IC9EIDE3IDAgUiAvUCAxOCAwIFIgL2EgMTkgMCBSIC9iIDIwIDAgUiAvYyAyMSAwIFIgL2QgMjIgMCBSIC9lIDIzIDAgUgovZWlnaHQgMjQgMCBSIC9maXZlIDI1IDAgUiAvZm91ciAyNiAwIFIgL2cgMjcgMCBSIC9pIDI4IDAgUiAvbCAyOSAwIFIKL24gMzAgMCBSIC9vIDMxIDAgUiAvb25lIDMyIDAgUiAvcGVyaW9kIDMzIDAgUiAvciAzNCAwIFIgL3MgMzUgMCBSCi9zaXggMzYgMCBSIC9zcGFjZSAzNyAwIFIgL3QgMzggMCBSIC90aHJlZSAzOSAwIFIgL3R3byA0MCAwIFIgL3UgNDEgMCBSCi92IDQyIDAgUiAveCA0MyAwIFIgL3kgNDQgMCBSIC96ZXJvIDQ1IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTUgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAwIC9jYSAxID4+Ci9BMiA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAxID4+Ci9BMyA8PCAvVHlwZSAvRXh0R1N0YXRlIC9DQSAxIC9jYSAwLjUgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgPj4KZW5kb2JqCjIgMCBvYmoKPDwgL1R5cGUgL1BhZ2VzIC9LaWRzIFsgMTEgMCBSIF0gL0NvdW50IDEgPj4KZW5kb2JqCjQ2IDAgb2JqCjw8IC9DcmVhdG9yIChNYXRwbG90bGliIHYzLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjcuMSkgL0NyZWF0aW9uRGF0ZSAoRDoyMDIzMDMxNDE2MTQyMlopCj4+CmVuZG9iagp4cmVmCjAgNDcKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMTUzMDQgMDAwMDAgbiAKMDAwMDAxNTA2OSAwMDAwMCBuIAowMDAwMDE1MTAxIDAwMDAwIG4gCjAwMDAwMTUyNDEgMDAwMDAgbiAKMDAwMDAxNTI2MiAwMDAwMCBuIAowMDAwMDE1MjgzIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM0MSAwMDAwMCBuIAowMDAwMDA1MzA1IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwNTI4NCAwMDAwMCBuIAowMDAwMDEzNjcyIDAwMDAwIG4gCjAwMDAwMTM0NjUgMDAwMDAgbiAKMDAwMDAxMjk5NCAwMDAwMCBuIAowMDAwMDE0NzI1IDAwMDAwIG4gCjAwMDAwMDUzMjUgMDAwMDAgbiAKMDAwMDAwNTU2MiAwMDAwMCBuIAowMDAwMDA1ODA1IDAwMDAwIG4gCjAwMDAwMDYxODUgMDAwMDAgbiAKMDAwMDAwNjUwMiAwMDAwMCBuIAowMDAwMDA2ODA3IDAwMDAwIG4gCjAwMDAwMDcxMTEgMDAwMDAgbiAKMDAwMDAwNzQzMyAwMDAwMCBuIAowMDAwMDA3OTAxIDAwMDAwIG4gCjAwMDAwMDgyMjMgMDAwMDAgbiAKMDAwMDAwODM4OSAwMDAwMCBuIAowMDAwMDA4ODAzIDAwMDAwIG4gCjAwMDAwMDg5NDcgMDAwMDAgbiAKMDAwMDAwOTA2NiAwMDAwMCBuIAowMDAwMDA5MzAyIDAwMDAwIG4gCjAwMDAwMDk1OTMgMDAwMDAgbiAKMDAwMDAwOTc0OCAwMDAwMCBuIAowMDAwMDA5ODcxIDAwMDAwIG4gCjAwMDAwMTAxMDQgMDAwMDAgbiAKMDAwMDAxMDUxMSAwMDAwMCBuIAowMDAwMDEwOTA0IDAwMDAwIG4gCjAwMDAwMTA5OTQgMDAwMDAgbiAKMDAwMDAxMTIwMCAwMDAwMCBuIAowMDAwMDExNjEzIDAwMDAwIG4gCjAwMDAwMTE5MzcgMDAwMDAgbiAKMDAwMDAxMjE4NCAwMDAwMCBuIAowMDAwMDEyMzMxIDAwMDAwIG4gCjAwMDAwMTI0OTIgMDAwMDAgbiAKMDAwMDAxMjcwNiAwMDAwMCBuIAowMDAwMDE1MzY0IDAwMDAwIG4gCnRyYWlsZXIKPDwgL1NpemUgNDcgL1Jvb3QgMSAwIFIgL0luZm8gNDYgMCBSID4+CnN0YXJ0eHJlZgoxNTUxNQolJUVPRgo=", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["mu = Tensor([128])\n", "sigma = Tensor([2.0])\n", "\n", "\n", "def discrete_logistic(x, mu, sigma):\n", " return torch.sigmoid((x + 0.5 - mu) / sigma) - torch.sigmoid((x - 0.5 - mu) / sigma)\n", "\n", "\n", "x = torch.arange(256)\n", "p = discrete_logistic(x, mu, sigma)\n", "\n", "# Visualization\n", "plt.figure(figsize=(6, 3))\n", "plt.bar(x.numpy(), p.numpy(), **plot_args)\n", "plt.xlim(96, 160)\n", "plt.title(\"Discrete logistic distribution\")\n", "plt.xlabel(\"Pixel value\")\n", "plt.ylabel(\"Probability\")\n", "plt.show()\n", "plt.close()"]}, {"cell_type": "markdown", "id": "10ab82c4", "metadata": {"papermill": {"duration": 0.033259, "end_time": "2023-03-14T16:14:22.245457", "exception": false, "start_time": "2023-03-14T16:14:22.212198", "status": "completed"}, "tags": []}, "source": ["Instead of the softmax, the model would output mean and standard\n", "deviations for the $K$ logistics we use in the mixture. This is one of\n", "the improvements in autoregressive models that PixelCNN++ [3] has\n", "introduced compared to the original PixelCNN."]}, {"cell_type": "markdown", "id": "f0b274f9", "metadata": {"papermill": {"duration": 0.03334, "end_time": "2023-03-14T16:14:22.312113", "exception": false, "start_time": "2023-03-14T16:14:22.278773", "status": "completed"}, "tags": []}, "source": ["## Conclusion\n", "\n", "In this tutorial, we have looked at autoregressive image modeling, and\n", "implemented the PixelCNN architecture. With the usage of masked\n", "convolutions, we are able to apply a convolutional network in which a\n", "pixel is only influenced by all its predecessors. Separating the masked\n", "convolution into a horizontal and vertical stack allowed us to remove\n", "the known blind spot on the right upper row of a pixel. In experiments,\n", "autoregressive models outperformed normalizing flows in terms of bits\n", "per dimension, but are much slower to sample from. Improvements, that we\n", "have not implemented ourselves here, are discrete logistic mixtures, a\n", "downsampling architecture, and changing the pixel order in a diagonal\n", "fashion (see PixelSNAIL). Overall, autoregressive models are another,\n", "strong family of generative models, which however are mostly used in\n", "sequence tasks because of their linear scaling in sampling time than\n", "quadratic as on images."]}, {"cell_type": "markdown", "id": "2129e328", "metadata": {"papermill": {"duration": 0.037205, "end_time": "2023-03-14T16:14:22.382840", "exception": false, "start_time": "2023-03-14T16:14:22.345635", "status": "completed"}, "tags": []}, "source": ["## References\n", "[1] van den Oord, A., et al.\n", "\"Pixel Recurrent Neural Networks.\"\n", "arXiv preprint arXiv:1601.06759 (2016).\n", "[Link](https://arxiv.org/abs/1601.06759)\n", "\n", "[2] van den Oord, A., et al.\n", "\"Conditional Image Generation with PixelCNN Decoders.\"\n", "In Advances in Neural Information Processing Systems 29, pp.\n", "4790\u20134798 (2016).\n", "[Link](http://papers.nips.cc/paper/6527-conditional-image-generation-with-pixelcnn-decoders.pdf)\n", "\n", "[3] Salimans, Tim, et al.\n", "\"PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications.\"\n", "arXiv preprint arXiv:1701.05517 (2017).\n", "[Link](https://arxiv.org/abs/1701.05517)"]}, {"cell_type": "markdown", "id": "f0504e1e", "metadata": {"papermill": {"duration": 0.03271, "end_time": "2023-03-14T16:14:22.450011", "exception": false, "start_time": "2023-03-14T16:14:22.417301", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/Lightning-AI/lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://www.pytorchlightning.ai/community)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/Lightning-AI/lightning) or [Bolt](https://github.com/Lightning-AI/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/Lightning-AI/lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/Lightning-AI/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "[{height=\"60px\" width=\"240px\"}](https://pytorchlightning.ai)"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Tutorial 10: Autoregressive Image Modeling\n", " :card_description: In this tutorial, we implement an autoregressive likelihood model for the task of image modeling. Autoregressive models are naturally strong generative models that constitute...\n", " :tags: Image,GPU/TPU,UvA-DL-Course\n", " :image: _static/images/course_UvA-DL/10-autoregressive-image-modeling.jpg"]}], "metadata": {"jupytext": {"cell_metadata_filter": "id,colab_type,colab,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16"}, "papermill": {"default_parameters": {}, "duration": 57.322524, "end_time": "2023-03-14T16:14:24.409065", "environment_variables": {}, "exception": null, "input_path": "course_UvA-DL/10-autoregressive-image-modeling/Autoregressive_Image_Modeling.ipynb", "output_path": ".notebooks/course_UvA-DL/10-autoregressive-image-modeling.ipynb", "parameters": {}, "start_time": "2023-03-14T16:13:27.086541", "version": "2.4.0"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"01818e9a25ea4c4f8791153c7d254cce": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "02aac6f47cc049429d9e7dee9b317524": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "0357cee424a042b1a55c4b9b4d6d969b": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "04f9f711df4e4d6eb6f091a5a654fb34": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "06936252278d484e9c7f3506a2ba22fc": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "06a906d73f9e4cc4abe3d5b2653b8bc9": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "06f04a1de53f42e594fa4c69a051b4e0": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_998c021b9a074185845f6bac5d8ede82", "max": 28.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_01818e9a25ea4c4f8791153c7d254cce", "tabbable": null, "tooltip": null, "value": 28.0}}, "0b1a1c27d211473e9afff4216eed9aa2": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_3a32bbd949274a6db3eb0e44940f783a", "placeholder": "\u200b", "style": "IPY_MODEL_fe9dfd7f6e424d47be3fb2605e010193", "tabbable": null, "tooltip": null, "value": " 28/28 [00:02<00:00, 11.61it/s]"}}, "0c497ea77e784611810df5814ce69f51": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "12cf946d79df4a198c8ae8b1f447b917": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_83f5020f3d574236b98c37013748013a", "placeholder": "\u200b", "style": "IPY_MODEL_44f007f8e73f47a7958bf67b2ceed639", "tabbable": null, "tooltip": null, "value": "100%"}}, "140c55663586492dab084df1567f8450": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "151d4062f70d4e628879740e5a4806b8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_2571f3228f274b76a96a3802a771cb36", "placeholder": "\u200b", "style": "IPY_MODEL_3f560107847f49389c7f200feb8b4858", "tabbable": null, "tooltip": null, "value": " 27/28 [00:01<00:00, 11.68it/s]"}}, "1543ec8a46464b3cb895c22690bdf1ce": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "1828e508c5694cf3b30a861835debb9d": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "19d1212dd5e04cb49c0e7108b8dec900": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "1c2b8a900add4dd5acf7dff7d6e2b976": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_06936252278d484e9c7f3506a2ba22fc", "placeholder": "\u200b", "style": "IPY_MODEL_398d91c8830e41c6b646cfb466cfd1b6", "tabbable": null, "tooltip": null, "value": " 4542/4542 [00:00<00:00, 308540.57it/s]"}}, "1c41f5ef72ba45398500957de2612a3f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "1d6b3964c7424a93b41b5b10424e1491": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": "hidden", "width": null}}, "1df41d98473842d0a311bdf1d316100b": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "213006b56c614315b892071fa9ac8001": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "21d4bf833b22495388551ca39743964c": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "2571f3228f274b76a96a3802a771cb36": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "25db831959f548fd9381c36990fab9c0": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_fb74849d43b240d9935a2c9bce5b58fd", "IPY_MODEL_06f04a1de53f42e594fa4c69a051b4e0", "IPY_MODEL_e4e905af47134d0f9df59d00b2335481"], "layout": "IPY_MODEL_2a8be261aa1846c4978d9a501258efc9", "tabbable": null, "tooltip": null}}, "265348bcd5dd44a290a56a961133234a": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "269e6035fd72458dbada712a08c91f2d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_21d4bf833b22495388551ca39743964c", "placeholder": "\u200b", "style": "IPY_MODEL_38fbee8f88ef4df48d2502ac58b7e297", "tabbable": null, "tooltip": null, "value": " 64/64 [00:18<00:00, 2.53it/s]"}}, "2a8be261aa1846c4978d9a501258efc9": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": "hidden", "width": null}}, "2dcb52e8c9194f45a2ba294e584342c0": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_d519dfa9173043d49458e772e9106ead", "IPY_MODEL_da5bb26651bc435baeff2d000018913b", "IPY_MODEL_269e6035fd72458dbada712a08c91f2d"], "layout": "IPY_MODEL_ab05b686f0d745db8e841e20f1c38c46", "tabbable": null, "tooltip": null}}, "30f4593977854cfd8cfb0f2379d9adf6": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": "hidden", "width": null}}, "31d90245833f404f8fd0b298c7ec9b70": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_fd59fd43c1754262a0751e64bc353721", "IPY_MODEL_aabd3a64b2754ced8bafb6c878adae65", "IPY_MODEL_9687e80f4ea4443c98e48be9a9e4c954"], "layout": "IPY_MODEL_06a906d73f9e4cc4abe3d5b2653b8bc9", "tabbable": null, "tooltip": null}}, "3481c83e22284388922615a3e4086cfb": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "34f59697fe4f4f22859549f369fb8116": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "362ea384e29d4f1db30d231b4dd7c1e9": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "368b127df1cc4d32a8a764386d40ed52": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_c74783c555074383b48b4d0b7f6e5b76", "IPY_MODEL_f87d7a1fa8224e17aa40f5634a397575", "IPY_MODEL_95c0ad1ca7134765b381daff083a06fb"], "layout": "IPY_MODEL_91010134429244828d3ca8675d4b1fe1", "tabbable": null, "tooltip": null}}, "38fbee8f88ef4df48d2502ac58b7e297": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "3951538957e94e54afbf4514bf78874f": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "398d91c8830e41c6b646cfb466cfd1b6": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "3a32bbd949274a6db3eb0e44940f783a": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "3e2972d57a584d8caa4c389ea733e5e0": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "3f560107847f49389c7f200feb8b4858": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "40b831bdc42242489a6fcad4ae48c8b3": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "418b15f4e98c47bc965e3234817bc316": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_526efa094480407eb1709ef91d600be3", "IPY_MODEL_aedfb680e317414ca95bd17a4da37307", "IPY_MODEL_0b1a1c27d211473e9afff4216eed9aa2"], "layout": "IPY_MODEL_4bc1cfad01fc4dd99ce97ebb975417de", "tabbable": null, "tooltip": null}}, "4249e64fa01248ed8719ce65e290b2d6": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "44f007f8e73f47a7958bf67b2ceed639": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "4bc1cfad01fc4dd99ce97ebb975417de": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": "hidden", "width": null}}, "4c6208a10b5b4c82b911d994dba528db": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "51b2d984fbc64f1aac12981b991e17b8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_3481c83e22284388922615a3e4086cfb", "max": 4542.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_7dcb42ef08f6485a950329febf578084", "tabbable": null, "tooltip": null, "value": 4542.0}}, "526efa094480407eb1709ef91d600be3": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_1828e508c5694cf3b30a861835debb9d", "placeholder": "\u200b", "style": "IPY_MODEL_c0713dfe654d4e4a93ba2ec56717ceea", "tabbable": null, "tooltip": null, "value": "100%"}}, "5a52020dab6c40c0878f7afc63ab1aa6": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_3951538957e94e54afbf4514bf78874f", "placeholder": "\u200b", "style": "IPY_MODEL_213006b56c614315b892071fa9ac8001", "tabbable": null, "tooltip": null, "value": " 28/28 [00:02<00:00, 11.59it/s]"}}, "5b8d25e810c1495eaa1f59d7fb3db880": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_12cf946d79df4a198c8ae8b1f447b917", "IPY_MODEL_de2239ef5230495d825e4125c695e345", "IPY_MODEL_6b09a36f4860475dbd29eb75410da6ac"], "layout": "IPY_MODEL_fe1e270d3e904386be44937f1d57109e", "tabbable": null, "tooltip": null}}, "630688864e1e433486b8e9a3c4215d6c": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "65047ce3c6ed4e54a39d6dc26e19395c": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "6b09a36f4860475dbd29eb75410da6ac": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_0357cee424a042b1a55c4b9b4d6d969b", "placeholder": "\u200b", "style": "IPY_MODEL_04f9f711df4e4d6eb6f091a5a654fb34", "tabbable": null, "tooltip": null, "value": " 28881/28881 [00:00<00:00, 1768585.02it/s]"}}, "771628e725394e8e97c6776826e02a5f": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "7b4e1fc2028e49758f8758813679b915": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "7dcb42ef08f6485a950329febf578084": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "83f5020f3d574236b98c37013748013a": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8f89073a1a834b3181a376f99992d0e4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "9055f54adb0f4a89b35ead9f42ca6732": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_34f59697fe4f4f22859549f369fb8116", "max": 28.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_19d1212dd5e04cb49c0e7108b8dec900", "tabbable": null, "tooltip": null, "value": 28.0}}, "91010134429244828d3ca8675d4b1fe1": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "95c0ad1ca7134765b381daff083a06fb": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_7b4e1fc2028e49758f8758813679b915", "placeholder": "\u200b", "style": "IPY_MODEL_362ea384e29d4f1db30d231b4dd7c1e9", "tabbable": null, "tooltip": null, "value": " 1648877/1648877 [00:00<00:00, 2108999.12it/s]"}}, "9687e80f4ea4443c98e48be9a9e4c954": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_af691126b7874b32951d46e540ba0722", "placeholder": "\u200b", "style": "IPY_MODEL_dc3f97b71b9f4492b2b02db598898d6e", "tabbable": null, "tooltip": null, "value": " 9912422/9912422 [00:00<00:00, 23001709.75it/s]"}}, "998c021b9a074185845f6bac5d8ede82": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "9cc6f7ca905c4a94a856e7f09207d112": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "a1f3e473ea24416fb91e2d12e4c6100d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_771628e725394e8e97c6776826e02a5f", "placeholder": "\u200b", "style": "IPY_MODEL_4c6208a10b5b4c82b911d994dba528db", "tabbable": null, "tooltip": null, "value": "100%"}}, "a553b6b34f564352946cc8c241adf593": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "a5c8c705bffd4f79991fb8037d77ee68": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "a924c60f89c4421e93f554f034c60e71": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "aa52a74e284345ae926aad0325164205": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "aabd3a64b2754ced8bafb6c878adae65": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_b1d4cc987e0542339551857e7af7863f", "max": 9912422.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_8f89073a1a834b3181a376f99992d0e4", "tabbable": null, "tooltip": null, "value": 9912422.0}}, "ab05b686f0d745db8e841e20f1c38c46": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": "hidden", "width": null}}, "abf9d8b0bcfd4b22817f873c064c37b5": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "aedfb680e317414ca95bd17a4da37307": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_630688864e1e433486b8e9a3c4215d6c", "max": 28.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_abf9d8b0bcfd4b22817f873c064c37b5", "tabbable": null, "tooltip": null, "value": 28.0}}, "af691126b7874b32951d46e540ba0722": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "b1d4cc987e0542339551857e7af7863f": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "b2fa0e0352484f98ac9090ad31a56116": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "c0713dfe654d4e4a93ba2ec56717ceea": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "c240025f9a724233b6b0be875d208cc1": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "c74783c555074383b48b4d0b7f6e5b76": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_e93b09afe68a4db6a77363e070f22828", "placeholder": "\u200b", "style": "IPY_MODEL_dcc2058101ae4fcbab7de7c077afd051", "tabbable": null, "tooltip": null, "value": "100%"}}, "d519dfa9173043d49458e772e9106ead": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_1df41d98473842d0a311bdf1d316100b", "placeholder": "\u200b", "style": "IPY_MODEL_a924c60f89c4421e93f554f034c60e71", "tabbable": null, "tooltip": null, "value": "100%"}}, "da3811299ba1485989875f3b2e221877": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_a1f3e473ea24416fb91e2d12e4c6100d", "IPY_MODEL_51b2d984fbc64f1aac12981b991e17b8", "IPY_MODEL_1c2b8a900add4dd5acf7dff7d6e2b976"], "layout": "IPY_MODEL_ddc3acd4771d4d18b219863a05e483f9", "tabbable": null, "tooltip": null}}, "da5bb26651bc435baeff2d000018913b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_4249e64fa01248ed8719ce65e290b2d6", "max": 64.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_140c55663586492dab084df1567f8450", "tabbable": null, "tooltip": null, "value": 64.0}}, "dc3f97b71b9f4492b2b02db598898d6e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "dc61c791bbc44c5a8d0a9d011354ebad": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_eb283f0beb32413cb3bc2095cbacbb03", "IPY_MODEL_9055f54adb0f4a89b35ead9f42ca6732", "IPY_MODEL_5a52020dab6c40c0878f7afc63ab1aa6"], "layout": "IPY_MODEL_1d6b3964c7424a93b41b5b10424e1491", "tabbable": null, "tooltip": null}}, "dcc2058101ae4fcbab7de7c077afd051": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "ddc3acd4771d4d18b219863a05e483f9": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "de2239ef5230495d825e4125c695e345": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_a5c8c705bffd4f79991fb8037d77ee68", "max": 28881.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_aa52a74e284345ae926aad0325164205", "tabbable": null, "tooltip": null, "value": 28881.0}}, "e1417a8dacf645a392c791b517d5041c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_b2fa0e0352484f98ac9090ad31a56116", "max": 28.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_e8fa9d7bfd594c348c965b3a1f37b248", "tabbable": null, "tooltip": null, "value": 28.0}}, "e362d85d356d4fbead3ab5bfa218935b": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "e4e905af47134d0f9df59d00b2335481": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_e362d85d356d4fbead3ab5bfa218935b", "placeholder": "\u200b", "style": "IPY_MODEL_9cc6f7ca905c4a94a856e7f09207d112", "tabbable": null, "tooltip": null, "value": " 28/28 [00:05<00:00, 8.86it/s]"}}, "e8fa9d7bfd594c348c965b3a1f37b248": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "e93b09afe68a4db6a77363e070f22828": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "eb283f0beb32413cb3bc2095cbacbb03": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_40b831bdc42242489a6fcad4ae48c8b3", "placeholder": "\u200b", "style": "IPY_MODEL_c240025f9a724233b6b0be875d208cc1", "tabbable": null, "tooltip": null, "value": "100%"}}, "f5bf61676010434faf1e705c33370955": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_f7c5d4cf775949fc8712fed9b571674b", "IPY_MODEL_e1417a8dacf645a392c791b517d5041c", "IPY_MODEL_151d4062f70d4e628879740e5a4806b8"], "layout": "IPY_MODEL_30f4593977854cfd8cfb0f2379d9adf6", "tabbable": null, "tooltip": null}}, "f7c5d4cf775949fc8712fed9b571674b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_65047ce3c6ed4e54a39d6dc26e19395c", "placeholder": "\u200b", "style": "IPY_MODEL_02aac6f47cc049429d9e7dee9b317524", "tabbable": null, "tooltip": null, "value": " 96%"}}, "f87d7a1fa8224e17aa40f5634a397575": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_265348bcd5dd44a290a56a961133234a", "max": 1648877.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_a553b6b34f564352946cc8c241adf593", "tabbable": null, "tooltip": null, "value": 1648877.0}}, "fb74849d43b240d9935a2c9bce5b58fd": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_1543ec8a46464b3cb895c22690bdf1ce", "placeholder": "\u200b", "style": "IPY_MODEL_0c497ea77e784611810df5814ce69f51", "tabbable": null, "tooltip": null, "value": "100%"}}, "fd59fd43c1754262a0751e64bc353721": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_3e2972d57a584d8caa4c389ea733e5e0", "placeholder": "\u200b", "style": "IPY_MODEL_1c41f5ef72ba45398500957de2612a3f", "tabbable": null, "tooltip": null, "value": "100%"}}, "fe1e270d3e904386be44937f1d57109e": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "fe9dfd7f6e424d47be3fb2605e010193": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/10-autoregressive-image-modeling/autoregressive_image_modeling.svg\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/10-autoregressive-image-modeling/masked_convolution.svg\")
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/10-autoregressive-image-modeling/pixelcnn_blind_spot.svg\"){kind=tracking}
![](\"https://github.com/Lightning-AI/lightning-tutorials/raw/main/course_UvA-DL/10-autoregressive-image-modeling/PixelCNN_GatedConv.svg\"){kind=tracking}
![](\"https://raw.githubusercontent.com/vdumoulin/conv_arithmetic/master/gif/dilation.gif\")