{"cells": [{"cell_type": "markdown", "id": "22b293d7", "metadata": {"papermill": {"duration": 0.012549, "end_time": "2022-05-12T12:18:58.534467", "exception": false, "start_time": "2022-05-12T12:18:58.521918", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 9: Normalizing Flows for Image Modeling\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2022-05-12T13:44:24.674574\n", "\n", "In this tutorial, we will take a closer look at complex, deep normalizing flows.\n", "The most popular, current application of deep normalizing flows is to model datasets of images.\n", "As for other generative models, images are a good domain to start working on because\n", "(1) CNNs are widely studied and strong models exist,\n", "(2) images are high-dimensional and complex,\n", "and (3) images are discrete integers.\n", "In this tutorial, we will review current advances in normalizing flows for image modeling,\n", "and get hands-on experience on coding normalizing flows.\n", "Note that normalizing flows are commonly parameter heavy and therefore computationally expensive.\n", "We will use relatively simple and shallow flows to save computational cost and allow you to run the notebook on CPU,\n", "but keep in mind that a simple way to improve the scores of the flows we study here is to make them deeper.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [![Open In Colab](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHUAAAAUCAYAAACzrHJDAAAIuUlEQVRoQ+1ZaVRURxb+qhdolmbTUVSURpZgmLhHbQVFZIlGQBEXcMvJhKiTEzfigjQg7oNEJ9GMGidnjnNMBs2czIzajksEFRE1xklCTKJiQLRFsUGkoUWw+82pamn79etGYoKek1B/4NW99/tu3e/dquJBAGD27NkHALxKf39WY39gyrOi+i3xqGtUoePJrFmznrmgtModorbTu8YRNZk5cybXTvCtwh7o6NR2KzuZMWNGh6jtVt7nA0ymT5/eJlF9POrh7PAQl6s8bGYa3PUum//htmebVtLRqW0q01M5keTk5FZFzU0oRle3+zxwg5Hgtb+PZiL/ZVohxCI+hL5JgjmfjPxZ26+33BG3dA+ealHPM4gQAo5rU59gsI8bRvl54t3Ca62mvHyUAhtOlLd5WSQpKcluBjumnoCLs1EARkVd9E8l3p9y2i7RbQ1B6pFwu/YDgW8KbHJHMTQrwnjz2oZm9M4pavOCfo5jWrgCaaMVcMs6/pNhDr0+AMN93XlxV7R6DNpyzi7W/OE+yIrsjU6rTrbKV5cd/pNyItOmTbMp6sbBB+EqaYJY4cWE3VUciNt1TpgfcRFv71Fi54xT5kSoyLvOBEJMOMxWXkFlBeBSX4u6Zkcs+3KszYRtiapbNRqF31UgetVuc8z9vBXIv1qD+F1f83B6uDlCUyfsZGepGPpmg01OB7EITQbhS9ribKy+DmP1DUiClLz4bnIHVOqa7BY+Z1wg5g3zgUvyehiNpnJKxSLc/ts76LKm0BzX3c0RNy1yXjDcB5lWoro4iNHQxM+f1kWeWQARAWQS++trISJTp061Kep25X/MycwtjuctSC5rxo7ppi7VNUox5+PhPHtrsS2O1qJ6yx1QujQUzm9sh6hbkBlvvGcN8hYnwjUjH6kjfZEd5c/jitz5Jc5U3ENnFynKl4eB7nyEgP2UZ+Yz3/rVEbyYr27qELrtC4FIC0J7sc7xWnmccdHfRRTs0VB+cA4lt+oFcRR/wUeH8FG5w2Mbx8FQ8TXEvv1xYf4wBP3O2WyL3/UVjpXWgIqaFeUPr+wTmDvUB7njH6/bOv+HRg4SqioAg5GDe1aB3ZeMTJkyRSBqkLsWqSEm0fZVBEN94zEZnYvrdx1JL5cxe+a+AbhSJecRRHW/ikTFRTa38dtQlNZ5CRKwFvUtZU/kvBoEF9Uxni/XqIM+dwKbTw3rhcxIf7gmr2M+H6SMwx8iBzJbw5oxeG3Lv5FX9B3AGaHPS8e8z77H7v9VMpvPG5ug1enh7eGK8h0LBTwUb+GInqzInlRUK65DmTPQu4c3+uQKjwKK77zwUxBX4Tq7yR1RuiwUsqlrABCM6esHdXoy47fk4+prYKy8ZF574x4V5BnHQBuf4g9Z9ld8U36L2aktZNNplNfw7zotwWTy5MkCUft4aLEopJj5/OPHl1BQqeAVOnHgNSQOqmBzq9V9cfEm/yx5ubMGKS9cYPZ3vx2OS/c6PVHUuUO7Y1Pci3BO/1zgq18byebfGemLtNF+6JRtOvMk926ibussZqM+1mNz4TWkH7rCbM5phwGRGDAaoF8fY5OHFnlldAA8sgoEXKnDukA1NgSeNjqkJT9brbN4pC9WRweYXyLugR73c+MYvyWfu0yC6+mjzN1Isfw3FKJS98CU/zI1IHFkFPR52cHL2FJk0sB6kMTERIGo9GzcPkLNfA0cwdwi/hfEYO86ZMd9w+y1egfM2T2Eh/vesMNwljSzuZRT420SW3eqy8N6aHMmwmnFUZ7/PGVPbIoNZvNU1BURdHs0bT2+HjL8sDSM2e6vi4Lj5NW8WOLVA6RTT2azxLV+bglaFNqLieqemS/gWkw7NyoAHo+2dEsiivengjKsPFoqWOvbSh/kxPaxyW/JRzH2Fl3EzD9/xjAefJqB3usKUFn/0Gb+S/d/jy3FN2yLOmnSJJtn6oehByEiHPSeXnDxFGPRnoFoaBJjcdQlbDwcjL1zTNuQpoxD7R0OG0uUTMi0fkVwdzBdYIwcwZunxrVJVLplNm54BZp7jfDfYLoNyqQi1K6KxIdHzmN+QQ2WjFIwUT2zTGdlRXo4NFXVUO4sgX5dFC7f0aP/ZlNeUjFBuL8Xjl6uRuP6aMjSjpjzsH62FDU7JhBuGccEXIvDfJFFBc/gHw80dklfCVYnRaDfpiJcutPA4F7qJsfJeUPQI+1fqMlNhFx1FM0GDqkjFVg7NojlQ0Vt4aM5ReSqcbpaCg8nCW5lRsBvbT4T1TLfFptsfh7gItzuKTdJSEiwKSrt1vcmnEXXrsLbYnWDA1bu+z2WKy9Arq+1KRqdfKsoBo0GcdtEpS/B1bO4v0cFiUhkjskvKcMrWwtAPHuwQq8Z+4LZ1vTQANfXt4J0DwZX9gWa9qh4XDM/voC9JXfwYEMMHJcfNtusn82ihvliVUwg5KrPGVf6GH94ZJpEZBen6EC4qYTHA1dXhW0JIex8txzv//c8lhzXIi/BFxOH9jGbQhZsRalTIBZZ8KkGyZAxeRQvXkFF1TWz/Hm46jNYUnjPbt3JxIkT7f6dSj8qfJJyVvBxgaIlblOyjtysNHWN9fjjqWi7glJfW3/S0Hlj2XnA8PhKT9w6g3Qx3XiXhvuxQsuT1proxBKI/AaZqY1Xz5muvY8G8XkRRCaHsfQsRAFDH/tZPbcYuHotOG0FRIqB4HR3wNVoIPLtz8ycTguu+jpEigE218vd1YCr5m+HpHMvEI9u4LTXwNWaLjl0iPwGAmIpeHx1VeCqTJdPs1/vweweQPO3HC24NhOhnTphwoQnfv6QSY2ICbkNmdSA4h87oaLaiYfn5diIEd4att2erOwJXbPUHp953p6orQVSUVWRAXBT8c/dJ5L9xhzaJGp71GR/wFP8P5V2z10NSC9T93QM2xUg8fHxT+zU9ijeU4naHon8CjFJXFzc8/kn+dN06q9QgF98SYSo2Xen2NjYZy5sR6f+4nLSK5Iam2PH/x87a1YN/t5sBgAAAABJRU5ErkJggg==){height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/09-normalizing-flows.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)"]}, {"cell_type": "markdown", "id": "79f5e211", "metadata": {"papermill": {"duration": 0.00964, "end_time": "2022-05-12T12:18:58.554488", "exception": false, "start_time": "2022-05-12T12:18:58.544848", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "66ca3c42", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2022-05-12T12:18:58.575803Z", "iopub.status.busy": "2022-05-12T12:18:58.575120Z", "iopub.status.idle": "2022-05-12T12:19:02.710809Z", "shell.execute_reply": "2022-05-12T12:19:02.709792Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 4.149137, "end_time": "2022-05-12T12:19:02.713186", "exception": false, "start_time": "2022-05-12T12:18:58.564049", "status": "completed"}, "tags": []}, "outputs": [], "source": ["! pip install --quiet \"ipython[notebook]\" \"seaborn\" \"tabulate\" \"torchvision\" \"setuptools==59.5.0\" \"pytorch-lightning>=1.4\" \"matplotlib\" \"torchmetrics>=0.7\" \"torch>=1.8\""]}, {"cell_type": "markdown", "id": "ba2a1738", "metadata": {"papermill": {"duration": 0.010226, "end_time": "2022-05-12T12:19:02.734320", "exception": false, "start_time": "2022-05-12T12:19:02.724094", "status": "completed"}, "tags": []}, "source": ["
\n", "Throughout this notebook, we make use of [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/).\n", "The first cell imports our usual libraries."]}, {"cell_type": "code", "execution_count": 2, "id": "6fb4c201", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:02.755979Z", "iopub.status.busy": "2022-05-12T12:19:02.755562Z", "iopub.status.idle": "2022-05-12T12:19:07.128426Z", "shell.execute_reply": "2022-05-12T12:19:07.127622Z"}, "papermill": {"duration": 4.386077, "end_time": "2022-05-12T12:19:07.130032", "exception": false, "start_time": "2022-05-12T12:19:02.743955", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/usr/lib/python3.8/site-packages/apex/pyprof/__init__.py:5: FutureWarning: pyprof will be removed by the end of June, 2022\n", " warnings.warn(\"pyprof will be removed by the end of June, 2022\", FutureWarning)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["/usr/local/lib/python3.8/dist-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for type is zero.\n", " setattr(self, word, getattr(machar, word).flat[0])\n", "/usr/local/lib/python3.8/dist-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for type is zero.\n", " return self._float_to_str(self.smallest_subnormal)\n", "/usr/local/lib/python3.8/dist-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for type is zero.\n", " setattr(self, word, getattr(machar, word).flat[0])\n", "/usr/local/lib/python3.8/dist-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for type is zero.\n", " return self._float_to_str(self.smallest_subnormal)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["WARNING:root:Bagua cannot detect bundled NCCL library, Bagua will try to use system NCCL instead. If you encounter any error, please run `import bagua_core; bagua_core.install_deps()` or the `bagua_install_deps.py` script to install bundled libraries.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["/tmp/ipykernel_5302/3974796986.py:28: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n", " set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "Global seed set to 42\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Using device cuda:0\n"]}], "source": ["import math\n", "import os\n", "import time\n", "import urllib.request\n", "from urllib.error import HTTPError\n", "\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pytorch_lightning as pl\n", "import seaborn as sns\n", "import tabulate\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "import torch.utils.data as data\n", "import torchvision\n", "from IPython.display import HTML, display, set_matplotlib_formats\n", "from matplotlib.colors import to_rgb\n", "from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint\n", "from torch import Tensor\n", "from torchvision import transforms\n", "from torchvision.datasets import MNIST\n", "from tqdm.notebook import tqdm\n", "\n", "%matplotlib inline\n", "set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "matplotlib.rcParams[\"lines.linewidth\"] = 2.0\n", "sns.reset_orig()\n", "\n", "# Path to the folder where the datasets are/should be downloaded (e.g. MNIST)\n", "DATASET_PATH = os.environ.get(\"PATH_DATASETS\", \"data\")\n", "# Path to the folder where the pretrained models are saved\n", "CHECKPOINT_PATH = os.environ.get(\"PATH_CHECKPOINT\", \"saved_models/tutorial11\")\n", "\n", "# Setting the seed\n", "pl.seed_everything(42)\n", "\n", "# Ensure that all operations are deterministic on GPU (if used) for reproducibility\n", "torch.backends.cudnn.determinstic = True\n", "torch.backends.cudnn.benchmark = False\n", "\n", "# Fetching the device that will be used throughout this notebook\n", "device = torch.device(\"cpu\") if not torch.cuda.is_available() else torch.device(\"cuda:0\")\n", "print(\"Using device\", device)"]}, {"cell_type": "markdown", "id": "da0e0e19", "metadata": {"papermill": {"duration": 0.010289, "end_time": "2022-05-12T12:19:07.151414", "exception": false, "start_time": "2022-05-12T12:19:07.141125", "status": "completed"}, "tags": []}, "source": ["Again, we have a few pretrained models. We download them below to the specified path above."]}, {"cell_type": "code", "execution_count": 3, "id": "58b482d6", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:07.173932Z", "iopub.status.busy": "2022-05-12T12:19:07.173273Z", "iopub.status.idle": "2022-05-12T12:19:07.720437Z", "shell.execute_reply": "2022-05-12T12:19:07.719635Z"}, "papermill": {"duration": 0.561044, "end_time": "2022-05-12T12:19:07.722543", "exception": false, "start_time": "2022-05-12T12:19:07.161499", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/MNISTFlow_simple.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/MNISTFlow_vardeq.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/MNISTFlow_multiscale.ckpt...\n"]}], "source": ["# Github URL where saved models are stored for this tutorial\n", "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/\"\n", "# Files to download\n", "pretrained_files = [\"MNISTFlow_simple.ckpt\", \"MNISTFlow_vardeq.ckpt\", \"MNISTFlow_multiscale.ckpt\"]\n", "# Create checkpoint path if it doesn't exist yet\n", "os.makedirs(CHECKPOINT_PATH, exist_ok=True)\n", "\n", "# For each file, check whether it already exists. If not, try downloading it.\n", "for file_name in pretrained_files:\n", " file_path = os.path.join(CHECKPOINT_PATH, file_name)\n", " if not os.path.isfile(file_path):\n", " file_url = base_url + file_name\n", " print(\"Downloading %s...\" % file_url)\n", " try:\n", " urllib.request.urlretrieve(file_url, file_path)\n", " except HTTPError as e:\n", " print(\n", " \"Something went wrong. Please try to download the file from the GDrive folder, or contact the author with the full output including the following error:\\n\",\n", " e,\n", " )"]}, {"cell_type": "markdown", "id": "153c5e1a", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.010051, "end_time": "2022-05-12T12:19:07.743999", "exception": false, "start_time": "2022-05-12T12:19:07.733948", "status": "completed"}, "tags": []}, "source": ["We will use the MNIST dataset in this notebook.\n", "MNIST constitutes, despite its simplicity, a challenge for small generative models as it requires the global understanding of an image.\n", "At the same time, we can easily judge whether generated images come from the same distribution as the dataset\n", "(i.e. represent real digits), or not.\n", "\n", "To deal better with the discrete nature of the images, we transform them\n", "from a range of 0-1 to a range of 0-255 as integers."]}, {"cell_type": "code", "execution_count": 4, "id": "b7465ad2", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:07.766358Z", "iopub.status.busy": "2022-05-12T12:19:07.765872Z", "iopub.status.idle": "2022-05-12T12:19:08.608429Z", "shell.execute_reply": "2022-05-12T12:19:08.607691Z"}, "papermill": {"duration": 0.855986, "end_time": "2022-05-12T12:19:08.610270", "exception": false, "start_time": "2022-05-12T12:19:07.754284", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n", "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /__w/1/s/.datasets/MNIST/raw/train-images-idx3-ubyte.gz\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "6512366631a949678ad0cfd766ffba0a", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/9912422 [00:00 make them a tensor and discretize\n", "transform = transforms.Compose([transforms.ToTensor(), discretize])\n", "\n", "# Loading the training dataset. We need to split it into a training and validation part\n", "train_dataset = MNIST(root=DATASET_PATH, train=True, transform=transform, download=True)\n", "pl.seed_everything(42)\n", "train_set, val_set = torch.utils.data.random_split(train_dataset, [50000, 10000])\n", "\n", "# Loading the test set\n", "test_set = MNIST(root=DATASET_PATH, train=False, transform=transform, download=True)\n", "\n", "# We define a set of data loaders that we can use for various purposes later.\n", "# Note that for actually training a model, we will use different data loaders\n", "# with a lower batch size.\n", "train_loader = data.DataLoader(train_set, batch_size=256, shuffle=False, drop_last=False)\n", "val_loader = data.DataLoader(val_set, batch_size=64, shuffle=False, drop_last=False, num_workers=4)\n", "test_loader = data.DataLoader(test_set, batch_size=64, shuffle=False, drop_last=False, num_workers=4)"]}, {"cell_type": "markdown", "id": "db5d08e8", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.010863, "end_time": "2022-05-12T12:19:08.633305", "exception": false, "start_time": "2022-05-12T12:19:08.622442", "status": "completed"}, "tags": []}, "source": ["In addition, we will define below a function to simplify the visualization of images/samples.\n", "Some training examples of the MNIST dataset is shown below."]}, {"cell_type": "code", "execution_count": 5, "id": "3c1a1c0b", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:08.657121Z", "iopub.status.busy": "2022-05-12T12:19:08.656514Z", "iopub.status.idle": "2022-05-12T12:19:08.814015Z", "shell.execute_reply": "2022-05-12T12:19:08.813306Z"}, "papermill": {"duration": 0.171707, "end_time": "2022-05-12T12:19:08.816069", "exception": false, "start_time": "2022-05-12T12:19:08.644362", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzQxLjY3NDgzODcwOTcgMTgwLjcyIF0KL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSIC9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTIgMCBSID4+CnN0cmVhbQp4nFWOSw7CMAxE9z7FnCDfKkmXQKWIZWHBAaJQiCioVKLXx61AhcWzPJbHHtnk1zXlQ9xidyS5qjSSRmE6KBRmgkZkOlKserKVFs5XwdYsb79SByW84Zla2wvRmQZ4YRas4TpvB69qD+2csAbPjBPukBv+MvKrwkx8PeI/2LD4HeYgH+v3cOoh9xrNAy219AYPKzF0CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQ4CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyMjIgKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/X19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Ofn5+bm5uXl5eTk5OPj4+Li4uHh4eDg4N/f397e3t3d3dzc3Nvb29ra2tnZ2djY2NfX19bW1tXV1dTU1NPT09HR0dDQ0M/Pz87Ozs3NzczMzMvLy8nJycfHx8bGxsXFxcTExMPDw8LCwsHBwcDAwL+/v76+vr29vby8vLu7u7q6urm5ubi4uLe3t7a2trW1tbS0tLOzs7KysrCwsK+vr66urq2traysrKqqqqmpqaioqKenp6WlpaSkpKKioqGhoaCgoJ6enpycnJqampmZmZiYmJeXl5aWlpWVlZSUlJOTk5GRkY+Pj42NjYuLi4qKiomJiYiIiIeHh4aGhoWFhYSEhIODg4KCgoCAgH5+fn19fXx8fHt7e3l5eXh4eHd3d3Z2dnV1dXR0dHNzc3JycnFxcXBwcG9vb25ubmpqamdnZ2ZmZmRkZGNjY2JiYmBgYF9fX15eXl1dXVxcXFxcXFtbW1paWllZWVhYWFdXV1ZWVlVVVVRUVFNTU1FRUVBQUE9PT0xMTEtLS0pKSklJSUhISEdHR0ZGRkVFRURERENDQ0JCQkFBQUBAQD8/Pz4+Pjw8PDs7Ozo6Ojk5OTg4ODc3NzU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLiwsLCsrKyoqKlwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiAgIB8fHx4eHh0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExISEhERERAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNDU1IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCAyMzEgL0xlbmd0aCAxNCAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NTUgPj4Kc3RyZWFtCnic7Z37nxZVAcbZlQphIVuMELQ0tUASgVq20iKoKEouEXQxW4LASLpIUWJYQRchFoKSpTQNdEOWDYtNkizUoFrSYLm4UCjwt/Q87MyH2XnPmTlzeZfl8Hx/AeedmWfOfF98z5wz55wB3xQ+MOBiX4AoBXn0A3n0A3n0A3n0A3n0g9Djub5CeVXJk0c/8uTRjzx59CNPHv3Ik0c/8uTRjzx59CNPHi10dna+HzwB+iQvL/KYjDw65fb7PHl0yu33eZepx4Ng+/btdaAGDACTJ0+25xbNcyZPXjeYOnUqCzILVD2vCPJoRx7l8RL2+F0wHdT2prGx0Z5bJC8TWfNeAlNATQ9rQFXziiKPZuRRHvPkFUUezVxOHv8HWBdoaWlpAINAbSVXXXXVT4E5t7yCpJA1bwYIHF4NWNSq5hVFHs3IozzmySuKPJq5XDz+ExgqNQYK1HMeAmxKWLhwoX0n3uJXQfKpspbvnSDwaPgWppM172fgJoAbxsw7AUt/7NixTHnyGEMe5TFPnjzKYyQvm8eVK1d+HKQ7JM3NzXnKiRIcuxXwFJs3bzbs8UewdOnSN4HxALWtltRyupXv3AWPdXV1W4HzcZnzjh49Og+wknhFDyxx8Ncrxo0btx0458ljDHmUx0x5fe7x7Nmz/wHLweDBgxPEDRs2bCd4JuDMmTN5yvkvEJ5x7dq13ES1vITdu3d/GYwAkdh7QGo5029IQOhxypQpzsfkysOPIIW9GfwY3H///VGPADd78A7glCePMeRRHjPlyaM8VuY5eOzu7k6QBz4BPgXa29tLKCfqNpvDMw8fPnwBeBswR78XsKW3QF6UBx98cCCgx0WLFrkdkyfvHwDPjLTVHty1/fv3/zJgKghcXg+6urrS8+QxijzKY5Y8eZRHc14hj42NjXx/5RVQXjmpha2MfBU2lnfllVeyjAtBsCW5juOUF8IXcyZOnBi0rdZ0dnY6lylz3mqAq/8KMLQQcxO+UfgSD2cZm5qa0vPkMUQe5TFrXp96PAX4UU+nSpR6MAkcOnQodgz/x9+V+D905/vKR9FVq1a9I4CPr9u2beMnnwW15x9Xh7EXLfk0znmbQM0FAo9bwMM9/BqknyY17znAGzhy5EjeLcte+KdxI+CVoOaRniePIfIYQR7lMYY8luSR/V+VdZsbQCtg5WZtBaycfBTw74888kiecibzGGDHI67ErXvQOY9vcEY8Tp8+fSbgQ1ywhT2F3LQo8dEyNY8txEEPY8LV4A6HXZIbN25M2FEeY8ijPGbJk0cij+a8bB6HDh3KHjG+mDcbVFqOMmLECFYPspYzmfeAoKp1AKQfkNejnWtAR0dH3rw5gIpWr15t2eMMmDt3Lne6DZw+fTq9fPIYIo/ymCVPHok8mvMSPIbttBEGDhx4LTA0YZvho3obyFLOZEKPS5YscTsgr0d8C0dWgi/yUA5YQGWv09ySnph3FFwHqMh+JRynH/Q/Jr+bG8mTxxB5lEfXvIvikaVxspXMk8C1nCls2bLljeB94MSJE27H5PF4Bzh+/LhhJz45801F7PQCyJr3bxC+1lj5WHgYLAavB8FO6ZctjzHkUR5d8+TxPPJozbtkPOKJrQMOecZMczDm8fg9YN6JoxPGAKi0dB6m5vH5kaWIdCvySXjTpk2sQEVu3MLE0dixPHkMkUd5zJInj/Joz8vtkb1xEyZM4FstHOlGW6gdxHbio/NukKWcFn4CgrP+Cjgfl8fjh0BlPQdbjnMmSNZzdu7cmTdvGWAVZsiQIR8I4EBAbGEdjuMr3g3wn/JoKGf6jvIoj1ny5FEe7XkJHpN7GPmSzKoe7gXmnb4BspbTAB+QOcMKzsi1UZIHyuXN+wyItJPju3IoeD+X/YFtbW0fBvxkLCiQx2m47gb19fVhkwD/TeCrEw4iD7SuA87lk8cQeZTHLHl97tH83qM7t9xyC9+Bz1pOA3l/GzPlcVKQC3Pokg8CDofmXY9sXgnKyQsHIP8OBFvZAst/IrD8InAunzyGyKM85syTR3fk8WnAt3Gy+hsNeGXmGdrzePwY4JlHjRq1H2Q6NlPeqVOnbgY1Rlgb4YDw10BJeZX8ArDq4zxdiDwakEd5zJNXiTwml9NtZy89kt+CLCo5D8pvQGquc8FeBu8CPPu0adOcj8ubx4oUvzcRgXxfZsOGDRxiXn5ejDuCRmrUftwOkEcz8iiPefJiVMkjaWtrS5b3VvBkD6eTBwedy15OTirPEP7q2kc5lZdXlAJ5fJOSz478fZTHgnlFkUcz8uiUJ48l0389lkvWvE8DemQFoC/yilIg788g7JKUx4J5RZFHM/LolCePJSOPBp599tnXAXrk/O8LFixImCKxhLwSkEcD8uiaJ48lI48G5NE1r3973Lt3b9jYwLVYnn/++ermlUCBvP+CzwMuULZnz55MefJYMvJoQB5d8/q3R+W55smjH3ny6EeePPqRJ49+5MmjH3ny6EeePPqRF3oUlzby6Afy6Afy6Afy6Afy6Afy6Ad6fvQjTx79yJNHP/Lk0Y88efQjTx79yJNHP/Lk0Y88efQjTx79yJNHP/Lk0Y88efQjTx79yJNHP/L6r8e/gYd6uBpw7FxkfcTm5uaS8yrhOiy3gh8A54Mq854BE0BNTc0N4JPgO+CBBx5g2TgzL1eymjFjRjjNJNeBnDlz5hfBUeCUJ4925DE9N31HeZTHPHmVyGN6rn2HbYCLInIZ4oG9CabtIPX19dzp6+AUKJBn50eA95VzMDofFMt79NFHx4Hvgw0bNiRf6lnARSZ5B1asWMFlIIeAOXPmcFnm5Dx5tCOPCcijFXmUx8wed4HYps3A+fjkvNbWVhiqD6TV2j1e2FK7GOTNS4TLJNPjG4DzQbE8iHsF5Inn3NJcJrXm/KKittXZ5DEVeUxAHl2poseGhgY+3MS2chEsBjqdIjmvq6vrOkBFXH1xSm+CLW8HEY+8zfYnvIvssQBcSPkLQB7T8xKRxwTk0RV5dM9LxF+PBw4cMHikQ8PmxFz7Dg+DdcC+RwdYtmxZtDJ033335c2z0N3d3Qjo8RrgfFxJHvft2zcZMH7evHn4encl58mjBXlMRh6dkMeseRYuQ49sByjPoxutra1Bj+T5LsnyPT7++OPhOp4/BM7HFSgf50N+DnwboGrF5UPZfnzkyJH0PHm0II/JyKOVvvOIH0I+LBo+uQgeq/v72JceHwN33XXXWwDz+GdTU9NfgHOePFqQR6dctzw78tg7Tx4teO2xoaHB0P9I6BG1IOdctzw7Vfc4ceLE6nqEO8hqCtfvwq3le49cZPo1kDVPHi3Io1Ou83VakMfeefJowWuPs2fPtnzCNlan13QuFY/jx4+vrsdJkyaFAXNAW1tb5muM5MmjBXl0ynW+Tgvy2DtPHi1465GvidpfwuGndsuVuZmu1cD8+fMvcY8nT578A/gqGAlQiOsBh9EdBlnz5NGCPDrlOl+nBXnsnefu0d6Cyv7Ha3uAytnckX8fcIGI4YIeT4A7QTAsmScfCrZu3Wo5IE/edlBXV0eHHAr9d+B8bJ68lwCU3gs4wGo0wFeH4wKc8+TRgDzKozw6IY/mPDePNNUAYptQtxnQG1Z3+MkBsLkXRcoZ4WkQGTdHh7eD1HJmCmkGKAw9LgWZji1YPnY6Tup5tHSbJkQe7cijPMqjE/JoznPzyGoL1ezatWtxRB6qL2xXDT7ZZa0IGXKdChWno6ODM0JFPH4OlJv3KmB7J24k6xtuL8kUyDPA2TmgEjWtOk7d4ZQnjzHkUR7z5BmQxxLy/PUIcfTY0DOonH9YJlopz+M+8C2wZs2aVjAf8E88k0cccp6ohImesuRFYO0iaB+/DTgflzfPwuHDh9l6zrFzTnnyGEMe5TFPnoWSPR4IRsoFTabmdx5DzCMGzLmWT1taWviTdzNIns8KDveDonmVPAEGA3hkif8KnI+153WCTKcBXwO8CU899VR6njzGkEd5tOfJozyW4TEQ6DajIx4nLaMGKnPtn9ZGuhZrexPZMnbsWM7Snn5ReeodowA8zgXs78x0bGXeEuA83VcETkvC6taOHTvS8+TRgDzKozwGn8pjQJkeM011HDS1Ju9kz2NdINJ6mlzPCeCCJnnzzPwccMYT3MD0xk2nvI+AqWDPnj2ZzvR7QI/Lly9Pz5PHGPIoj+a8/uCxfOx5L4De0tI9lt9vxQWtgsbVl0G2whnz+Pb/HWDQoEFsLuViVivB+vXr+b6j5TR4Nr4JsIzt7e3pefIYQx7dkEd5lEdrXj/yyDcqFi9eHPN4I/gSeBE0Nzdzbttgh+r0PwZ1nJpp06ZlK1hiHh9D8RzIJlOOjmPAmDFjOOjg9gBUhNYEcIb90aNHs4wrgFOePMaQRzfkUR7l0ZrXjzySjo6OEeBuQI8o9p9AZAfekHU9VKf/kYPyyvcYgaPwuK7KwYMH94J7QE0ls2bNSlh8pTJPHmPIoxvyKI/n5NGW1888Ki9nnjz6kSePfuTJox958uhHnjz6kSePfuTJox958uhHnjz6kSePfuTJox958uhHnjz6kSePfuSFHsWljTz6gTz6gTz6gTz6gTz6gTz6wf8BhQQ12wplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjM0NDQKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMjA1MTIxNDE5MDgrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My41LjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My41LjIpID4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDUxNTUgMDAwMDAgbiAKMDAwMDAwMDYwNyAwMDAwMCBuIAowMDAwMDAwNjI4IDAwMDAwIG4gCjAwMDAwMDA2ODggMDAwMDAgbiAKMDAwMDAwMDcwOSAwMDAwMCBuIAowMDAwMDAwNzMwIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM0NCAwMDAwMCBuIAowMDAwMDAwNTg3IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU2NyAwMDAwMCBuIAowMDAwMDAwNzYyIDAwMDAwIG4gCjAwMDAwMDUxMzQgMDAwMDAgbiAKMDAwMDAwNTIxNSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDE1IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAxNiA+PgpzdGFydHhyZWYKNTM3MgolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:08.697788\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["def show_imgs(imgs, title=None, row_size=4):\n", " # Form a grid of pictures (we use max. 8 columns)\n", " num_imgs = imgs.shape[0] if isinstance(imgs, Tensor) else len(imgs)\n", " is_int = imgs.dtype == torch.int32 if isinstance(imgs, Tensor) else imgs[0].dtype == torch.int32\n", " nrow = min(num_imgs, row_size)\n", " ncol = int(math.ceil(num_imgs / nrow))\n", " imgs = torchvision.utils.make_grid(imgs, nrow=nrow, pad_value=128 if is_int else 0.5)\n", " np_imgs = imgs.cpu().numpy()\n", " # Plot the grid\n", " plt.figure(figsize=(1.5 * nrow, 1.5 * ncol))\n", " plt.imshow(np.transpose(np_imgs, (1, 2, 0)), interpolation=\"nearest\")\n", " plt.axis(\"off\")\n", " if title is not None:\n", " plt.title(title)\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "show_imgs([train_set[i][0] for i in range(8)])"]}, {"cell_type": "markdown", "id": "3792e705", "metadata": {"papermill": {"duration": 0.012654, "end_time": "2022-05-12T12:19:08.842792", "exception": false, "start_time": "2022-05-12T12:19:08.830138", "status": "completed"}, "tags": []}, "source": ["## Normalizing Flows as generative model\n", "\n", "In the previous lectures, we have seen Energy-based models, Variational Autoencoders (VAEs)\n", "and Generative Adversarial Networks (GANs) as example of generative models.\n", "However, none of them explicitly learn the probability density function $p(x)$ of the real input data.\n", "While VAEs model a lower bound, energy-based models only implicitly learn the probability density.\n", "GANs on the other hand provide us a sampling mechanism for generating new data, without offering a likelihood estimate.\n", "The generative model we will look at here, called Normalizing Flows, actually models the true data distribution\n", "$p(x)$ and provides us with an exact likelihood estimate.\n", "Below, we can visually compare VAEs, GANs and Flows\n", "(figure credit - [Lilian Weng](https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html)):\n", "\n", "
\n", "\n", "The major difference compared to VAEs is that flows use *invertible* functions $f$\n", "to map the input data $x$ to a latent representation $z$.\n", "To realize this, $z$ must be of the same shape as $x$.\n", "This is in contrast to VAEs where $z$ is usually much lower dimensional than the original input data.\n", "However, an invertible mapping also means that for every data point $x$, we have a corresponding latent representation\n", "$z$ which allows us to perform lossless reconstruction ($z$ to $x$).\n", "In the visualization above, this means that $x=x'$ for flows, no matter what invertible function $f$ and input $x$ we choose.\n", "\n", "Nonetheless, how are normalizing flows modeling a probability density with an invertible function?\n", "The answer to this question is the rule for change of variables.\n", "Specifically, given a prior density $p_z(z)$ (e.g. Gaussian) and an invertible function $f$,\n", "we can determine $p_x(x)$ as follows:\n", "\n", "$$\n", "\\begin{split}\n", " \\int p_x(x) dx & = \\int p_z(z) dz = 1 \\hspace{1cm}\\text{(by definition of a probability distribution)}\\\\\n", " \\Leftrightarrow p_x(x) & = p_z(z) \\left|\\frac{dz}{dx}\\right| = p_z(f(x)) \\left|\\frac{df(x)}{dx}\\right|\n", "\\end{split}\n", "$$\n", "\n", "Hence, in order to determine the probability of $x$, we only need to determine its probability in latent space,\n", "and get the derivate of $f$.\n", "Note that this is for a univariate distribution, and $f$ is required to be invertible and smooth.\n", "For a multivariate case, the derivative becomes a Jacobian of which we need to take the determinant.\n", "As we usually use the log-likelihood as objective, we write the multivariate term with logarithms below:\n", "\n", "$$\n", "\\log p_x(\\mathbf{x}) = \\log p_z(f(\\mathbf{x})) + \\log{} \\left|\\det \\frac{df(\\mathbf{x})}{d\\mathbf{x}}\\right|\n", "$$\n", "\n", "Although we now know how a normalizing flow obtains its likelihood, it might not be clear what a normalizing flow does intuitively.\n", "For this, we should look from the inverse perspective of the flow starting with the prior probability density $p_z(z)$.\n", "If we apply an invertible function on it, we effectively \"transform\" its probability density.\n", "For instance, if $f^{-1}(z)=z+1$, we shift the density by one while still remaining a valid probability distribution,\n", "and being invertible.\n", "We can also apply more complex transformations, like scaling: $f^{-1}(z)=2z+1$, but there you might see a difference.\n", "When you scale, you also change the volume of the probability density, as for example on uniform distributions\n", "(figure credit - [Eric Jang](https://blog.evjang.com/2018/01/nf1.html)):\n", "\n", "
\n", "\n", "You can see that the height of $p(y)$ should be lower than $p(x)$ after scaling.\n", "This change in volume represents $\\left|\\frac{df(x)}{dx}\\right|$ in our equation above,\n", "and ensures that even after scaling, we still have a valid probability distribution.\n", "We can go on with making our function $f$ more complex.\n", "However, the more complex $f$ becomes, the harder it will be to find the inverse $f^{-1}$ of it,\n", "and to calculate the log-determinant of the Jacobian $\\log{} \\left|\\det \\frac{df(\\mathbf{x})}{d\\mathbf{x}}\\right|$.\n", "An easier trick to stack multiple invertible functions $f_{1,...,K}$ after each other, as all together,\n", "they still represent a single, invertible function.\n", "Using multiple, learnable invertible functions, a normalizing flow attempts to transform\n", "$p_z(z)$ slowly into a more complex distribution which should finally be $p_x(x)$.\n", "We visualize the idea below\n", "(figure credit - [Lilian Weng](https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html)):\n", "\n", "
\n", "\n", "Starting from $z_0$, which follows the prior Gaussian distribution, we sequentially apply the invertible\n", "functions $f_1,f_2,...,f_K$, until $z_K$ represents $x$.\n", "Note that in the figure above, the functions $f$ represent the inverted function from $f$ we had above\n", "(here: $f:Z\\to X$, above: $f:X\\to Z$).\n", "This is just a different notation and has no impact on the actual flow design because all $f$ need to be invertible anyways.\n", "When we estimate the log likelihood of a data point $x$ as in the equations above,\n", "we run the flows in the opposite direction than visualized above.\n", "Multiple flow layers have been proposed that use a neural network as learnable parameters,\n", "such as the planar and radial flow.\n", "However, we will focus here on flows that are commonly used in image\n", "modeling, and will discuss them in the rest of the notebook along with\n", "the details of how to train a normalizing flow."]}, {"cell_type": "markdown", "id": "a5b52653", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.012573, "end_time": "2022-05-12T12:19:08.868607", "exception": false, "start_time": "2022-05-12T12:19:08.856034", "status": "completed"}, "tags": []}, "source": ["## Normalizing Flows on images\n", "\n", "
\n", "\n", "To become familiar with normalizing flows, especially for the application of image modeling,\n", "it is best to discuss the different elements in a flow along with the implementation.\n", "As a general concept, we want to build a normalizing flow that maps an input image (here MNIST) to an equally sized latent space:\n", "\n", "
\n", "\n", "As a first step, we will implement a template of a normalizing flow in PyTorch Lightning.\n", "During training and validation, a normalizing flow performs density estimation in the forward direction.\n", "For this, we apply a series of flow transformations on the input $x$ and estimate the probability\n", "of the input by determining the probability of the transformed point $z$ given a prior,\n", "and the change of volume caused by the transformations.\n", "During inference, we can do both density estimation and sampling new points by inverting the flow transformations.\n", "Therefore, we define a function `_get_likelihood` which performs density estimation,\n", "and `sample` to generate new examples.\n", "The functions `training_step`, `validation_step` and `test_step` all make use of `_get_likelihood`.\n", "\n", "The standard metric used in generative models, and in particular normalizing flows, is bits per dimensions (bpd).\n", "Bpd is motivated from an information theory perspective and describes how many bits we would need to encode a particular example in our modeled distribution.\n", "The less bits we need, the more likely the example in our distribution.\n", "When we test for the bits per dimension of our test dataset, we can judge whether our model generalizes to new samples of the dataset and didn't memorize the training dataset.\n", "In order to calculate the bits per dimension score, we can rely on the negative log-likelihood and change the log base (as bits are binary while NLL is usually exponential):\n", "\n", "$$\\text{bpd} = \\text{nll} \\cdot \\log_2\\left(\\exp(1)\\right) \\cdot \\left(\\prod d_i\\right)^{-1}$$\n", "\n", "where $d_1,...,d_K$ are the dimensions of the input.\n", "For images, this would be the height, width and channel number.\n", "We divide the log likelihood by these extra dimensions to have a metric which we can compare for different image resolutions.\n", "In the original image space, MNIST examples have a bits per dimension\n", "score of 8 (we need 8 bits to encode each pixel as there are 256\n", "possible values)."]}, {"cell_type": "code", "execution_count": 6, "id": "9d6d5907", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:08.893950Z", "iopub.status.busy": "2022-05-12T12:19:08.893239Z", "iopub.status.idle": "2022-05-12T12:19:08.906436Z", "shell.execute_reply": "2022-05-12T12:19:08.905728Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.026966, "end_time": "2022-05-12T12:19:08.907991", "exception": false, "start_time": "2022-05-12T12:19:08.881025", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ImageFlow(pl.LightningModule):\n", " def __init__(self, flows, import_samples=8):\n", " \"\"\"\n", " Args:\n", " flows: A list of flows (each a nn.Module) that should be applied on the images.\n", " import_samples: Number of importance samples to use during testing (see explanation below). Can be changed at any time\n", " \"\"\"\n", " super().__init__()\n", " self.flows = nn.ModuleList(flows)\n", " self.import_samples = import_samples\n", " # Create prior distribution for final latent space\n", " self.prior = torch.distributions.normal.Normal(loc=0.0, scale=1.0)\n", " # Example input for visualizing the graph\n", " self.example_input_array = train_set[0][0].unsqueeze(dim=0)\n", "\n", " def forward(self, imgs):\n", " # The forward function is only used for visualizing the graph\n", " return self._get_likelihood(imgs)\n", "\n", " def encode(self, imgs):\n", " # Given a batch of images, return the latent representation z and ldj of the transformations\n", " z, ldj = imgs, torch.zeros(imgs.shape[0], device=self.device)\n", " for flow in self.flows:\n", " z, ldj = flow(z, ldj, reverse=False)\n", " return z, ldj\n", "\n", " def _get_likelihood(self, imgs, return_ll=False):\n", " \"\"\"Given a batch of images, return the likelihood of those.\n", "\n", " If return_ll is True, this function returns the log likelihood of the input. Otherwise, the ouptut metric is\n", " bits per dimension (scaled negative log likelihood)\n", " \"\"\"\n", " z, ldj = self.encode(imgs)\n", " log_pz = self.prior.log_prob(z).sum(dim=[1, 2, 3])\n", " log_px = ldj + log_pz\n", " nll = -log_px\n", " # Calculating bits per dimension\n", " bpd = nll * np.log2(np.exp(1)) / np.prod(imgs.shape[1:])\n", " return bpd.mean() if not return_ll else log_px\n", "\n", " @torch.no_grad()\n", " def sample(self, img_shape, z_init=None):\n", " \"\"\"Sample a batch of images from the flow.\"\"\"\n", " # Sample latent representation from prior\n", " if z_init is None:\n", " z = self.prior.sample(sample_shape=img_shape).to(device)\n", " else:\n", " z = z_init.to(device)\n", "\n", " # Transform z to x by inverting the flows\n", " ldj = torch.zeros(img_shape[0], device=device)\n", " for flow in reversed(self.flows):\n", " z, ldj = flow(z, ldj, reverse=True)\n", " return z\n", "\n", " def configure_optimizers(self):\n", " optimizer = optim.Adam(self.parameters(), lr=1e-3)\n", " # An scheduler is optional, but can help in flows to get the last bpd improvement\n", " scheduler = optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.99)\n", " return [optimizer], [scheduler]\n", "\n", " def training_step(self, batch, batch_idx):\n", " # Normalizing flows are trained by maximum likelihood => return bpd\n", " loss = self._get_likelihood(batch[0])\n", " self.log(\"train_bpd\", loss)\n", " return loss\n", "\n", " def validation_step(self, batch, batch_idx):\n", " loss = self._get_likelihood(batch[0])\n", " self.log(\"val_bpd\", loss)\n", "\n", " def test_step(self, batch, batch_idx):\n", " # Perform importance sampling during testing => estimate likelihood M times for each image\n", " samples = []\n", " for _ in range(self.import_samples):\n", " img_ll = self._get_likelihood(batch[0], return_ll=True)\n", " samples.append(img_ll)\n", " img_ll = torch.stack(samples, dim=-1)\n", "\n", " # To average the probabilities, we need to go from log-space to exp, and back to log.\n", " # Logsumexp provides us a stable implementation for this\n", " img_ll = torch.logsumexp(img_ll, dim=-1) - np.log(self.import_samples)\n", "\n", " # Calculate final bpd\n", " bpd = -img_ll * np.log2(np.exp(1)) / np.prod(batch[0].shape[1:])\n", " bpd = bpd.mean()\n", "\n", " self.log(\"test_bpd\", bpd)"]}, {"cell_type": "markdown", "id": "98940a88", "metadata": {"papermill": {"duration": 0.010911, "end_time": "2022-05-12T12:19:08.929952", "exception": false, "start_time": "2022-05-12T12:19:08.919041", "status": "completed"}, "tags": []}, "source": ["The `test_step` function differs from the training and validation step in that it makes use of importance sampling.\n", "We will discuss the motiviation and details behind this after\n", "understanding how flows model discrete images in continuous space."]}, {"cell_type": "markdown", "id": "32009fa8", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.011237, "end_time": "2022-05-12T12:19:08.952316", "exception": false, "start_time": "2022-05-12T12:19:08.941079", "status": "completed"}, "tags": []}, "source": ["### Dequantization\n", "\n", "Normalizing flows rely on the rule of change of variables, which is naturally defined in continuous space.\n", "Applying flows directly on discrete data leads to undesired density models where arbitrarly high likelihood are placed on a few, particular values.\n", "See the illustration below:\n", "\n", "
\n", "\n", "The black points represent the discrete points, and the green volume the density modeled by a normalizing flow in continuous space.\n", "The flow would continue to increase the likelihood for $x=0,1,2,3$ while having no volume on any other point.\n", "Remember that in continuous space, we have the constraint that the overall volume of the probability density must be 1 ($\\int p(x)dx=1$).\n", "Otherwise, we don't model a probability distribution anymore.\n", "However, the discrete points $x=0,1,2,3$ represent delta peaks with no width in continuous space.\n", "This is why the flow can place an infinite high likelihood on these few points while still representing a distribution in continuous space.\n", "Nonetheless, the learned density does not tell us anything about the distribution among the discrete points,\n", "as in discrete space, the likelihoods of those four points would have to sum to 1, not to infinity.\n", "\n", "To prevent such degenerated solutions, a common solution is to add a small amount of noise to each discrete value, which is also referred to as dequantization.\n", "Considering $x$ as an integer (as it is the case for images), the dequantized representation $v$ can be formulated as $v=x+u$ where $u\\in[0,1)^D$.\n", "Thus, the discrete value $1$ is modeled by a distribution over the interval $[1.0, 2.0)$, the value $2$ by an volume over $[2.0, 3.0)$, etc.\n", "Our objective of modeling $p(x)$ becomes:\n", "\n", "$$ p(x) = \\int p(x+u)du = \\int \\frac{q(u|x)}{q(u|x)}p(x+u)du = \\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right]$$\n", "\n", "with $q(u|x)$ being the noise distribution.\n", "For now, we assume it to be uniform, which can also be written as $p(x)=\\mathbb{E}_{u\\sim U(0,1)^D}\\left[p(x+u) \\right]$.\n", "\n", "In the following, we will implement Dequantization as a flow transformation itself.\n", "After adding noise to the discrete values, we additionally transform the volume into a Gaussian-like shape.\n", "This is done by scaling $x+u$ between $0$ and $1$, and applying the invert of the sigmoid function $\\sigma(z)^{-1} = \\log z - \\log 1-z$.\n", "If we would not do this, we would face two problems:\n", "\n", "1.\n", "The input is scaled between 0 and 256 while the prior distribution is a Gaussian with mean $0$ and standard deviation $1$.\n", "In the first iterations after initializing the parameters of the flow, we would have extremely low likelihoods for large values like $256$.\n", "This would cause the training to diverge instantaneously.\n", "2.\n", "As the output distribution is a Gaussian, it is beneficial for the flow to have a similarly shaped input distribution.\n", "This will reduce the modeling complexity that is required by the flow.\n", "\n", "Overall, we can implement dequantization as follows:"]}, {"cell_type": "code", "execution_count": 7, "id": "f9117a9f", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:08.976090Z", "iopub.status.busy": "2022-05-12T12:19:08.975369Z", "iopub.status.idle": "2022-05-12T12:19:08.984311Z", "shell.execute_reply": "2022-05-12T12:19:08.983622Z"}, "papermill": {"duration": 0.022493, "end_time": "2022-05-12T12:19:08.985814", "exception": false, "start_time": "2022-05-12T12:19:08.963321", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class Dequantization(nn.Module):\n", " def __init__(self, alpha=1e-5, quants=256):\n", " \"\"\"\n", " Args:\n", " alpha: small constant that is used to scale the original input.\n", " Prevents dealing with values very close to 0 and 1 when inverting the sigmoid\n", " quants: Number of possible discrete values (usually 256 for 8-bit image)\n", " \"\"\"\n", " super().__init__()\n", " self.alpha = alpha\n", " self.quants = quants\n", "\n", " def forward(self, z, ldj, reverse=False):\n", " if not reverse:\n", " z, ldj = self.dequant(z, ldj)\n", " z, ldj = self.sigmoid(z, ldj, reverse=True)\n", " else:\n", " z, ldj = self.sigmoid(z, ldj, reverse=False)\n", " z = z * self.quants\n", " ldj += np.log(self.quants) * np.prod(z.shape[1:])\n", " z = torch.floor(z).clamp(min=0, max=self.quants - 1).to(torch.int32)\n", " return z, ldj\n", "\n", " def sigmoid(self, z, ldj, reverse=False):\n", " # Applies an invertible sigmoid transformation\n", " if not reverse:\n", " ldj += (-z - 2 * F.softplus(-z)).sum(dim=[1, 2, 3])\n", " z = torch.sigmoid(z)\n", " else:\n", " z = z * (1 - self.alpha) + 0.5 * self.alpha # Scale to prevent boundaries 0 and 1\n", " ldj += np.log(1 - self.alpha) * np.prod(z.shape[1:])\n", " ldj += (-torch.log(z) - torch.log(1 - z)).sum(dim=[1, 2, 3])\n", " z = torch.log(z) - torch.log(1 - z)\n", " return z, ldj\n", "\n", " def dequant(self, z, ldj):\n", " # Transform discrete values to continuous volumes\n", " z = z.to(torch.float32)\n", " z = z + torch.rand_like(z).detach()\n", " z = z / self.quants\n", " ldj -= np.log(self.quants) * np.prod(z.shape[1:])\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "61405f9c", "metadata": {"papermill": {"duration": 0.01093, "end_time": "2022-05-12T12:19:09.009683", "exception": false, "start_time": "2022-05-12T12:19:08.998753", "status": "completed"}, "tags": []}, "source": ["A good check whether a flow is correctly implemented or not, is to verify that it is invertible.\n", "Hence, we will dequantize a randomly chosen training image, and then quantize it again.\n", "We would expect that we would get the exact same image out:"]}, {"cell_type": "code", "execution_count": 8, "id": "6e314bbd", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:09.032925Z", "iopub.status.busy": "2022-05-12T12:19:09.032430Z", "iopub.status.idle": "2022-05-12T12:19:09.042478Z", "shell.execute_reply": "2022-05-12T12:19:09.041770Z"}, "papermill": {"duration": 0.023233, "end_time": "2022-05-12T12:19:09.043890", "exception": false, "start_time": "2022-05-12T12:19:09.020657", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Dequantization was not invertible.\n", "Original value: 0\n", "Reconstructed value: 1\n"]}], "source": ["# Testing invertibility of dequantization layer\n", "pl.seed_everything(42)\n", "orig_img = train_set[0][0].unsqueeze(dim=0)\n", "ldj = torch.zeros(\n", " 1,\n", ")\n", "dequant_module = Dequantization()\n", "deq_img, ldj = dequant_module(orig_img, ldj, reverse=False)\n", "reconst_img, ldj = dequant_module(deq_img, ldj, reverse=True)\n", "\n", "d1, d2 = torch.where(orig_img.squeeze() != reconst_img.squeeze())\n", "if len(d1) != 0:\n", " print(\"Dequantization was not invertible.\")\n", " for i in range(d1.shape[0]):\n", " print(\"Original value:\", orig_img[0, 0, d1[i], d2[i]].item())\n", " print(\"Reconstructed value:\", reconst_img[0, 0, d1[i], d2[i]].item())\n", "else:\n", " print(\"Successfully inverted dequantization\")\n", "\n", "# Layer is not strictly invertible due to float precision constraints\n", "# assert (orig_img == reconst_img).all().item()"]}, {"cell_type": "markdown", "id": "e4bb61fc", "metadata": {"papermill": {"duration": 0.011138, "end_time": "2022-05-12T12:19:09.066797", "exception": false, "start_time": "2022-05-12T12:19:09.055659", "status": "completed"}, "tags": []}, "source": ["In contrast to our expectation, the test fails.\n", "However, this is no reason to doubt our implementation here as only one single value is not equal to the original.\n", "This is caused due to numerical inaccuracies in the sigmoid invert.\n", "While the input space to the inverted sigmoid is scaled between 0 and 1, the output space is between $-\\infty$ and $\\infty$.\n", "And as we use 32 bits to represent the numbers (in addition to applying logs over and over again),\n", "such inaccuries can occur and should not be worrisome.\n", "Nevertheless, it is good to be aware of them, and can be improved by using a double tensor (float64).\n", "\n", "Finally, we can take our dequantization and actually visualize the\n", "distribution it transforms the discrete values into:"]}, {"cell_type": "code", "execution_count": 9, "id": "51af4cde", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:09.091044Z", "iopub.status.busy": "2022-05-12T12:19:09.090281Z", "iopub.status.idle": "2022-05-12T12:19:09.776862Z", "shell.execute_reply": "2022-05-12T12:19:09.776116Z"}, "papermill": {"duration": 0.701406, "end_time": "2022-05-12T12:19:09.779430", "exception": false, "start_time": "2022-05-12T12:19:09.078024", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzkzLjczMTI1IDIyNi4xODg3NSBdCi9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUiAvVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJy9ncuuLcdxpuf7KdbQGmir8p45tNo2AY9aagI9MDyQaUpNQceGKdlC++n7/yOicmUE9xIPSbQE0Ob5WCdXVV7ikhGZ8Yu/+/q/vvnq619/8cvH//hfb794/umrP76lx+/xz+8e1+P3+OfPj/T4Av/87u3Cnz69lVXeR0m54U9/OP6Uc39Pc44Gerk//Z+3t9++/eJv0cQf8Xe+eHur+Dtj8i+V8d5qwVNod8z3FugfTpqv8j674mcLJ7UfyvJD76nhG/At+OfP+scvjL79B77vevz8wn8q9f1CG/oF76s9vvr09ssvH7/4h/RI1+PL38pXf/mvb//0+Juf49GfPf758eU/vv39l2+/OltJ+XovZeVrdnz597eU3terllrD3+25XLO18TktpVctzes9t9VTmnV9f0PXe3vRELu4ltpS6nPN723petlLObf3nMuoo6arfEZDL9+orveWR8qlXrN+b0Ov+yjP9p5qTbX1mr5/BrwetjKu9/4Zc+j1FMLceW9NJuJ6zzq5X7fz3y9beZ8jcd0VLJHa+W8/aqh2OyNhMX5vO6/nzt1OutZ7+b6G0me8UKoDY/a9DX3GG836vr6voRzfiG38nK2l+j4WBc+qWGdTBNbrdv7nzx4J780f+5tvf/bI6V3+9d//5Tf/8s0fvvnT/3U/8vhAQhZIqvlIvb+X/Pj268f/fvzb4xd/W0zW5dQ6ZA+eu95rt/+Nt+t9XG3OXFpej29/9/rJh3vy11+4J99eP/nt716I8yesE4u+JwgzyO36ntZIdZKWvPq1lPbU08ikkDRXmaTt/SotLWmhlVVLV1pbzWmR9trmZXSOkqu0MLCKutD+nhd+W1qYeIWSlI6U8pQW1rimvALGplANPCBaLry3NsDvWaU00rQAm9LVZ2mLtFztWoUUH4R3mNJCTRMyXCnesOIlKbJKuYY8ixWe8RXSQq+z16W011qbtDB6y5Nd1rBIe6lDWkC3r9WU1pnx0w/+9yuN2ZWuC5+ZSTF+fUgLCV19tdZJ0Ue9J6WjrDakhdLraOz0lqEhZ4d0ll8Yqy+lbYyepYUOsZuq0rU63x10lDGW/BqUBSbOkBbwZRgrpROrf0kLC4Nyya9x5ZSBwcTb4H11koD2kUctpLmlrF+B6bDSwLjwe/Dtsylt6RrSQEuN2kDgymsmaaDzKwixcOqcRf7+mBDYRkdn5wGutLRNrtOORx/4wquNnLPSfrV1DVLMylnl2QnzAVOEEPNkyGwErBV9KxQjpy/FKZjXkL/fMZmq9AtmAqYMELoXE0zRuNKV5W/j9zCioJ0Del0Vf73gpTDBilKsDPQlYOYsUIa3S1clLO3qMj07Om1AY8rfb9e15OdBMcTQfICYcLpAen7H0k9DGpgXpb9STEOsXtLV8CHsv86XKeggDBqGrzaDFb1WK2keGTNZ6VxYWIO0lmvJo/W94FtXIkTvzlaVjrw4y0BHy0lflrp6liINLPxrHUo52zCnOUP0pTD+sAnR7UB4jbak0f5e8TEcQMgQGDOy6EAnHiny1yFNkky2PvD3YPJIC6NdSYYQtPdc0csyw2AOybOTAgPPPDCZLiqUpBS93HIlzX2lZc/i3xrkHyiXehEKMVJmG9ICDMgpywt0tNGw3EHxYjpisDO4CJK0sGbmvwnFg72hBUxeTnm2ANV9QVNCHoNi8o+rKK2ldFhyoA1DKqMDOmFSZWkB/y9J5w5MhYGZIC2sa405lXZMTEwLTOp0YVZy3o6Cf19jZVKMKAS2Ugz/5BTDqsDwrqR0oR+LtNDnBY1CisnQqXJI5+DXKx0TCgctQDSPnopQjOxVZerTXMJod6UwnRcFIIQwfkAW6cC4QNg2aaHPLAsKsGIZUnRQro4qEgkUL3td+YFlhMFsMkkGug+fCWUGCjVR61QK0YWXIG2pXiKZ4bCkgi+SFgaUT+pKYc7zCS5PTMJSlS58EOQfFqIITfk1TIeF9vJDFm1tIplBMS+o2EAxoxMfpYlfSprSADpZ9BMg1jKlFtYsdOmS9TsxGbD+MVRc3zB35Q1A0YmYn6QYGzShFIONFUg6IHYGe2xCxOUJ3Uq6IGv7UAqDHs7PAwscUqmJdpn4995LL6QQVkm0ISjENoeqU95AH8hHwCi+KrrvISICP5CUVtj3BS1AtFC2Gp01w1wnhaWhkwxWPBYCBgWw50smNBgmVrvk709obhmdySW6GgQtpUmS0QVr0NeNqKDnUlZI4w5qDBS2SZOhmZgI6OQlf51mpMw6UExWfCGECVaT6quJeQBhgPejyYa+knkAiglMMwoUStd+DBO7yCB0XWVJIXyuAS0GuYM1qPYPKNYxtC0pPBeV+QvzYGIqEY5Ee0AhrBssAtI10S5/a1HTDkzQB4QRjJCrJaWU84WwUREthas3iuQuZkwX/bQwDSaECRqgqQTtkZXOi5MAkxcKNcsLQCllNFZJuRBEE4Fi9mH5k6KPR+PAwIi+YA3iyyHLYBwtkZegdVLAkKIBVUWgc2HWDlJ8VxJ7cWH04PPicyDLoP6um6KzLshDyr0LzrC8bedyQ4+QDsjpm0LkwA55QMJhuWdZIqALyw3aDLRcNCNJ0fsJU3WS0iS66SwwTqUFbgeIzF5TzBhIdcg9GE89G+0Dan6StlqHiKXFPkX/J1IYYipUQLHeMrQQpGGiYV2VrjwKBBdoxbSRAcbw09GCWCCGtFNblhhGZUFvDypSaCfF6Cw6L5P4npCkMALRX6RQPrr6oKlh7sCERBudixIr1DCdaCxAkaqQxsMwvqDir1KsQgQ3xeiEsSo+bNCTK7l1w1BKWInEUO7dfpKGBeZIJkYXNFEVxA2LlQ7q5OoYYnkQQ843KDdKXAxgVtwoSKFaiSHrWyqG4cLBi3gMWuS0dQRDY9MDWMQw9K7WDHcMHhQc8IIUsn4dMIEgIdtjinKwb8c39EWBTnGMBZ+r4QllA1lBeYxPr4rhPFwY1kZcr2vK3CWGBsEMJx4qQehHY/QwzR4Qvez0mzbofXQlaMtYS9nwQmPQcsBzQbdJwwnGPawuPDNlMFS6Ec9MP4cYy32IAk4JZjjEM4QlRTWGZi7Dnes0E0M6jaJPY44MrM9ODEFQdXgT9QvNmAeldb7U/iaGVX9BTFJgY4XrqCeYrqVD5DwosWGMWNvwKlujywYMN0MNVshxTgEsQmJ84mxGG41ItAFzLqvsgIZDe7DdFilWyZWWYRiBCRIKIhqvM1czjBmHx4krfAF9WEw6jOjkbCppJaPwGjJ0HoQ0DKCuQ063H+YAhgkYGlhNCmKoIbw/xDQ0gg4XIGyOAqUHCk26rOVJDw0qkHjR5GiGId0xVSCo4WZVMUfxDbCMMF6FmAbXujHkToXQhKjO2m6GjVMvTP9FNQ+VPozSZIbqg6jG2KgFk2TjZELMEEPbXiId0ZVoDTb6QyX4pcMKiunK2Q2MAWv6fRkzFhII6m9xoS6bG8BNfIdFyzyrw0a6aNhXYjhkQ9dbxtTAFMUyW9wLgbpLhmeieQQ67hbgfkBiw/yHYIYIVkVBDOMfdi0xXhOiSTBmBn4FIhuimZumKgqBqe16Je54T5UdwOuCpkMji1q/6NTIfCe4X4kYIrTbS+OlsFTRZZTD8BdK01ehuU7bXzhGvBbDUElYrQ9KYk5HFW950tnHBBAOw7qryAevGRbbfIiIhrju9/OzwN5JwiE6sw3P4pbVGlU41qpNEvAOx26xnUJ7kDav48fe/GtqbTu63yRQe+9A7Ssd3X0SqHWgo7u7A7WxcXQPZKA26oHaFHF0z6dAbfI5ek/TAG1KO7rnf6C6VgK0deXoXoSB6oJ1cC/uQE0SBKpS44RP8RKoySJHb7kVoMk4R7dADFRlZ4AmZx3dQjlQFeAOblkfqCmGQE2LOLo1TqCmnRzdqixQ03uObiUZqGnUQE39Orp1daCm2B3dVkCgZjKc9GlfBGrGSKBmuDi6bZxAzSBydFtPgZqpFaiZZY5uGy5QM/gc3dZhoGZKOrrtzkDNSA3ULFpHt/kbqNnKjm7DOlCzwgM1k93Rbd8Har6Ao9txCNS8jIM+PRIPzXnx0PycE26XyEPznk64HS0PzSc74XbfPDRPz0NzCk+4/UcPzdU84fZKPTQH1kPzdU+43WIPzYM+obnaHplPfsLtvnuojr5ntiVwwr174KHuMxzsuSPhoW1eeKjbHCfbGyIe2t7JCe9dFs9sP+aEe+vGQ93k8cw2g06494081A2mk+2tKA9t18pD2+A64d4L89C2zU64d9g8tM24E+59Ow9ti89D2w084d449NC2GA/43Iz00PYtT7i3OD203VAPbeP0hHuP1UPbjj3h3rn10DZ5PbTt4BPunWMPbZP5hHs/2kPbuj7h3uX20DbEPbS98xPubXYPbUf+hHvz3kPb5/fQQgIn3NEDDy3QcMBnTMJDC1+ccEc6PLSgiIcWPznhDrV4aFGZE+4AjocW6znhDgt5aOEjDy3SdMIdlPLQ4lcn3JEuDy0o5qGGz062A20eWkzuhHf0zjOL851whwQ91NihZxZlPOAzIOkhA5cn2QFODy0U6qEGTU+2w6seWiT2hHfI1jML7p5wh4E91ICxZxZaPuGOQntoAesT7ti2hxYGP+GOmHtowXUPLQ5/wh2y99Ci+yfciQAeWs6Ah5ZecMBnJoKHlrRwwp3f4KGlQpxwZ014aAkWHlouxgl31oaHluBxwp0L4qGljXhoGSYn3MkoHlreygm/m/ciOYi/fPvV44em8CRm4yyadVjmzNuB9MIIro71+e3vwn9+uP/86y/+0n9+Y4YOuhMzHJb1IaY+nZR+xkKv6S5TkVgLlS59FfhxS8LJ5Ks0prCIv4OVqXunBVMQwhYLgN5RnWveeDA3VCi3fnRTrjB9InOji34XHO4u0VRy2Am0KcR3g78sYZJUICNll5YYuiPplhRwrWiHzUyGOZPGIgrDJpPDnySjDpJUfxZLd8LTlEQ0WBllVsPj6rCkhDfuwmgzUAFY6pBH9HwZKZTwG3mrtO+IOzpc95qBF/5mI4c13VuXgCcsW+aqQCIL76usVY3Dc68PuutQCkX3MAv3qWmbCIex23Tbu3CnenE9coMATni3X8WHt1kxmuQTGidV43WMutgODZOq8S3yuWjiCYdZVTQGVKDBE3eVHrLdgcetHUh+6Nu+hGNYNVkqVebRYrKwnS45BJJZQN7QxZxQCUM4L/3Zyk3rBIkimHkuOs8qLBTMgc7OYRBnZR0T8JlXX0k43Mmlm6UVfQUnR7qN6gdDm42jEXog5LC3lrVTGJ+E9an5uavpFm1lAkwbeIYYuunSPVpweFrUF9wjg3eYrXl0YS2wtR/cZ+MW05rGIbbpX5F3JlhoL2DqzATX6cGNvQQN1KtxxvOuJHx0jRPDGsNkX+iTB7cMYdR360zYMPiUOoVPGGj15jC2II0esknJzU9tHpOH8yyjvVJyNga9RNVNzO1obWIyWYdpG7J/OhjLN47B5JwHZkxNIxjMk0MvLbbCtD18g3Ykpg1TRYZwvGG2X8W0WYnDlJmFcRWdBsw+S1fCK4jk4eZBM96ZL5KFM/VZN7Ubd74nJxMkCaySpv0CDAM2cwmKhIEtfvMJA4Rb6RAlpYnoIWfUlilIxIvbMcXw4MLDW0JkQN4VFXJQ3JgJJbMZ+G1Q1xJMJ2dadhOM2WbBTCZDYcpAtFEG5DZtDjPHCYbUlYVPTL5lGEKTuzEUATVbCgKzvimRKHKx2OHFTN2QB8cLw3AULmtd374zFaZS5GJRQ2hqKgV5hWdDaQk+6tRsJPKVqAEfstht3TMtSDMrSGGy1mwYQ8+sKS7duuqlkSVm+4xEl4BL9OIW1jIOy63XJByyz2J7zNdhB7IdLF1onZSM10y7TvjEwtdQEhNxILcS2+GO+JiSbgRVBO+7M2VG1ihWukaTwCEIBycU1iJMII3MdAn/M4lClmJhrpnxhk6+mvDBPbhpHCMyOUNoUMCMXNI7sLnhF0wGT7jm8H3aDcyJgWgbmooM61Yj0kx/WYuygQsG+mtpUJWpLnCfOY+5vvjC+jxmF+YNW1kQp6PWbpgSl0qW64i7f8M4fpJxGllIi3Er4Zg5i+G1JPmW7VJhs/Fhm7yAu+FA7TUC1Xd2cH9goNYdju6+C9R62tE9LoHaKAZqY+7oniCB2nQ66XPyBWpTNVCb2I7uZRCoLRpH9xIL1Najo3vtBmorPVCTC45uKRKoyRxHt4QK1MSZo1v4BaqSMkATq45uIRyoSmwHt3gP1JRBoKo5HNxqJlBTSifdGixAU3eObuUYqGrSAE3tOrp1dKB5hL++FX+gZiYEakaFo9sCCdTsFUe3dROo2UKObsspULOzAjWjzNFtwgVqBp+j2zwM1IxJR7fpGajZqYGaVXvSpw0cqFnMjm77OlCzxgM1293RbekHan6Bo+ZDBGb+hqPbOwlUXZkAze9xdHtJgapL5eD2vwI1by1Qde0c3H5goOY1Onq7mAGaO+rodl4DNVc30Ogu/2jPntuo2dzxN4qgUdu1mHJy/hc7nPP88+snf/3F22c+Ka5/p6lTXez400lFhGIuZ7UmJK3vokvGExGYzNVwk9wRyuYGY2EVwxBHiR4T5XtOehondegxvAKtTR4sgD5p2TjTNWH/UKEwL0AdBmai5skYA/lYXQ9XkFuOLJVVaUkTrxMTT7sa49zkuyz1mrzOxqCiqMdW9agLOR6iq04Ve2HOaf7TgFcI3XFN4ei2rN45+KiMejyoviHbs+SzJiaWMv+8CafHJ0nj5G1ejJvSNEijWPPswSU+APdQ09V0b2FkpkSLE0BbBP62Jr2BT+Z7J+FjMJdAOHx+eH/M1uH27lWb2svgfeCtp/DJxHRtv0rqOi1OWlYN35iNN7w9/XBwJq6prwK+mIXJdmh3w9XSdqTPG/1w8AU/7rr5gDigg0czEYtSbU7mhEL+Keb5CMkXJu4Q+cLhsGZmPQjHUOTBQxTk3BTQWcX0T8n+5Pyq8ML7jSedtyocs6iowTwmtwIG/T6mRF/dEtPAB8ckCZ883aO9wHStzMXC+VV4FnUZ12M1xJNHM27MfTTu7DDQActR7XEme47FkCs556x6eOCDG2CcRFAx9vITHiw3OIpw2Dzzxr10WvgytxjIkHefPHjUluJOi7AYrlQkeHVmKGOV6PqR3M4qe2QjS3hJ19vEaEKYcZUzH7nfWwuTh7Yztz04h7AMh2bAzQoNyO4gnrBfND0PuI0LpsuDU4VninVe8vwk+rUs4R2zw5rHKF+S8s+pAkGsp97IZx5pduHwVGybacLrqtDGSTjdUN1rA+/QplzmmEMQ80uX/+TeEubRFD6umTVDEbxBpEz+Lh6BaaobOOAr50LHT1KL89BmMPoVimU+dG6ltflAR3G3jXOLAWZthktbTnyS44mh6YST+4aTuRTkcAtyDvzULS+pte3ofpNA7b0d3R8ZqHVJoNaBju7uDtQGx9E9lIHawDu6p0mgNqkC1Rno4J6ugdrkdnQvhUBt4QSqq8zBvSQDtfXrqK31wEwunPQpRQJVkROgySdHtzQL1GSfo1tSBqpiNUCTwY5uiR2oincHty4I1DSHo1vPBGpaKVDTYY5ujReo6UdHtzYN1HSvo1tTB2pqPVAzAhzdJkOgZmC4DYhtjgRqxkugZuo4ug2jQM2McnQbXYGaieboNugCVesvQLMUHf2utfkTLOcpAS2orUKTt0H+w/vuTa7xkKPpHF2xnJ9PPl4/KZbzZz0plvOCF4B5eGYrfDopVyosQz2QTU47iNv1g3nRd6QHfMJRHHohBwRTUmNvQaKMKju4FBl0BbPxvors4DIfZ6WpWfkLDlCS8yIURmxHdTZ4K4w0C+fhebU4lpzRE+ONIq01S8FfEJtdsgbIsZyrKmFe5DLRK/xdHhCkvyccGv8aDGeRt0733zjsOwaiRObCJFUtuaDxK5N7HpTQcF3VJmBmf2cWk2D8pp40JJ9YprTeeOZEhRk5JsRFRU28JLVN6aCXoZc3QCXYJujiQaJLtnCnblymZpxJmbSoJ13Ya2lMEZy7y6PrJRBl2uNTdp0ZN+PFINw7rMZnknxhck6gqs0smgdivjFt7JpmIDP1v1UJ0VBR4v+IQZOZ+s+JwdfvELwpy/ogr0tjnFS3+CkxzMlhDV30o6ZktPSq7WCgSxL7jXlt8MGvmw+e7RY8OhPjBWPqdA3IMDcuM4fWeJuThyLUIoDi1J+FW34Nsd94cqZCzWTjlQeOm3CG+GX/X67gqeoe0t7IthGPF+dBUzof5B121DDcp5z6Ih7zShKjyBe6/MpZ34aKJt8YakACNTyac7U8lvFFD3IKR28Mex4zp3EFCefh1TyND577b8qnpKmSD8b2iww6c9B7Gdk4zyPRZOfZHyp1/SpGm3leTzhW1WW9yQNCG7fZbQzliFBd+vowhYd1Gs8IZeb7CofvaXOBnmKSSA35LHoWMzMKDeuu6+tjfan1SY6p1qa+PvPfxVclp8Gf9PX5fRJ2OPghoV9TaztQe5OTPt87UPtKR3efBKodGKD1tqN7bAK1kXR0j3ugNksc3XMqUJuBgep0dXDP7UBtITi6l02gtsgCtSXp6F7Agdpyd/SWDQGaIHF0i51ATUgFaiLtNAOeAjBQE5eObuEaqIliR7fcDtSkfKCmExzdGiRQ1TYObtUUqCmyQE3rObp1ZKCmUR3d+jdQ09aObt0eqFkCgZrd4Oi2MgI1m8TRbcEEavZOoNFm+gn2X7OUpr54BdF1W268gkistsr7VsT+ez75eP2k2H+f9STtP7i81HHuVNank4pc3UIxyQn5HqUt03saVUGQzpkZduOaUZoXpuON8pT+qlzAuW/R+tYWsldGvnhnydraxX6XmSQXzRGvjXhpVVk8nu+VV2rMN5ymNJ/KDrxhyExpPpUj04GWXAdnylSVWpJTuSvnoHvBJ+V9D7o68VhmWSOodmYJoUUmnDhTIHEzlvfcbMthLeMtwR5O3tAAhsBiQNzbJYk3ka1rzmDHJF67NFN62j2yr5nzxYu7UunBTgLvNEGDWZVhnlPllW2GZcO11szsGme08SqwDi8xBSMvZ2bY5taCUQg+rqSm8WlEMtcoyxacNzrBGxPbuzdSyaHaS13eqM1MQmIkrnojmHysroP1tJnlTj+q/WBik8NXqLXdJrlmtPBiyKsxXXab8LMbhwdUmQV0WPzEczHz2fsHOQ8mXjP36nQniHmQuRfnfUBLM6G7yYgfzgp5YywybeemDuNr8iDGdoZkGjOdiYbP8r4T+cxyqsZ8rapDVbi7X3SVH74ZOU/Xl7R9OdnHxRxgKvyQrxLfb0imGTlU95AhP3xFcm5rX8G3zMx/Kjz16H1R8sGA7PK+68FP1f6SWtuO7jcJ1N47UPtKR3efBGo9eNJnfwdqo+PoHspAbeADtWni6J5TgeoEdHDP1kBtagdqC8HRvWwCtUXm6F6Rgdr6dXSv9kBNNgRqksTRLXcCNSnl6JZpgZoEDNTEpaO3bA3QBLHzR7bYDtSEvKNbJQRq+iNQ0zaObt0UqCoyB7fWC9R0pKNboQZq6jdQU9aObtUeqBkCjm6zIVAzMgI1k8TRbcAEauaOo7dtFGC0rn6KpWj57pidvIlXY+D5jYlyZtXZRuHHDz7cg3L95fPBt5cPiplYeElGn2dy1aeT3kEhlcyFR60md/3OGFJmqmmqSyJ4GnOqxXgtTAjxMSry2TJv9rpDWqYpGo+HrRVCYORQJ1dqPmSWS4d3cF0SwjtCbOSNeeZ1h+RM43SqZjGdNIQ3lrYzeO1TshDeDvmRjylXtt0hwqTt0DpsSUJ4GlJU0wy8Z94X5kOQuSzegpElhCcRyz5vzrPYEsOTCKdeC0k+OX2qj4jmyit6LsmTPiOo5Oi2IjE8ibh21Yw18VBikd3eI0BL3tqQz3qGc0kXp3TZ0d+qrWRmn4vpdMaKs2S/Vm71nKFlKDweuWQ+qAtFk/eWxXSyyLVaeLXyMCcPVLlINznmLK9vcZFx8nWtFgLpuTZmn4vtdAbeyZkLzrCtxOn7JWm8eBFJPE47rF+a4cZWuk8DwAziKVgxnixroDXjFfOKUVvNMhhqrIDPmumNuayEzPtkeT5n7SwGFShVbxJk2NayHhQvHvId3Oq1JInNoXQGt/3OpAryVca05AwmYegM4S2plGxzJ23o6zR5SBb6meSReaMplgr3eo+cEGJMdVnomkIy1GTjVaNwQyRl5kg5Ia9w03vzKSrkE26jpMxoSosuuCbnQ2ShWwpMvTke551od8qM2te8tzNDOqedYtNu3ijem0/JIWdKvKTMHCk8mX8ZGqmElB/yefGmpp0ilPX9O68CT5Iyc6QUkcPby5Iz80xAyvzGDiu57XylMjw/dcsLupsO1F7E0f3agdpHOrq7JFDrwECtux3dgxOoDaWje+ADtWkSqE0qR/cUDNTmq6N7dgdqa+Gkz5UTqK2zQG1VOrqXcKC24B3d4iFQEyaBmuhxdAuqQFWqObhFYKAmMB29pWuAJooDNcHt6BbzgZpScPTWIAGaunF0K6dAVZEFaFrP0a0jAzWN6vy2rX8DNW0dqOl2R7clEKjZDY5uKyNQs0kc3RZMoGbvBGrWkaPblgrULC9Ht50WqFl1gZoN6Oi2GAONRudPibPfpm79zlXvQkuCtNU4+37yO9fHP5/UOPvnPCkGdKMBe+VTmnw6qTvYQj7Ry+N5DkZtKWjNIntudmpG7erGzdgh9pudsjG9wT9Q+/pDOeRtULe4MzzEa9UiJ6L0yI/EnWEK8caZyt2p84gQOZdPvU8UVTkiBCnGq2zEgLMTSHL6i7yPS/a+zhNLmff3QiLTgLMDTmqd9yx36tFqtwNR2TCjiIzb2vkp3XKj+QTpRwPOzlupTdB5CpE33uzzWdppvHB31k5dbue5jMp9dsmf/sow/q6cO7egz9Ni5BU9w/3I+3DZ/fzEDODm1PMoGmxfXnQ0aji4Rt6ZTVXug262Fcc7cenA9n0wTndZwZsak+U4R0e8WhPr7Tx3l9WOnnICzo7p3ZxTiNbbeawvw06Hk7sYtbVjgJv3Ild036cG1egdPIt9ifl2njIk5+3H83kqUc1DXmjLlZP9IUZIFkhZyVm1M4/rxoM6sNxnJKtah4M3g3YOxH2kUk15+kKdp/DcCUziNTGRyz6xqWtNXS0eKXcHPMlnzuyE8zxopidXGTj050fJOw+45H3cVFfPkAvb5eDr83AqaeU2Rt1HWdXdED8UVsDY517t+c4r06RxPSY7jGKGFi5BO1VrHSl3KRemyxyHcInbKrTR7zO76kINkQK8bPs+42s9wyCJum52Jnganjzg0/cRYhvuxfvmeG+qO3JM3rmRU/YJZZ318+JddtzAv080q3UM3rCA6EOdJ6DJF+8JL/7EdOauc6+yv32esCYfs3Q5aK4nsnXvn5vdF36g7APcKhHAO8tJdH/gO8/CqwCHPx5OWtsazJw4j5OTzzEH/drz+HlmHABCbNyn1W848IuMMOnZ9q47DYxJlCbHgs+z8OSy79782XlyBte4U3KetceEYMxjyhn242R+lhtkE/OH3En+zIgQj3nNffJfd1wYcKrwU/K+KEA9Fga0uqQSnfcKENc5r1T8PQTk62KGp7+2AC/IjQE6wectB8Q888G4ot2KoCKdl8lCC3FIzksUyDE70vPOBZ1/K1FGJm4FnHc0ZL1QlmvgvtFBt394o2y9eIH+fQOEjiCvlKWjXPeNEe3mvAycatZumFAv9clPo+YltbYd3W8SqL13oPaVjt5dEqD130mfvR2oDo2DexwDtVEPVKeIg3s+BWqzz9E9VwO1me3oXgeB2qoJ1NaYo3tFBqqL18G90gM1uRDoVWOHb3kTqEknR7csC9Qkn6NbTgZqUjVQk8EnfUrsQE2+O7q1QaCmOgI1RePoVkuBmg5zdGu8QFU7OrhVaaCmeANVJe3g1uiBmv539DYWAjTLwtFthwSqRkuAZuE4uu2hQNV4cnBbWoGaXRaoWXEnfdp8gZqF6Oi2JwM169PRbasG2r6zsfG0gh3dNnOgZmE7uq3xQM12D9QsfUe3XxCo+RCObo8jUPNPHN3eTKDm+gRqjpKj260K1JwwR2+PLUBz79zu13YGA1XPMUBzMx3dTmmgJUZBn85uoNFh/gnO/3Fx1O/e3B+/iP8V+rHwZGw+Z+ynk97Xhps1Ii4Pd6zdNePkq7RCt9muJdfN7lV5crfUtK8xVyscHLO50G+2a8/VxlqNh4ILAx92Tbpa84ulBHg/pbtVPS+6E3hq+UvYySvDtMXd2U7MtClMS7vhXaMzvEp+Dl6jZvfB5xtD8HDz/7w9PvMPufF6NrtrPg3DjYbxfTF9S0ZXLzyk97zEPuPHC8zf3O/77tWYBoZBAZF0Xo5fLhZPuXjt3HmVPjEsVlrQ5737hZ0mm7vnJf2kFQodfWZX+ucbz9Wp6e3+f7EsCwaqcD/OagUUYwOuEhS9FRawRwsP2LNs1y5BQNYGk9ysWIFYw6QLInSVu7TB0HYZN+JVfXcdhJkNsy5edUUTysXaX3Je6aiwQNppwwxXjQGOCOYDK7BY6QbJbSpiK0vU7Cz0QAyFeGFJWFGIqZhFtnj+zZWQIB7QE1A/VnCi6LdwljB566xOQcrwGqbDUcmi0FNgIkK9y15Uo7VIYTGrkSGRG+LZqHHvihraBpwTyBFKOKu/IZVAiDEtoInuah2SbV7o+vBwTHK1PYhhYfAOTKsEsqZh+LQYcVc3pMDfKrD2IBasykidhgePA1ZXk6TQm2MOxrQKJk1cXmJKEMiEo9wJ3kB+vrejMgph7dCn866ikpfhOcWAOGuuYNLylhF6iWeFFmL8G2tuHeVcCrxeBixZnkhrv0yj0E/9anehGPGiiXnpV553WZmuX8IY7MXDpFaERtcV8EyLV7KeJWugCng7S9dKNlLgxmhvGMZdDSfpS0tlNN4Ka7VzbMwX74hhLQirtKPSIHE7rvK2WavLo7JDahhLWp9V8ZH4LPGAyIXUtJo/stkIv5L32vDGW6sQJL4/Mcv4zXaWEyrcI7kWt7at9pCuL+AKRwWm8FmpiBhCYuEPZ12jkgvv7WHky4ogiStMzIPN+KGzYhK0K8z3BmHuyisRw1vG5LprMdkLshRs4X11VrhJgsGYK/AiMCu7q/JEPJhwtFxJKDiD3ILjH876UcSdqXndFZvCYIg/D/fPKlOpcAGmSMG3WRkr0TrEvPgeHtVZ86pkltprPCJiBbLaMjxSZY6wVNPqWuup5MUBlApuR+kt4saJZ2W6hkpK0MVcq3LX9JKPKdymuqSE21H9i3hKTUmrFKbignuScBCLLytGzNOa7VmCTL6lZB51ZDm9o1wZKSRY0zJoUtvM6GLI/lkGTXHhecvWd8k0g4OHnYsrr0abCKPfr2cptmW4yYq1sm06yUCXBqOtxpuKoUIlO1lm1erBqcySrWSFLB1X9dnOU6qUMneduWEYa3/UcVSkK9y7XpUtHLXrSFuS9Airc6daEXhhxNO4a+IV/b3JcD4rkJ0F9IjHuLiNeVbbg4/Jw73MYL5L8xXDcLIWZOdZx6/Ui8eGeW2bFf1TjcT9f/xNDqxVCEyGIX95372VE+z6NGYHbA4oRqs9aE9zkVQW8bNChWJfQSyi3wvv3D+rGhI3phTWuwKiNcJkIJgKc1dL1J/E/Jgs3OFKKxKPxTu27jqMOmIV/iw/ebqijcSddwCnu8KjvSC8I2iD1KwcZDIo2+BzV468McXeSHeZSes/pmBh4bW7JqUkRREPJlPOXb9SpgPvycV8qL7YJXHjDn5zlTGJITrLnK6MZmHFZZap8zU3iWkMlGYFOrsqet4BXHmdqKvmSYxJSmvsLv0pPcK7hWdlwbazTihxu6R69F1UtBpeTCTNrgJp4f3GMHlG2+VKi+HBC6OXq21aeG8yNEfOrhAqMeYfS3lY1VRVvSyly3uf1y6x2gzD5GGRkLse6zQMQURbw4q32uewxm1l+RGt9Kpp/cRYQ4xKWVlYWdiNazxLweGjhCwx3CFep3vWmyXmKmIl5KM4beFt151Jza6SLTFvTJzLlb3Fuudxo5XKs0AuGewWCgorpSv+R2HmEf4adIjV3VUlw7HGb7Ag8lGkl5iRzaufBX1LY01s5g6ftX9JodPoHpyFggvvG+dua3dFhYlbZ0FdrT+sgotrEJ/BusharFgibIULGaKldlfYuIg0YEmmuwiyQEY5O2v5HPWSSWHq5PQsrSy9zLvZ5STTXYZ5Gq1MmC53zeZilDn881ncWXqI8hgLOu060MUohpg1jaxotNpwFP88k+ULTBM3Xkm5i1Er7TwPyOMHR+HqIml7nSHus8Y18WQm2FkOu/AyfR7fqrt09jDMW7qv4cpsl84FLPWhz5rcRdS6+CN3Ae9qGOKysda1VfvWRhbPXMrAHqXBiWHMsVreXUdcuokFCBbzbF3RceLOMHC6K5SrmcDCBszarLuceTdcOYGmq31OzFhETbtQurZNtzTz1I9VVV9GGaqH5DwrsBfWbIDFCNV4lmsvYm0u2o9nbXfixY3jeReCl9nEshFzsKbZXTS+GabIvdquMK+4SWXUPHc5+mG489BecrXrC8tcwOnH+N117rPhOgucI2Kx35bhBR8TAww8GmskCYZwhf3bpBEKfhVuLMuBlT+kkQZXrmi3YpY0Fo0jrsVKsBI33ZHizfB5mXf9xMc21mtqLTu6XyNQe2dH9wcGar0RqHWdo7ufA7VBcXSPYKA23I7uqRGozaNAbdI5umdooDabHd1TP1BbJ4HaonJ0r8BAbbme9Lm2AzVB4OiWGoGaiAnU5JGjW3gFapLO0VsoBmgCNFATto7egjlAE+KObokfqGoHB7cmCdS0TqBXPGt7aLNATfM5eivJAE2hOrq1b6DtO8O4dfpJnwZAoCUeNDrsikDNCAnULBZHt3kTqBlCjm6rKVAzsRzd9ligZrwFapaeo9sqDNRMSEe3vRmoGaeBmiXr6DZ7AzUb2dFtUAdq1rej21QP1Oz6QM0JOOnTYwjU3AtHty8SqDkujm4vJ1BziQI1/8nR7WwFap6Zo9uNC9R8vkDNPXR0+5KBmuPp6PZSAzWX1tHt/wZqznKg5lk7ut3wQM1nd3Q7+IHabkCgtnVw0uc+Q6C2KeHo3sEI1LY7HN17I4HaRkqguuni4N6eCdT2chy1bZ/AbIfI0b2dFKhuPQVo21SO7j2tQHX7y8G9VRaobasFesX6RMd2XaC2t+fovQ8YoO0ZnvS5vxio7kUGaPuWju5NzkBtR9TRvX0aqO21Bmobs47uXdxAbcvX0b0/HKhtJju6d54DtW3qQG1P29G9AR6o7ZY7urfWA7V9+EBt097RvcEfqEUDHN2hg0AtznDSZ1AiUItgBGrhDkd3bCRQi6M4uoMugVqExtEdzgnUYj+BWpzI0R1TCtQCUI7uaFWgFtoK1KJgju6IWaAWXnN0x+ICtcCdozvKF6iFBAO1+KGjO9gYqEUmT/oMYwZqMc9ALT7q6I6lBmpxV0d3kDZQi+g6usO/gVqsOFCNKzu4Y9CBWrza0Tu2HaDFwR3dQfNANcAeoEXiHd1h+0BLOHBz5AIEaokDgV7xBNmRkBCoZS8c9JnpEKBlRTi6EygC1WSLAC0xw9GdxRGopXw4uvNDArVkkkAt9cTRnacSqGW1OLpzYAK1jBlHd35NoJaNE2jM6PnhyUTp8Y8PyIHHnx+ynaYXabbvHCbCPMDsoofw6y8ev/i7r//rm6++/vUXv3x89cePiul9OiDPAOaeJMPh7X/9dV+NZyIw19dyB5QOeic9ydbsj327v1QB8Luv9IMK+v11Xum7xYU+vapO9BNG8HNqKXz4bp9fBOGv/nYfXAn+6eW14j/+7T7rNt3vvt0Pugb3r/12H11c+enl5Zc//u0+6ya6D97uB1wh91d/uQ/ud/r08o6on/B2n3GRygcv9wNuVflrv9sH1wt8enlFwU9YEZ9zKvfDt/v847R/9bf74IzPp5fnhH78251ZzG/xFX5IDvP/l1f4KMj16VWc7Ee/wSN2/of21KejBPQZ33xWzz6ixrviuIXgJez3LMt+Ji88C9XfWR7UXozNXRctpTsfhnRmntCUvDTLG+LK4aWK+EzMBMujkrPYC18nV7+dKWUwrxOPFQyXYscs78FzgO1OO9TyRPC0yuBFBHciplZXgRe5ZuGHnnmpPJPIQjy1n4m6cvzy4vUnd+Ky1uvMhSeyej8SuOXoLG/ITTvbXQvOvKxU/KIa74v6sa8qkL4qwvSyytDHZXdeVa55VfvlRVGVl9fdv7pe/tVN7x9fuf7y6vMXV5C/ugv8xaXcry59fXUr66tbU1/cX/rqItFXN3q+ulrzxS2XL6+VfHXp2ItLvl5envXq/qkXNzW9vLvoxWUNrw7jvziE/erk8KszsB8eFnl5ZOLjUwUvcu5fJKR/nK39KpH54xTfVxmw48Mc0VdpkuPDXMFX6XIfp4e9SIn6ONnnVQ7L96WIfKARM1Td9fg9bRGoux+0nSB3r/dtd3+vPg7Pf9j6x5kr7ulnM3+p9Yvfhen/+B20+e/xz5/tj2//8QaX/PHzC0zSkIv8CKWW/vWvPr398svHL/6BGuLx5W/R0vX48l/f/unxN3/39X/852/+7U/f/Pdv/vTNv//b41+/+eOfvv3mX/5T/vDbf//2MYm++vZnaAza4f7f42++/tPXj//6zR/+8+s//uzxz29f/uPb33/59qu3X/xt5dmxhDe8B+AL/ulNQlAQc0MOM2Di26em+1M3/YPkp/CW+jafVLJe7ElKgKZn7zaclz341dtBk+zE2R7iiSdTUi4Jku3fkvvm5eHjtZ7wq+MTnlTjXXPK+xwtMAi2H71/64D9aPWm+xv+cMDja58/9OyW7/bqV7pH9ywyube2vnPdznNrS4tMfvzkW9wE++w23yTKaZ9MG4lGJfMfHPrDiaj1Nfv9O8g+K9tn3atA/vWLDxYCxY/7oWMRXH4RXJjDj+ccLjqHnaXu9ri+1en9lx5BP6W3v9SC65whcpUTznXQE7tOGuvA5UP8Ezrr+aOvOyx90GGfXXvURMPn7on9gJqmR2fQt4B6Ch26qetPuY2yx+486E/ozf2Lrzszf9iZn1mO6pazn7lJ9gPKXB1dwSvmtjZrH2DXnUyo3bh8iH9Chz5/9HWPlg979DMLPFiPfu7e2Q8oHHF0Bq8Zn3F23tD1Josxzzg3n/An9OT9c6/7sX7cj59xU/KzGz9vI+1zW3WdyMuqYbLHbnxi15G8X2nj8iH+CZ35/NHX3dk+Xuifdx/evdA/c3ftB9yz9+wM8aFnWOYbut6Ep7Rp+Yj++L7cP/i6J/uHPXleSWDd9RcvLXi+OGuO9h6//EnPT+c+06blI/rjP/35i6+/fZzf/qu3/wfWcznvCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTMwNjUKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjQgPj4Kc3RyZWFtCnicPZDBEUMhCETvVrElgIBAPclkcvi//2tAk1xkHWD3qTuBkFGHM8Nn4smD07E0cG8VjGsIryP0CE0Ck8DEwZp4DAsBp2GRYy7fVZZVp5Wumo2e171jQdVplzUNbdqB8q2PP8I13qPwGuweQgexKHRuZVoLmVg8a5w7zKPM535O23c9GK2m1Kw3ctnXPTrL1FBeWvuEzmi0/SfXL7sxXh+FFDkICmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNzAgPj4Kc3RyZWFtCnicPZBLEsMgDEP3nEJHAP+A87TT6YLcf1vLmXSDFGPLL0RXdOyVh8fGlI33aGNPhC1c5XQaTlMZj4u7Zl2gy2Ey02+8mrnAVGGR1eyi+hi8ofOsZoevVTMxhDeZEhpgKndyD/X1pzjt25KQbFdh0J0apLMwzJH8PRBTc9BziJH8I19ya2HQmeYXFy2rGa1lTNHsYapsLQzqjUF3yvXUeq7zMBHv8wPfQT5kCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NCA+PgpzdHJlYW0KeJxFkU1yBSEIhPeeoi/wquRXPc+kUllM7r8NzbwkK1qF5gPTAhNH8BJD7ImVEx8yfC/oMny3MjvwOtmZcE+4blzDZcMzYVvgOyrLO15Dd7ZSP52hqu8aOd4uUjV0ZWSfeqGaC8yQiK4RWXQrl3VA05TuUuEabFuCFPVKrCedoDToEcrwd5RrfHUTT6+x5FTNIVrNrRMairBseEHUySQRtQ2LJ5ZzIVH5qhurOi5gkyXi9IDcoJVmfHpSSREwg3ysyWjMAjbQk7tnF8aaSx5Fjlc0mLA7STXwgPfitr73NnGP8xf4hXff/ysOfdcCPn8AS/5dBgplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMyID4+CnN0cmVhbQp4nDVRSW7EMAy7+xX8wADW7rwnxaCH9v/XUsoUCEAltrglYmMjAi8x+DmI3PiSNaMmfmdyV/wsT4VHwq3gSRSBl+FedoLLG8ZlPw4zH7yXVs6kxpMMyEU2PTwRMtglEDowuwZ12Gbaib4h4bMjUs1GltPXEvTSKgTKU7bf6YISbav6c/usC2372hNOdnvqSeUTiOeWrMBl4xWTxVgGPVG5SzF9kOpsoSehvCifg2w+aohElyhn4InBwSjQDuy57WfiVSFoXd2nbWOoRkrH078NTU2SCPlECWe2NO4W/n/Pvb7X+w9OIVQRCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzEgPj4Kc3RyZWFtCnicNU85kgQhDMt5hT4wVRjbQL+np7Y22Pl/upKZTpDwIcnTEx2ZeJkjI7Bmx9taZCBm4FNMxb/2tA8TqvfgHiKUiwthhpFw1qzjbp6OF/92lc9YB+82+IpZXhDYwkzWVxZnLtsFY2mcxDnJboxdE7GNda2nU1hHMKEMhHS2w5Qgc1Sk9MmOMuboOJEnnovv9tssdjl+DusLNo0hFef4KnqCNoOi7HnvAhpyQf9d3fgeRbvoJSAbCRbWUWLunOWEX712dB61KBJzQppBLhMhzekqphCaUKyzo6BSUXCpPqforJ9/5V9cLQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM5NSA+PgpzdHJlYW0KeJw9UktuxUAI2+cUXKDS8JvPeVJV3bz7b2tDUqkqvIkxxjB9ypC55UtdEnGFybderls8pnwuW1qZeYi7i40lPrbcl+4htl10LrE4HUfyCzKdKkSozarRofhCloUHkE7woQvCfTn+4y+AwdewDbjhPTJBsCTmKULGblEZmhJBEWHnkRWopFCfWcLfUe7r9zIFam+MpQtjHPQJtAVCbUjEAupAAETslFStkI5nJBO/Fd1nYhxg59GyAa4ZVESWe+zHiKnOqIy8RMQ+T036KJZMLVbGblMZX/yUjNR8dAUqqTTylPLQVbPQC1iJeRL2OfxI+OfWbCGGOm7W8onlHzPFMhLOYEs5YKGX40fg21l1Ea4dubjOdIEfldZwTLTrfsj1T/5021rNdbxyCKJA5U1B8LsOrkaxxMQyPp2NKXqiLLAamrxGM8FhEBHW98PIAxr9crwQNKdrIrRYIpu1YkSNimxzPb0E1kzvxTnWwxPCbO+d1qGyMzMqIYLauoZq60B2s77zcLafPzPoom0KZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzNiA+PgpzdHJlYW0KeJxNj0EOAzEIA+95hZ9AIEB4z1ZVD9v/X0vYdtMLHsmAbFEGgSWHeIcb4dHbD99FNhVn45xfUiliIZhPcJ8wUxyNKXfyY4+AcZRqLKdoeF5Lzk3DFy13Ey2lrZeTGW+47pf3R5VtkQ1Fzy0LQtdskvkygQd8GJhHdeNppcfd9myv9vwAzmw0SQplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nE1RSYoDMAy75xX6QCFek7ynQ5lD5//Xyg6FOQQJr5KTlphYCw8xhB8sPfiRIXM3/Rt+otm7WXqSydn/mOciU1H4UqguYkJdiBvPoRHwPaFrElmxvfE5LKOZc74HH4W4BDOhAWN9STK5qOaVIRNODHUcDlqkwrhrYsPiWtE8jdxu+0ZmZSaEDY9kQtwYgIgg6wKyGCyUNjYTMlnOA+0NyQ1aYNepG1GLgiuU1gl0olbEqszgs+bWdjdDLfLgqH3x+mhWl2CF0Uv1WHhfhT6YqZl27pJCeuFNOyLMHgqkMjstK7V7xOpugfo/y1Lw/cn3+B2vD838XJwKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDk0ID4+CnN0cmVhbQp4nEWNwRHAIAgE/1RBCQoK2k8mk4f2/40QMnxg5w7uhAULtnlGHwWVJl4VWAdKY9xQj0C94XItydwFD3Anf9rQVJyW03dpkUlVKdykEnn/DmcmkKh50WOd9wtj+yM8CmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA1NCA+PgpzdHJlYW0KeJwzNjNUMFAwsVQwMjZRMDY0AmIThRRDLqAIiJXLBRPLAbNAqnK4oMpzYKpyuDK40gAFGA4yCmVuZHN0cmVhbQplbmRvYmoKMjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MiA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlxAvqmJuUIuF0gMxMoBswyAtCWcgohngJggbRDFIBZEsZmJGUQdnAGRy+BKAwAl2xbJCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0NyA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlyWEFYuF0wsB8wC0ZZwCiKewZUGALlnDScKZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIyID4+CnN0cmVhbQp4nDVRu23FMAzsNQUXMCB+Jc3jIEiRt3+bO9qpSNO8H1VeMqVcLnXJKllh8qVDdYqmfJ5mpvwO9ZDjmB7ZIbpT1pZ7GBaWiXlKHbGaLPdwCza+AJoScwvx9wjwK4BRwESgbvH3D7pZEkAaFPwU6JqrllhiAg2Lha3ZFeJW3SlYuKv4diS5BwlyMVnoUw5Fiim3wHwZLNmRWpzrclkK/259AhphhTjss4tE4HnAA0wk/mSAbM8+W+zq6kU2doY46dCAi4CbzSQBQVM4qz64Yftqu+bnmSgnODnWr6Ixvg1O5ktS3le5x8+gQd74Mzxnd45QDppQCPTdAiCH3cBGhD61z8AuA7ZJu3djSvmcZCm+BDYK9qhTHcrwYuzMVm/Y/MfoymZRbJCV9dHpDsrcoBNiHm9koVuytvs3D7N9/wFfGXtkCmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTggPj4Kc3RyZWFtCnicPVC5jQQxDMtdhRpYwHrtqWcWi0um//RI+fYi0RZFUio1mZIpL3WUJVlT3jp8lsQOeYblbmQ2JSpFL5OwJffQCvF9ieYU993VlrNDNJdoOX4LMyqqGx3TSzaacCoTuqDcwzP6DW10A1aHHrFbINCkYNe2IHLHDxgMwZkTiyIMSk0G/65yj59eixs+w/FDFJGSDuY1/1j98nMNr1OPJ5Fub77iXpypDgMRHJKavCNdWLEuEhFpNUFNz8BaLYC7t17+G7QjugxA9onEcZpSjqG/a3Clzy/lJ1PYCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MyA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDUxID4+CnN0cmVhbQp4nDM2tFAwUDA0MAeSRoZAlpGJQoohF0gAxMzlggnmgFkGQBqiOAeuJocrgysNAOG0DZgKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0MyA+PgpzdHJlYW0KeJxNUbutAzEM6z2FFjjA+tm+eS54eMVl/zaknASpREMUScnDU7pkymF9SkZIji4PbRpLbLo8N0JTh4qCqWuJ6pSrmabMUyxN0PPeWa7mGOB7VTfU3/SIXgKRUYJVYYEOkDu4YPjZayZsUQsiMYZQM4BpwgpzuBIxBBmMtWcYlCoMTtXPKlf7L6dl2CqweDCdIj+ymminX7oceOspB0LY3JW7eiFNCO6NBmPMLFx3qbKdABxMdJmJjFi8DcfTIQwNXpoGrHDWjZggsRsjpQ9eBxnTsHdFHnW3GPG+W8aUu9XPfVF95l3tHwjBGyf4ewHKG11eCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjAgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzNCA+PgpzdHJlYW0KeJwtUktyxSAM23MKXaAz+AfkPOl0uni9/7aSk0VGDmD0MeWGiUp8WSC3o9bEt43MQIXhr6vMhc9I28g6iMuQi7iSLYV7RCzkMcQ8xILvq/EeHvmszMmzB8Yv2XcPK/bUhGUh48UZ2mEVx2EV5FiwdSGqe3hTpMOpJNjji/8+xXMtBC18RtCAX+Sfr47g+ZIWafeYbdOuerBMO6qksBxsT3NeJl9aZ7k6Hs8Hyfau2BFSuwIUhbkzznPhKNNWRrQWdjZIalxsb479WErQhW5cRoojkJ+pIjygpMnMJgrij5wecioDYeqarnRyG1Vxp57MNZuLtzNJZuu+SLGZwnldOLP+DFNmtXknz3Ki1KkI77FnS9DQOa6evZZZaHSbE7ykhM/GTk9Ovlcz6yE5FQmpYlpXwWkUmWIJ2xJfU1FTmnoZ/vvy7vE7fv4BLHN8cwplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzAgPj4Kc3RyZWFtCnicMzM2UzBQsDACEqamhgrmRpYKKYZcQD6IlcsFE8sBs8wszIEsIwuQlhwuQwtjMG1ibKRgZmIGZFkgMSC6MrjSAJiaEwMKZW5kc3RyZWFtCmVuZG9iago0MCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMyMCA+PgpzdHJlYW0KeJw1UktuBTEI288puECl8E/O86qqi777b2sTvRVMMGDjKS9Z0ku+1CXbpcPkWx/3JbFC3o/tmsxSxfcWsxTPLa9HzxG3LQoEURM9WJkvFSLUz/ToOqhwSp+BVwi3FBu8g0kAg2r4Bx6lMyBQ50DGu2IyUgOCJNhzaXEIiXImiX+kvJ7fJ62kofQ9WZnL35NLpdAdTU7oAcXKxUmgXUn5oJmYSkSSl+t9sUL0hsCSPD5HMcmA7DaJbaIFJucepSXMxBQ6sMcCvGaa1VXoYMIehymMVwuzqB5s8lsTlaQdreMZ2TDeyzBTYqHhsAXU5mJlgu7l4zWvwojtUZNdw3Duls13CNFo/hsWyuBjFZKAR6exEg1pOMCIwJ5eOMVe8xM5DsCIY52aLAxjaCaneo6JwNCes6VhxsceWvXzD1TpfIcKZW5kc3RyZWFtCmVuZG9iago0MSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE4ID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iago0MiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzMyA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjQzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzQwID4+CnN0cmVhbQp4nDVSOW4EMQzr/Qp9IIBu2+/ZIEiR/L8NqdkUA3F0UpQ7WlR2y4eFVLXsdPm0ldoSN+R3ZYXECcmrEu1ShkiovFYh1e+ZMq+3NWcEyFKlwuSk5HHJgj/DpacLx/m2sa/lyB2PHlgVI6FEwDLFxOgals7usGZbfpZpwI94hJwr1i3HWAVSG9047Yr3oXktsgaIvZmWigodVokWfkHxoEeNffYYVFgg0e0cSXCMiVCRgHaB2kgMOXssdlEf9DMoMRPo2htF3EGBJZKYOcW6dPTf+NCxoP7YjDe/OirpW1pZY9I+G+2Uxiwy6XpY9HTz1seDCzTvovzn1QwSNGWNksYHrdo5hqKZUVZ4t0OTDc0xxyHzDp7DGQlK+jwUv48lEx2UyN8ODaF/Xx6jjJw23gLmoj9tFQcO4rPDXrmBFUoXa5L3AalM6IHp/6/xtb7X1x8d7YDGCmVuZHN0cmVhbQplbmRvYmoKNDQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTEgPj4Kc3RyZWFtCnicLVFJcgNBCLvPK/SEZqffY5crh+T/1wjKBwYNi0B0WuKgjJ8gLFe85ZGraMPfMzGC3wWHfivXbVjkQFQgSWNQNaF28Xr0HthxmAnMk9awDGasD/yMKdzoxeExGWe312XUEOxdrz2ZQcmsXMQlExdM1WEjZw4/mTIutHM9NyDnRliXYZBuVhozEo40hUghhaqbpM4EQRKMrkaNNnIU+6Uvj3SGVY2oMexzLW1fz004a9DsWKzy5JQeXXEuJxcvrBz09TYDF1FprPJASMD9bg/1c7KT33hL584W0+N7zcnywlRgxZvXbkA21eLfvIjj+4yv5+f5/ANfYFuICmVuZHN0cmVhbQplbmRvYmoKNDUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNzQgPj4Kc3RyZWFtCnicTZBJDkMhDEP3nMIXqIQzwOc8v6q6aO+/rUMHdYH85CBwPDzQcSQudGTojI4rmxzjwLMgY+LROP/JuD7EMUHdoi1Yl3bH2cwSc8IyMQK2RsnZPKLAD8dcCBJklx++wCAiXY/5VvNZk/TPtzvdj7q0Zl89osCJ7AjFsAFXgP26x4FLwvle0+SXKiVjE4fygeoiUjY7oRC1VOxyqoqz3ZsrcBX0/NFD7u0FtSM83wplbmRzdHJlYW0KZW5kb2JqCjQ2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjQ3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTQxID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iago0OCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc2ID4+CnN0cmVhbQp4nD2MOw6AMAxD95zCR2h+JAdCiIHef6UptIv99CTbxdFgWpECt8DJ5D6p03LPJDt8EJsh5FcbWrWuytKaDIuajL8N391N1wumOBfACmVuZHN0cmVhbQplbmRvYmoKNDkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTUgPj4Kc3RyZWFtCnicNVE5DgMhDOz3Ff5AJIwveE+iKM3+v82M0VYewVyGtJQhmfJSk6gh5VM+epkunLrc18xqNOeWtC1zgLi2vC+tksCJZoiDwWmYuAGaPAFD19GoUUMXHtDUpVMosNwEPoq3bg/dY7WBl7Yh54kgYigZLEHNqUUTFm3PJ6Q1v16LG96X7d3IU6XGlhiBBgFWOBzX6NfwlT1PJtF0FTLUqzXLGAkTRSI8+Y6m1RPrWjTSMhLUxhGsagO8O/0wTgAAE3HLAmSfSpSz5MRvsfSzBlf6/gGfR1SWCmVuZHN0cmVhbQplbmRvYmoKMTUgMCBvYmoKPDwgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucyAvQ2hhclByb2NzIDE2IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ1IC9oeXBoZW4gL3BlcmlvZCA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXgKL3NldmVuIC9laWdodCAvbmluZSA2OCAvRCA4MCAvUCA5NyAvYSAvYiAvYyAvZCAvZSAvZiAxMDUgL2kgMTA4IC9sIDExMCAvbgovbyAxMTMgL3EgL3IgL3MgL3QgL3UgL3YgMTIxIC95IC96IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNCAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvQk1RUURWK0RlamFWdVNhbnMKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTMgMCBSID4+CmVuZG9iagoxNCAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zCi9JdGFsaWNBbmdsZSAwIC9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxMyAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNiAwIG9iago8PCAvRCAxNyAwIFIgL1AgMTggMCBSIC9hIDE5IDAgUiAvYiAyMCAwIFIgL2MgMjEgMCBSIC9kIDIyIDAgUiAvZSAyMyAwIFIKL2VpZ2h0IDI0IDAgUiAvZiAyNSAwIFIgL2ZpdmUgMjYgMCBSIC9mb3VyIDI3IDAgUiAvaHlwaGVuIDI4IDAgUiAvaSAyOSAwIFIKL2wgMzAgMCBSIC9uIDMxIDAgUiAvbmluZSAzMiAwIFIgL28gMzMgMCBSIC9vbmUgMzQgMCBSIC9wZXJpb2QgMzUgMCBSCi9xIDM2IDAgUiAvciAzNyAwIFIgL3MgMzggMCBSIC9zZXZlbiAzOSAwIFIgL3NpeCA0MCAwIFIgL3NwYWNlIDQxIDAgUgovdCA0MiAwIFIgL3RocmVlIDQzIDAgUiAvdHdvIDQ0IDAgUiAvdSA0NSAwIFIgL3YgNDYgMCBSIC95IDQ3IDAgUiAveiA0OCAwIFIKL3plcm8gNDkgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNSAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EzIDw8IC9DQSAwLjUgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMC41ID4+Ci9BNCA8PCAvQ0EgMC44IC9UeXBlIC9FeHRHU3RhdGUgL2NhIDAuOCA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCA+PgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKNTAgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkwOSswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCA1MQowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAyNDM3OCAwMDAwMCBuIAowMDAwMDI0MDk4IDAwMDAwIG4gCjAwMDAwMjQxMzAgMDAwMDAgbiAKMDAwMDAyNDMxNSAwMDAwMCBuIAowMDAwMDI0MzM2IDAwMDAwIG4gCjAwMDAwMjQzNTcgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQyIDAwMDAwIG4gCjAwMDAwMTM1MDQgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDEzNDgyIDAwMDAwIG4gCjAwMDAwMjI2NDkgMDAwMDAgbiAKMDAwMDAyMjQ0MiAwMDAwMCBuIAowMDAwMDIxOTU0IDAwMDAwIG4gCjAwMDAwMjM3MDIgMDAwMDAgbiAKMDAwMDAxMzUyNCAwMDAwMCBuIAowMDAwMDEzNzYxIDAwMDAwIG4gCjAwMDAwMTQwMDQgMDAwMDAgbiAKMDAwMDAxNDM4NCAwMDAwMCBuIAowMDAwMDE0NzAxIDAwMDAwIG4gCjAwMDAwMTUwMDYgMDAwMDAgbiAKMDAwMDAxNTMxMCAwMDAwMCBuIAowMDAwMDE1NjMyIDAwMDAwIG4gCjAwMDAwMTYxMDAgMDAwMDAgbiAKMDAwMDAxNjMwOSAwMDAwMCBuIAowMDAwMDE2NjMxIDAwMDAwIG4gCjAwMDAwMTY3OTcgMDAwMDAgbiAKMDAwMDAxNjkyMyAwMDAwMCBuIAowMDAwMDE3MDY3IDAwMDAwIG4gCjAwMDAwMTcxODYgMDAwMDAgbiAKMDAwMDAxNzQyMiAwMDAwMCBuIAowMDAwMDE3ODE3IDAwMDAwIG4gCjAwMDAwMTgxMDggMDAwMDAgbiAKMDAwMDAxODI2MyAwMDAwMCBuIAowMDAwMDE4Mzg2IDAwMDAwIG4gCjAwMDAwMTg3MDIgMDAwMDAgbiAKMDAwMDAxODkzNSAwMDAwMCBuIAowMDAwMDE5MzQyIDAwMDAwIG4gCjAwMDAwMTk0ODQgMDAwMDAgbiAKMDAwMDAxOTg3NyAwMDAwMCBuIAowMDAwMDE5OTY3IDAwMDAwIG4gCjAwMDAwMjAxNzMgMDAwMDAgbiAKMDAwMDAyMDU4NiAwMDAwMCBuIAowMDAwMDIwOTEwIDAwMDAwIG4gCjAwMDAwMjExNTcgMDAwMDAgbiAKMDAwMDAyMTMwNCAwMDAwMCBuIAowMDAwMDIxNTE4IDAwMDAwIG4gCjAwMDAwMjE2NjYgMDAwMDAgbiAKMDAwMDAyNDQzOCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDUwIDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSA1MSA+PgpzdGFydHhyZWYKMjQ1OTUKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:09.492340\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["\n", "\n", "def visualize_dequantization(quants, prior=None):\n", " \"\"\"Function for visualizing the dequantization values of discrete values in continuous space.\"\"\"\n", " # Prior over discrete values. If not given, a uniform is assumed\n", " if prior is None:\n", " prior = np.ones(quants, dtype=np.float32) / quants\n", " prior = prior / prior.sum() * quants # In the following, we assume 1 for each value means uniform distribution\n", "\n", " inp = torch.arange(-4, 4, 0.01).view(-1, 1, 1, 1) # Possible continuous values we want to consider\n", " ldj = torch.zeros(inp.shape[0])\n", " dequant_module = Dequantization(quants=quants)\n", " # Invert dequantization on continuous values to find corresponding discrete value\n", " out, ldj = dequant_module.forward(inp, ldj, reverse=True)\n", " inp, out, prob = inp.squeeze().numpy(), out.squeeze().numpy(), ldj.exp().numpy()\n", " prob = prob * prior[out] # Probability scaled by categorical prior\n", "\n", " # Plot volumes and continuous distribution\n", " sns.set_style(\"white\")\n", " _ = plt.figure(figsize=(6, 3))\n", " x_ticks = []\n", " for v in np.unique(out):\n", " indices = np.where(out == v)\n", " color = to_rgb(\"C%i\" % v)\n", " plt.fill_between(inp[indices], prob[indices], np.zeros(indices[0].shape[0]), color=color + (0.5,), label=str(v))\n", " plt.plot([inp[indices[0][0]]] * 2, [0, prob[indices[0][0]]], color=color)\n", " plt.plot([inp[indices[0][-1]]] * 2, [0, prob[indices[0][-1]]], color=color)\n", " x_ticks.append(inp[indices[0][0]])\n", " x_ticks.append(inp.max())\n", " plt.xticks(x_ticks, [\"%.1f\" % x for x in x_ticks])\n", " plt.plot(inp, prob, color=(0.0, 0.0, 0.0))\n", " # Set final plot properties\n", " plt.ylim(0, prob.max() * 1.1)\n", " plt.xlim(inp.min(), inp.max())\n", " plt.xlabel(\"z\")\n", " plt.ylabel(\"Probability\")\n", " plt.title(\"Dequantization distribution for %i discrete values\" % quants)\n", " plt.legend()\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "visualize_dequantization(quants=8)"]}, {"cell_type": "markdown", "id": "c276d1a9", "metadata": {"papermill": {"duration": 0.013705, "end_time": "2022-05-12T12:19:09.808110", "exception": false, "start_time": "2022-05-12T12:19:09.794405", "status": "completed"}, "tags": []}, "source": ["The visualized distribution show the sub-volumes that are assigned to the different discrete values.\n", "The value $0$ has its volume between $[-\\infty, -1.9)$, the value $1$ is represented by the interval $[-1.9, -1.1)$, etc.\n", "The volume for each discrete value has the same probability mass.\n", "That's why the volumes close to the center (e.g. 3 and 4) have a smaller area on the z-axis as others\n", "($z$ is being used to denote the output of the whole dequantization flow).\n", "\n", "Effectively, the consecutive normalizing flow models discrete images by the following objective:\n", "\n", "$$\\log p(x) = \\log \\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right] \\geq \\mathbb{E}_{u}\\left[\\log \\frac{p(x+u)}{q(u|x)} \\right]$$\n", "\n", "Although normalizing flows are exact in likelihood, we have a lower bound.\n", "Specifically, this is an example of the Jensen inequality because we need to move the log into the expectation so we can use Monte-carlo estimates.\n", "In general, this bound is considerably smaller than the ELBO in variational autoencoders.\n", "Actually, we can reduce the bound ourselves by estimating the expectation not by one, but by $M$ samples.\n", "In other words, we can apply importance sampling which leads to the following inequality:\n", "\n", "$$\\log p(x) = \\log \\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right] \\geq \\mathbb{E}_{u}\\left[\\log \\frac{1}{M} \\sum_{m=1}^{M} \\frac{p(x+u_m)}{q(u_m|x)} \\right] \\geq \\mathbb{E}_{u}\\left[\\log \\frac{p(x+u)}{q(u|x)} \\right]$$\n", "\n", "The importance sampling $\\frac{1}{M} \\sum_{m=1}^{M} \\frac{p(x+u_m)}{q(u_m|x)}$ becomes\n", "$\\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right]$ if $M\\to \\infty$,\n", "so that the more samples we use, the tighter the bound is.\n", "During testing, we can make use of this property and have it implemented in `test_step` in `ImageFlow`.\n", "In theory, we could also use this tighter bound during training.\n", "However, related work has shown that this does not necessarily lead to\n", "an improvement given the additional computational cost, and it is more\n", "efficient to stick with a single estimate [5]."]}, {"cell_type": "markdown", "id": "c38cf9bb", "metadata": {"papermill": {"duration": 0.013767, "end_time": "2022-05-12T12:19:09.837449", "exception": false, "start_time": "2022-05-12T12:19:09.823682", "status": "completed"}, "tags": []}, "source": ["### Variational Dequantization\n", "\n", "Dequantization uses a uniform distribution for the noise $u$ which effectively leads to images being represented as hypercubes\n", "(cube in high dimensions) with sharp borders.\n", "However, modeling such sharp borders is not easy for a flow as it uses smooth transformations to convert it into a Gaussian distribution.\n", "\n", "Another way of looking at it is if we change the prior distribution in the previous visualization.\n", "Imagine we have independent Gaussian noise on pixels which is commonly the case for any real-world taken picture.\n", "Therefore, the flow would have to model a distribution as above, but with the individual volumes scaled as follows:"]}, {"cell_type": "code", "execution_count": 10, "id": "20e5f49a", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:09.866816Z", "iopub.status.busy": "2022-05-12T12:19:09.866355Z", "iopub.status.idle": "2022-05-12T12:19:10.297455Z", "shell.execute_reply": "2022-05-12T12:19:10.296698Z"}, "papermill": {"duration": 0.448967, "end_time": "2022-05-12T12:19:10.300018", "exception": false, "start_time": "2022-05-12T12:19:09.851051", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzg0LjE4NDM3NSAyMjYuMTg4NzUgXQovUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicvZ3LrmXHcabn5yn2UBroMO+XodSyCXjUkgn0wPBAoiiaAsuGqRvaT9//HxErV0bW2cUjEk0Dgqs+roq9Vl7ilpmRn/36q7998+VXv/38V4//9a8vn91/+/LPL/HxJ/zv60d4/An/+/sjPj7H/75+Cfjbh5c8ymscJfeKv367/zWlhj8O/OlbPLv/7T9eXv748tkvIeTP+Eefv7zk8ppyTPWR+2uVfwzBbb4GD7/dYAr5dTSj1z/fof1Ikh95jRVfgC/B//6uf/3c6Mt/4+vC4xcB/ymV19Kzvf3rrI8vP7z86ovHZ/8cHzE8vvijfPMXf3j5t8fPflFew88f//744l9e/umLl9/sUmIMr32mnHPDV3+/pPg6n0kq7bWEmWsdtfb3SIrPJHW8U0w9t1Hm9wsKr/WZoIkmHr3l1sYc3yspPG2lFOtr6230VGLI7xD07I1Snq9zhFRGCaN8r6DnbZR6fa2zpx5aid8/Ap53W27xNfQhI/J7hHxiFE0M9dAgY74mFfZczP88+6TwOnrkrMv5tZfGP32ijb9XSuU71ZRb+PT7PG3iJQl6IrWM5g6fFJS+V1AM87WmOOsIubdPiMrfLwqKpENMnKGG+Kle+35RDSM/jzbYhZ96q6ej+hY16mtCO2Ga9Pqpt2peFKX8gvIivmpSp020eVXl/FzK//45lMWr/PLPvvs5pumr/PG/fv+733/z7Td/+b/uRx4fa+4MNPD97TWnx3dfPf7P4z8fn/0ymxrGrG+jJVqL19Ls//oLlFOoY6Rc03x89/XzJx/uyd9+7p58ef7kd18/sTKLzdc2auxT7UmrAcqSFEJmHKT1NbSJnhCa8HRSWsbApxKWPu3JGWBHisA2cyhiu9Ai0Jv672cYSS0a/lnNk5CTvIgAdEArJYqAAUE9Kq2D1klobZg8YnExJVNpAvvACyosKUIjks4AO9KUQkfWIAJmwhOZFPqlzpqjUPxUG0pbh9VRCT3kzB8rGNWz164SMKikZUArOhxThv899hyLUliulovQEmOSZ+NrrqXBmpE2/LuotNM2qYQxi0pIr3FCHYuEGPOYCluMHcII86hKM6abzA9SKJbYlJYye1cBYzRpx0InYbCrQNFKocqPYSSg/ZJIgE2xZzF3AmymSEitSp+XikFXIUPgxFwvSmspY4qAjJGUutLZ8oQ9IS14QFoBY2GkmUUCWhFClI4QZ1MJc1ib99eUwhwioaTapIdBWwkhiAQMoNjkWQyGOgOGI+kIszalBXqtioSK5p1Z6UDHdpFQM1qU47FMvHuLQSRgjI0clXa0aVIJs6CBQSs9KHSVSGiwCjIgQSuGQBcJreaQotI58R0ioY3UAr+ixlcODHxREf+kNIUD/6qKgI4vlklZ02sqk/qAtKcpHwwKRyRNEQDVTWULitEwek4iAVOm1qm0BoxHkTAwBGWgg85U8e5CYf5lNNTCzspTJGBOhaQQjk/B0CJEI+krYDSMhGeFzpznVNo47yAAzgA+N0qLNfw5lNGEYuimpBT/vKJFSGfC8FaKwV2zSEDTtyCfhtHQe60iIdaZp8I2G9uJcOYi/35gzuBfZcIEF6IqrLmwlQjhXchgAoUCaEX+PTRcFcUAY49R0boIyDkNbQQqr4ifE4qpAdY4j0KXNy2h5DAUYqj2Iv+8YJbJfKBrVEbv8s9Lx+TKSkvjSIzQJJgtRRlmLPS30BqDfEBLrxkNV+TfQ73XbhR+5OiE+Lgs06nl11igNERAa5gkCjmDojwK52NKb4HOEWcRWKKq8YZoIATMXqGYTDIVQPHk1N8aiDgmu6VV6gxqQMAmDQXU6JTIP58hQHOTYgTgq6v88wmPVToAtMwWpAHwIk2mLSD0RqT6Sxg2Sadi6+iLAldeaCsIDpR2KIFKiHGZsrwBRkBPsYsAWgOZBqCwaHGKgDghX8TCs8OsT4QIXab264TCgaEQARh2qiJAR8EMFgEZqomwUz126jFCDKU+lMI00x6AMpLpfK+OQRoqRykpNIgMYtCKadRFQoGy6F3pLHgfkQD92gebpvM3EDgMoRMTsSqFowy7SQpLO4v8Wn5NIZQmEtrIVTQwaEOHD5HQc2gy53qh11ajSOj0BYvSgnGWRQLM9sj2LOIghGZCewns3k5fDUwEYHab2ArlVlqQ7pk9quXuiIgZI0JAplqY+hENgx2WJgrtYtOE4t/xM0EjnpVh0zEW8PFTJETIjVEpBmtPIiElaE354IE/4zVFAnybod02YEY6NSFohqrVr8BggAtF7QfaGcwqLdBIUSRgkvPThU7M6CwS8P9bZKMPGLAE/1ckwLgEGU9D9PkYIgHdNjs7c2A4oC+CSMA0xYhTWjnnREKHpROTA4oZO6tIgBpMohQHBkyE7p6kA5rionitEETCaKqqBj6+StBRqPhz1kczY9FQRACDSWlIxCgY3KLXOC7wc1VpjXDGm1DoTRkNoFD9kZasQC/D8RG5Fd4O/FORkGDo9CMwHDqUvEhIPWbxKgaGAzTIFAl4rxmnUpjKlEQCOnvqr3UMmEFzWTg24cMYxQRJTSTAWmfRAqD4+EQ1WGiZu2jXgeEw6DaQwpzXahTuVs4iAeZ8KJ2vFNZEApqpiS0DrYV9TAo1PexRGMYSRADTPaKzJkbDoFzSCd9sGu1zFpoyuNZw2uSDJyxQHPQpADGbZQIBMq1DWwZrW9qUITKhAOGiJBEAZ1aHCGiRriLsU20/ILzK2kUADGAVDTsxutH6UwTAkYH/oBS+A4cuffuo3tJkr8IpFwG14mOKUgymBk0FijGs/x7fOKYiNIH4kROi4JIk+ecdvqO4DqADaq7Is7AEQfyf2eDpmEz4ahp9ADZa+EIKVQBrQIpxgLaiNYNdKfiYpBRNhf4AhAtQxH8CxDAaTSlCFm3tgdZgXo4U2qrJUARFN1OXw0+Fuz7kEzAK0F9ZBKCHdZKBVnxtFQEwMWLOY5A4YA6RgB6Y9cJQ8CGICAw6bRtihDMhiYwu2k0wRgL6vIqQPuBZN8PQ7KGLEI7sPAQnhghhipAJf0NamBguVKQy5JxJ2WQzLxMjLRtirYCZWQVT3QVGQMRwmVs0jMgqThECXy5P/UnMXhjwKEIyXykbRjSaighBkFWnYoxneF5dhCC8nKUbxnBMU4TAsRkSTBAPhOpRhGC0RH0Y/RFo/knRNvYsmgdf1irp4H/WH8Qn4I9DRMDj4W8oRmMX2jiGjpEOnmIoULgTguEeN316MBIptHL0JuKs1TBGHbwAYijUYRixJN4jiBA40Mn6HcEkZmQSIfioKLOPeVK0GWMq4BqHGhTigsixixBoGxtTwBNu8BQhGAGjTcH0TWNLIqTDNPdhGO3TiggZjEf0aYwS+K5UkYOBmTqVxHALGg0eTB8svWFYYHhrtHgY/IgHhtFCt3iSQqNrHEGMdupNZOTUW1HMRFeh913oTQUNh4jhi0C7EENXj6EfCfUGzV5ESLscOWKYEQx2YhrDbpQe5BAZ7Luhv4hBgikXRcbQcasYbUMVVZgwpSslGKMEHQ8tWRl6B5tOwNAGEx3MKA8f07VFMDZyC5j6wPAiomh9YvQjxjQxZlDWiR2HOEYYBJWpGQS32u9otoloXYTAUMYcDcvkFyHw17s+nRjmo4dESKcZiB5v6wNP6C35oPYajq53Pqh94EGtNRxdTXdQa2dHV6cc1HrQ0dXdB7WhcVAbR46uQXdQG6GOruF8UBv7B7V54uiaVAe1Gejomq4Htbnt6FIEBzWtcVBTMTu99dFBTXk5ujTdQU0tOrp06EFN4R7UtLOjS5Uf1NS+o8tEHNTsyUHN+Di6LNVBzaw5umzgQc1gOrqs60HNFB/U7Lajy8gf1DwCR5f7cFDzNQ5qjslGbx/GQ3N3dng5Rp6ZC7XD5W15qH6ZZ+bB7XA5ex6qW7iz5UB6aL7mDs0p9cicVw/Nz93h5RF7Zr7zDpeb7aE65J6Z677D5eV7qPHAzlbk4KEFGRu84xEPLXLx0IKcHa54yEMLnXa4oiwPLSDz0GK3Ha4wz0OLCHe4gkcPLc7c4QpJPbTo1UMLdHe4YmIPLXze4Qq0PbSYfIcrfPfQIn0PLSmww5U/8NBSDRu8sxIeWgLDQ8t17HClRTy0DMoOV7LFQ8vL7HClcDy0bI+Hlhja4coheWjpph2uxJSHlsPy0NJdO1yZMQ8tibbDlW/z0FJzO1xZPA8t4eeh5QZ3uNKIHlrGcYN3btJDS2N6qAnPna3UqIeWRd3hlW/1zDKzO1xJXA813euZJYZ3uHLIHmqueWcrK+2hJbB3eKW6PbOkuIeWPt/hlWj3zFLyO1zZew8lze+RrQfscC0deKiLDBu7FyM8tHWLHa4lDg9tNcRDWzfZ4Vph8dDWYna4lm08tBUeD20xaIdr3chDW2La4VqN8tDWrXa4lrg8tNUwD23hbIdrjc1DW47b4Vq589DW+Ha4lgM9tJVDD22RcYP3eqSHtnS5w7XK6aEtiHpoa6c7XMusHtqK7A7X4q2Hts67w7Uk7KGtHntoC807XEvSHtrq9Q7XQreHtibuoS2f73CttHtoa/I7XMv3HtpK/w7XpgAPbf+Ah9dWgxtuuxIcvDYwbPDe6+DgtS3CQdtAsbG+tlo4eO3K2ODav+HYtdNjg/emEAdt94hj1z6TDd5bUhxM5w7J/3h5+dXLbx7/4M6cyE02E4ZmwonkdpxQCz1P2qjvvj7+88P9599+/qn//MKNN8zK1Eqtdr/Mh53Cuwj0GR6awZlt0tx2rkgNSx0njKYBuzWIEXU02fZBPGATMO47B3vVpQloG8xQSKwPuDiIUrqsPRNzfQnzie5Q6JY/Tfz5Qk8CGK6grpMSw4Okke9ctcjB3gTaNCBSgJAq6+qSoUtQvCmyz7mLMOSmKWJgrgxCVdB/w6vaL1au3SNKe8DX4y6lmAzXjpA+EeMXQpyG52QWg7jFoTnBBJsSmbchHXNeovGXzFTVozOFjXdSDJ1V8e2VuLChpmEYSk6oLgnlmLX9BvchcPJ1priqrq8Sw1lshLARyUTAsmZ2HzGCmF40bQdtDCeniIgx4ZNVw3Cr2UyTm8y6LP0i/udfEPET45MsQQzcguS+u2ykqrI6GTN3VFT+zuAeIsRC0XBBuAUzx5AAQ9BkR9kTDJMI3KBi9UUyhshII4oQ5k41DQ7cA5oYQqJsA5LVMrjjXHVlkMFoJaMTpmHMfTg6xBgr1YRgiNQxYe4RBeEFuixEc3shF1Mxchgcoc1GN9wx5PHJQ5aisya2M9e0KgOoodpZM8TALdeAlgDOuoqDccc5F+DIMGrLuWuSGbhAYCWdeOnZjY6ZoItkE2aelwy4rzHCXSSufL1uGCKoSIBH6UVHH7RmLJNmeXAsJ92GQFxhB+DTAyPgGJpNBoZvGmclhtcyij6NzsbIQaww6P1ioE3DA04EGg04i2Uh5marUjiOGC3Dzde0MfcTNSgKEcJtA7pUwq00cD/xFwTcWV19xRzlGObAGK1N9n8Qo0kyYqbBtZLR7Wm4tmggTPEhwV/QpRxuDqg0L8Q1SwKGmIvw6OtJPGQkKoW+wtR7MMWAJtFnEXpxaa6SFm6ISoYLfDJMZeDeh+x2IIVHyQ00U7bHtaANwgXOhIYHhZNSNM/PZUc4F1CgTJPEaJl7WQsM3HkwOX1StWZCAJom+oCYW9ZNNNfQBqwdceMGRH0aTdk6Fy+BEXh2aw+MkdHohk/MKuhm1cJcGkJAg+EFDGXSdVZz+SZxIBEPbuFRPLmCCeGPiXacVD+GmbjEfJv0WGdXDSVLG3HETDwQm8qKfeTyA7y13LjZFfojyHI5MV5jYHhNNjDcH3lvuKbMwGLoTOrpGWUvDzE/LUBIpXZO6+nJ3ZykGV+rgxWOMDx7WqnJDe012Yvg0/CSXWRMeMWGEUdDITOXB1dTdQuYbJPLxNwRrdMUrjwsG5eEgfFwVhsBjBeFr/aYnXnaqW0NDAqnlLjyq1QIXhWGDaMZmO3Vi+Fe8H4QMriBttcDbx7Fc2qSHV2vcVB754PaBzq6WuOgyTuNWwsf1LrD0avrDmjdfFAbE46uAXRQG207vYfmQW0cH9QGvaNrhhzUppOja+4d1Caqo2tWH9RUwEFNXzi6lMtBVRE5uJTWQU3BHdS0oaNLcx7UtKyjSyUf1PS3o0vZH9Qsw0HNjDi6bM5BzUDt9LZmBzXT5+iykwc1o3pQs8COLnN9ULPtji5H4KDqNBzQHAxHly9yUHNcHF1ezkHNJXJ0+U8HNWfroOaZObrcuIOaz+fochAPat7kQc31dHT5qQc1p3anywE+oDnLji7P+qDihB/M/HVHl3N/UIsEHF1hw0EtxnB0xSMHteDloBbpOLrCooNaDOXoircOasHZQS2Sc3SFfQe1GNHRFVAe1KJPR1eoetAz2P2hYTl3XiSLpR/MiaLRufGqvLj/ogdm3n7S/xc9MLP+/vL8SYnbK2xA625198NOI1OtCIt1Ewh4HQgyknDMXouja5dDY1FPRWHsDvXnK3dHcR0OHEYXjZKycQ4jDANyeLB5qrUd3LEOvR8l4zzMtYOthK2FK0VMP0ndbrwCHBFuzowtM92ezVHidhtuuMC4ZrIz6lYVvnEeXBMkhx+H4Ju8MZTsJQJX/nzUXT2NZxAaFxvJR2lRpfNDZuXKJDu0pN40aQCOSLLwU9karcnmZr4xXpFhLXmFm6ibWxr3RSfazsjlB8TfOrYadxrD9RLcE9xYaXl8INxJ+tGxcVOLndoghzdZ5xQ+WtCkBD4QGqOhtXksFD0QNT5tjPvwVU04QvY4Lj55pgpiuGaTRxgqHi7gqHSrMHCggBDQVuMYT9y+Q97ytE1jaJGUMkP+2KG1uRRXjDdpKuEdClrjKTYVovvB54vEFbKplrzwHZJwZkmskTvzIFBj4JXBTLL3H7B+g84bOZqha8TBNsyIsSiHidpikXBjnNboGJJPnuMrxmuv3HodmUTBKEoXn7PwI8mpskU82xZjF8oT38DNCBZEdZ5JyVwuJZ88MCE/i7ZFl3NtNTILkmE7unFMt8DBw2XNnKpxaXOuiMTBboTLOYxjRnNbPvmkQmzGJ9oW34gnX7liqp/FPBt6C14h+eB2rGwcHqTM80E/CipNf5dHvCt3gpDDqGtChbmzUGSayyox5pByxFCJLUXMlWvtLGD4SnyFyDPjLerIlw7C6OyCYVWD7sxERyQezYxyHjKUGnVXIDNiPHTThBf2pjZmh0YZhidzC9Ew4nVMdHAq3NmsbdBvtXEbNXkv1/ZHZroQHPHxzuC9a5ISuM/CEyXklbpFxUzuQy69C6dwEzMxDdixA/KYe5N3ZPaqRm6CIO+XAbnxblbeppfgg9lLOLpe+aD6fQ6uxjioNd1BtZ0dXJ1yUOtCR1eHH9RGx0F1KDm4xt1BbZA6uob0QW0COLqmy0Ftch3UpqKja+Ie1Ka5o0spHNRUyE5vfXNQ004HNV3m6NJ8BzU96ejSqgc1HXxQ09iOLv1+ULMGji7bcVCzNI4uu3RQs2IHNZvn6DKQBzVz6uhlew9ohvqgZtYdvXyAA5rD4OhyLw6qvojLQSzH5aDq5RzQXCJHlwN1UHW2HFye2UHNjzuoeX2OLh/xoKef+cNd5iHLUNwvIhU+0izcLcXD4XZOHLZDXeYnTz7ck+Iy30++PH9SXGZa4s71L+cy35R7ZfH6eiCbfHDjYRZek+xRJo90cLm5jrzDoUiG4fQxUyPbc3W7N5cYII7mOxbuZVInBrjCitFp4wZfvLDaXfAJf5VeOnjhznMVAyeZ+7abcLhmWQ3yYHoVHygYhrFoSnqIzN6U8zR7T8YbHCza71hlZ1VQ84W/yL5LYoRXXb132myE9LkIhzbI2TDiKjiID9vfXEw6l8WaeGwsH4Jv1YU92uxaMWeFM4DTl4Rx6oX+qeydLkNTyjTZM9Np5E7rwKMl2gSIohGK040GR1/k3Ixj1IcQhQOar8Isf42Brg04op2svsdgxQ7m44UjKNb4aLDEwBR/LdJAFuvViVEURpj6s4M7ModxKJaoj3MTkfoYk8tVTdy1iDkWp3mzzN/zPH0XjjBUD4qTIywXdw0cLW9eOlP1IYu/FpkesvOt5Jj4iV46eAl29J+FXwKGS8vKefB+GC/w0ob+buWewWh8dMqUDfGllqCviYHDo+36+p1O9DAO/xj6TfmYNhRmfY255aFyuPavefDJY1i1BH19jBb1Nplt74XZQcG92trbRMfNLB4bd+2nqUl2YNiwIl0+5PiDxqDMrWdpGubHW7V371yFr9Lh4M0Od8U5uGTP2g3kCd5mTcYL1zeS8qLHs4gxDOpU8ehiPcTMii9Qrpy/wicPXhqHguGeXHKORDlgjP/PDQlNenxwvViz++SIAJuKyYiq5ZtSiNzowON/whsLQBgvnJDDODfQeb5p5efUZDt6vcgB7a13E3B/40GtRRxd7XdQa+yDWtc4ujryoOnMKN0D5KA2mhxdY++gNlAPasPa0TUJDmpTxtFrfh3QJuNBbeo6uib6QU0tOLqUyEFN5Ti6FNRBVZsd0FSfC3WWojyoaVVHlw4+qGnsg5p+d3RZg4Oq6XBwmZmDmlFydJmwg5q9O2g4DyVslvSgZncdvYz0Ac2iO7rs/0HNWziouRaOLi/koOayHFGrOTgHPZ2kH+7vVdt4hHfnMvjlqbGmkHhp+CVLkT558uGeFH/vfvLl+ZP096j4BiLlffnhw04ZPSFsHqqBE3eq8aSV4pCK4QYnIUzFUG2qfzN3wGHeCy5NN64QY8rRlRGMVsmGJ2sbRMG5DElUplBkK0wogmHJzBYU7thDfCs42uH4FCp3982q7x1qlswUceO26EYMo6RFbOBhsBifsA5/MRsrNXD5lhj65XpyIAgt8s5QkVobB4byFe5n6CojtWCGB38JjMsEIxAr+rSYYtgo4kaXvBquPAIv79x4ZvJ6eja2jmC0TNEvhEUZmdt3muz/qDMZpu2P8tpwfocY9hQDfVVEqYJrktQyaeNamLw2nXZxPVJkSZeZtHfLUA+DtKATg7y1WLhpeMzOE/pNDgkXceLwW9zDmqu8deZYmIZhFbLIyEW9W7wsd8ay7gQpfZduGPOtaNdyG/Sohid6rspLp1Sz+EVJYgD8uowPTOIZpmG8c5nSTDF3PZLNmn2pYHjIWzMNEqJhjKBa5K15XqboTzYm2DlQuTEHhrEbhTtVZxFc0EXTMDqU9TKaLEtqAYVE1xTREnuXK1TsUsNSg7ASY0JpFZZErwyDHMOTGEbTPpIb02Jn78LxBp/6fpMbsDucfOKokQ0pd4iwd5tY7jINQ5nDCD5k935Rrxudzz3gI8gvwuYMHavAmK8yd6G08MeumOscdVT5xRyqzo7EpHqBMSflnjOd0YlbbPIMIoP7cHUEp8Tt7TOJDGaXdDQkpmLhVshrc8ueePMw79w1P9m/3HZ8DeEb767ZE7okH9Rew9H1zge1DzyoNYajq+UOas2807tPDmodeFDrbEfXyDioDSNH15g7qA1QR9doPqgN/YPqNHFwTamD2vxzdE3Wg9rMdnSpgYOazjioKRhHL110QNNbji4ld1DTiAc17enoUrUHNbXsAoulww9qCt/RZR0OaqbkoGZ3HF1G6qBm0Rxd5u+gZioPGvLxz5f5PajZakeXYT+oeQGOLpfhoOZfHNScEUeX53JQc3OOEPH0lH6E02cbzPvAX7jflSC9sByfOWiW43v7wYd7UF2+9eDL0wfF44OSbCwbtrXPh51yk9XMXa1GKrCHkzUfiBGImsYvUg6hFCtWMiUxQDwGRpVQBL9mBypMRePWU54xmaWZwq+wcDVq7ZeBgMlwY6WKqNVRMKNbN0qLoNVf4O8VdVmAZZl3aokVvqBg5n0iZxJxS3V0wwOdywoSTc7fqHVNg0YrKaWGjUZbHSxTw1ovdBq1QSaPubHqEnENI2bDXKLXh5dHBTpZS0kwlzZk3TlldhAaXt66sorsMMwDjqz3BAzXVapkYFTReBZ9vwpfXspKENcZadyLbHjSbFnKiccCuRONuNiCPDHielbSIkbTaIsAw8+mPpaTPviH8jU50/GqRYRwY0+6cGPdQa2rE7v1OsZ5xAdP6RsoQK0DRlwT4mj5SfhrvXbDs2S6QcSJixqCuSc10ThpeR9dvySGa05LRgxvRv2kzPQmtJQIYVmqMQ1zk3gWIaw0oc4MxnngarQ8jTdN9oL0Z3vXOkMY4rkPw3h2BKlfNEfXnc34iWtjAnFNU+MF4A7Pq2pdI+7YUtmTR2CHFUEaWbdREOPnpc4ed12MHqPh2aEMRQgz1+ptSRVNamjWXOpDKzsmKRSJiaqlmOCUaqAkRRLFaScOLAJtuBVWPuTORxYRkfaT0ntd5q9sgYs6MFm4bnAnmBSJmpBWDLOi8BTKVXf9QR5voYUWDAOjE4R1zUpWiiCiaqqPRcUQcTd56cJESjaMYCUOK2BVtTpeYuksKTTDYld9WLwgj7DkmdbAilUjETZZHtxtTRyKhWwy6KTMapWtF8YaXCAtw4XZqVu0EysrwetLIiGyHuqFoSKyND9+bGgXLrrbhmfQxDq63uGg+rYOri87qDXDQbXJHFzNe1DrC0evfjug9bGj13g4oI2dg9pAc/QalAe0AezoGu0Hzeci9jaLDmpT7qA2Px1dk/mgNvMdXWrioKZTDmoKyNGlrQ5qqs3RpQcPakrT0aVhD2rq+KCmux1div6gZhUcXSbkoGZvDmrGydFlyQ5qZs/RZSMPagbVxVzL+h5ULfUBzao7ulyAg6q74OByLQ5qfoijy2k5qHk4B43nkc3NczqouVmOXi7ZAc19O6j5ekdMfbqLP2Jt+/JRy0f104VmGGFb2377yYd7Ute215Mf1U/fZb6IQuaq3K5LPuyUJ37ho5mKkaUn8cairC5qqohF2qAjWH+TpYaj1j1LrK+G8MQqEEORpwujR4pWFqYPpJiVytIoU4XkUjV7ySJjmBxRhfDUeDQ8MMeLFlKGMxb06UhnqzYRAgdfC0oTQ99WrQUczZOoicVhmlYNjpk3TRiGWPHFmMuYYUbDs0Vu/ZJ6ztPyuTWzQE3TysGBGeZsuOOZKEJCiyHqT7Jg6OAGT2KEqprtrKzW2bvWDw7XIcxUeVCpdSssHaxWM3HhVlYpXy4Fxath+HZDa6jzLLo68pVeKeuBCC5Fj1sS98zCJ1oLW9dCYVVZKGhoFfXJSokXrswWiowx0GTZsLhDVnu7dfX+6mCpIvHFAk/MJXVPgUdCo0ihbfxJ930ipnzFaNPS5COMovkwUIxXet4sDD64A5iYFzCMyq2YxE1UluIapD4pNWHQiuOk8AhiVBk52BhuGLmskt+1PHmsumDKXckIyJvWMp+sviY4sWRkHFoOHWNY9iwQ14mHrnroWs48NQTUkbcTCIb2jUYLY0+VEasewiSGR8/trFJ/vQ6NShrLx/JgsuDeLHHIDcywYFpVvdZuPdYqK21lLb1fsx7NJIWPxYEhx/BnGheeLMWtMiRLQcrzmHQRtb48D14YRsMX7d3CRUXpmSYH1IqWVkcslTRAYOIdXrfWVi+sAKOyB0uUVa2uXmBUo1FWa9UC/HnYuW/iMQo3whIjvFR/uHE/OuJBraBfGE8Y7qzFrEI0VQbcA0uwNavCTz+vGGaxdS3DjzjFXptbovvkQglxL7qxAJ44K8F1LcTP6vx1GB6RR1f1OoFuWdrO6tp0AvTugWnJZW6TrvR7BIcY7WlWgC5DL1bAYBkXLTzTVPWug6LLQqATwaAW44+lXd/IWsKynEGchlYKJu68CUBlxGCKoVcW52O5Zym/APVdDFdGg/Ixges3+nRj2b+pVznU1uxbuO+d2z8F52EZCB5SL8y1Ck7Rwp3ONYqqCoo10nXRpHOBp7A2vFzuw6I8gge8Lc5OwbIEZBj+E3fgE8OxLtfTTN0wU8k9crlq62GI4DWa3NiBWCVpjMsD5hhFQ2RAU2lpAOhRWAx4Jno/x7Cz98TQbEmv/cCw6YYj60HKQTjWIerdhEQWj0xdheRguSFutkbIoNdxIGo1LTcSB11OTe8J4SUmhhF+5iJCWJlc7cHIPGDOuupy10jWwsI73nyRJ/SW7On1Gju939nT6wM9vVpjp3fTeXq180a3TvH06sGd3r3t6TU0PL3G0U7vQeepDlAH12A+qI38g+oscXDNqIPa9HN0zdWD2sR2dGmBg5rGOKhpF0eXKjqo6S1Hl5I7qGlER5f6PKjp2oOaYt7prcUPairf0WUfDmrG5KBmeBxdVuqgZtIcXfbvoGYsHV2G9aBmhQ9qFtvRZd4Par6Ao8txOKh5GQc1j8TR5b4c1HwdR5djdFDzohxdLtdB1T07oLlyO739voOqj+jg8icPas6no8tTPai5tQc1H9jR5S8f1JxrR5cnflBz2w9qPr6jKyA4qEUPjq5Q46AWlzi6gpiDWsRzUAuPHF2R1EEt7HJ0xWgHtYDuoBb9uZTWChUPanGloysIPahFrI5+HPT+8Ph9q5L09Yv76+fnfxVrXavcL7OH2xuVG8f0Zg3iCa0Q1T2AGlCndRQeXC1F3QMoVfV2WMoFur+JkMaNpmqFmdWFG1f0OrSocSso14ui+FHoJNtOypIt3NYr94bhH1lYPXgNQObyAzHLN1fDMyTevUIMByAo7jzy26IKCVFP1BF3WmIRwssvDfMs5GxVhNTG5jLMzPcQIdwg3K6nMdi5/5w4D71AKbHeSmySK+bNgVnvJSMeLB+tQpjNEtmTTn3h8Uue2OelaUYRwkplNdYr6lGDpinXN0kVNqn2EDTDPqOcei0qA8ZDAx6WUIET0lRIhv+jTydGLlx7JI496FITcOcinwpB5K+U5yj7LCIj876NZhhfO9Ur5NVeugdoFp5e5+4XYqZku2EG+9rBiESHfXqhaxt07MDc6JUXaWKQwIltKoR3zgzDULthyABkVQwNVibXl2KMIoSrkbkbrkWqTRGzWmMxPGHKqwhh5TtdQJmdyxg8jUpc7Jo9YvhVXJIhzmFaSw3oPj27wpNwtvoB2nKV2oSTO1wtX8FbNisPJxFDVQRd0pjs1iw1D1kHmmlMw+gCqY84X+XGErkDMEj4mPUSvlimfjuxlXOSdLmF5zlEPsPsT5aawzq4iWsbUs9xMiSTmUo66XGLDG7N0RsGA69TYtwhWM4fG+YylyqHgGEZVEhmq1VVDixyOC6MV6pFhZQmu7YQHbPiRVXdEHIees0gMIaI9gzCNz1TSoqJ36KKCDwrK5jHb2ejY8hJ2PScMrHcfSIU88FemrubMLazYIQGPRnGXOdGS+Jm1XIQerBUSNeHWQJgGMXw5NFS4tyKfYvuvuQyxeCNh5oEwqSTHKPAyEI8BtuMUrGU+8uj3l2BUcXO4LYulhQcU09PZLl1VOqt8r4AKQ2glJVzqUCHpLkkIZij5Sj06Vo1ZCfuMsSlWCEP8AqNLPMym+Kc9SwBceNVTiojxWkyEnqu0IdnnfowZfiS4lslQQ3Mha1heCDIaiKD1dy74iwFsqYI4eKLjhqePGIuQoT01vXiIHTDK69oyyqEXtEwzELETYVIhTnDk/pRhWSDGB7Mw6oIuFGxG6bNySriyt9mnsWBA1lVRKjWTNw+ILmZzI6rQQckj8RgCGmD8LJjvVCTJ1amWBdizCIdH3JyJOSiQhDm2YvwlMWUysRcwqhqUIgRL2XtMN4np9M8UhV0qXgsOqaNZLiy1o4KgRMrCa2cmOeQEUAcxdoqLkluhyQOPVwPj5KlQDNLE3fd6QCLKXf3ZZHBO06bUdrsKiJgEPXiIigrnvqjApM6MtPuKAWuLFeqMrjwWAwjUpTQa/DWvZb0RTKPHvLAEnHhFXuGMXM5JYi5a0baLxUeapQIkPF3UfNDjOiNuRJiWn1pqFSpZnh+hxhuSZiGudDNpMKga6QlEIh5MW5IioOmIHOSiloS4vIciFZ1IOX4LCqj61EjDBpmwpieJUUPaK8nelld4mze9tGuT+dB7sbbuIjZ/4oH9+cycy44JJPN+QYFLFqrZLtiNU2edaWBlxKkxeZMouPEaj2CY9OHM4sa0kIJxTep5CwZ11BVRgi6Z4N4skCtyOC6mhSigN8FkzSjas88ZO+X4p6H1ITnyW/MPRk6OXGpI6r6lO2q0TA6XWrNy4EhLX5BDB9C6tIPqfGq5iRn3qGVVIFyl/IYhqHFpMAqDynp3lt6ybDFqStMegkRKTdZTBWR5FwsceVmY6m4L0eU05yGa5AbGkhjkjUPUvhgWS1E5sqJiuZOnC5+s5yt0ptriDGAtRd5zafO9My90LUUkQFhRadj5pbxUlR/8kDz1NfjgbKsgyw1GW5KeQFSVBksETAMD/SRqs/EOxD0F+kdhapvB/dPrb9451w7V0wtSCw5W9axEIz+UtF6j2vTh2E3VAaXZypjK8GpRL3iOPLgf1PLCCdJT4ES91kMRpvlvPyRWUOVALmy8JLlksXU9Wlek6btz1rCNXaj1wxg2eEeuupOpq16N8zTl2oZI0/bq4zCDe1DdWfscNum4c7VIBXSoKFViNwJN9Q0chdquTAvSVblyZW8dmF2kdpGmBstEZjlUjMMD9FZkTfcXhgTcKpt5D17qmtZArrAOKoQboWrhluLQbUnxrFehpjlRq0Q1DbGVDTkIKajr7aRHaqahTdTRd5iqDhn9U9Y3homSa1jRFO2ZLiz7G1VnDQBnOXWphrVPNJvTEapstQ68r4Ju9kacQuMoypPOId6KSsxNFhS6xiGlVkhHjUmtY4BjpsOHVYE54WqKqRblR9ihC1ZrWPg+o0KyTz5lXXioWXsa1iVPPWs1hG+eFClzwrm6BK1jnCIkqyWZVY7b3IglLjo/Wuk8GGKWsdQLt1SK93+ouYRnW6uEguu8xtVBmvGyOBmcfbCM7eCEbjYtzeer5N6z8TlaqnGs3hVzSPr0agSZoH4yWrwiqve20WM8c8bkAXr/dOZdeczq8oJRZPoxGGNeqZLmmILZ4hZaVdT7tzfo64cCzJhWmgOg1OyG4XO75rLR3SitY0yK+qzOK3K4G1PRhvjXxWBEFsls1A/nQeVwZM92TDMu10pxXsw9Bcbr7zl5gHBzc6mZBZuSs0uq+LCszoGvFWgVLvZiuUY9BczV7emZgjRuFpCkxgK0WobTMxfe7/CdW+7Xwten27JI4ZjPXUPyixavZYUysUu7mIFHnXCpMSTlbPDD+hlqKRoZ7sRjGe8Zc0485gFmkfTcRPxud3njrAlWooN0XpV7cKjLmgU3TbDFeOhohHYVjnYnaXsbGoX7t2u0Jgsu6M/qPrUXo8xdTIMNZN0m9wM9ixDN8tLTd49FU96pbKewF3oTu83uOn2tju8v+ymezPsdDXZDu/mveneFztd/XbDvY93eo+Hnd6D56b7SNvpPSxvug/hnd7j/ab75NjpPZN2ek+7m+5zdKf3fF7UTf6d3oriprtW2emlgjy99NVOb93m6aUId3prTU8vFevppY93eitvTy9Nv9PbLHh6mZCd3vbG08s4eXpZsp3eZs/Ty0bu9Daonl7W19PLVO/0tuueXj7ARjeHwdPLu9jp7Yp4evktnl5Ozk5vj8jTy33a6e1reXo5Zju9vThPL5fP08s/3OntTHp6eZ47vd1UTy+f1lPzf3d4+8qeXo71Ti8f3LPLXd/p7dp7amGAh1fIsNEtvvDUYpEd3nGLp1eQ42k4NxTswZOnV6S10xWVeXhFcDu9wz1PLTT08Aojd3rHnJ5afLrDO5T19Ip7d3oHyZ5eEbWnV/i90ztW9/QK7Hd6ZwE8vVIGnl75hY1uuQhPr8TFTu8kh6dXRmSnd/rE0yvX4umVmNnpncTx9Er47PTODnl6pZI8vfJOO72TVJ5eGa2d3ukvT69c2U7vxJqnVxbO0ytlt9M7v+fplQvc6Z049PRKMnp6ZSQ3uqUvPb1ynTu9E6OeXlnUnd4pV0+v/KynVzJ3p3fm19MrS7zTO6Xs6ZV/3umdqvb0ymt7eiXBd3pnzD290us7vXPxnl6Je08tyb/De0HA02v1YKdrpcHDa1Vio9sShqe23OHhtTSy03sdxdN0nM/ZV2c8vZZyPLVlnx3eS0SeXutJO11rTx5e61Q7vRe1PLUFMA+vxbKd3gtrnl6rcDu9l+w8vdb3dnovBnp6LRx6eq0y7vRekvT0Wr+86b7Y6em1MurptYy603vN1dNrfXan92Kup9fK707vZWJPrzVlT68F6J3eq9WeXkvbO73XwT29Fs09vVbYd3ovx3t6rd3v9F7n9/TaFLDTeweBp9d2A0+vvQk7vTcyeHptetjotkPC02s7xU7vvReeXhs1PL12dez03gLi6bVfZKf35hJPr50onl7bVnZ673Hx9NoPs9N784yn106bnd7bcjw9d/b8w3uK4uNfHujox99ZhDJZGcr60REeXi7GLQaTh30++/VXf/vmy69++/mvHl/++Y3r4z5sjBuAEMYw7/7yrz/pe7GCfuR6jjtntFFexsKLouMPf7dPXXj35gu99/66n+R93riR58OzS31+eOe955qAN97tH7kz4Kd+uTeKaH94Vof7B7/bu4rFvvlu768c+xO/3FtVHz88rRz5g1/uXWXX3ny5d9dg+6nf7Y2SSh+eVWX64e/2jtolb77auwuZ/MSv9nE9gA9P6gn88InwnmOvb77Zu8/A/tTv9sbxng/PTgj94Hfb9yy/fPwC796x/P/jBd5Yy/rwZC3sh/7842z1t1ymD9u90TzMWHOURr+vK+cw0ZqL9932rAARpu5+HFQrsm1XLkELU5b1Jo9Asr7eo/BG2qgbBhB7sdZIlRlbRtGLMQJ3Aw0ughRuoYh2BwtLQ8/OlS5uox8h6QWdbztXz67offuK2Se3nj65LPPZFYRPLhd6elPOk7tjnl6y8uyWkicXgDy/o+Ptgu7PyqL3t0uOPy3m/axK9tsVqD9R8vlZaec365O+XabzWa3Kt0sz/sOFCN+qm/Wk+NHTkjJvVSB4exP8s82nT/ZlPVuh/9719VPTJOiQ8PgTtTv0yD8QibH0Gu+nM4vyPTrufPotyW+t+O/P3jI+ITnwe9AJD14P/if87+/215f/fkE08/hFAON1D5ibTf75tH/+5YeXX33x+Oyf4yOmxxd/hKTw+OIPL//2+Nmvv/rvv/7uP//yzf/87i/f/Nd/Pv7wzZ//8t03v/+r/OWP//XdYxB9+d3PH0lmt/3f42df/eWrx99+9+1fv/rzzx///vLFv7z80xcvv3n57JdF7zb/ejU87yr/+kVS9FAyHCddqobbpyb71AW/leX+3AqXyBaV7QL6IG9XqYUp1MWoGuSxL19uKHXErfF2yg0VMIjwJu7fIdVn7ze62Zf7299YVw0winmGapOQ4v7s9WM7bZvghddHfLvT7XPvH7ub5Y1W/VLTGvd9disj8FFZkjsjoPfZvf3ky5k7eLfMF1nxuT4bChnK7YMsDnv2rWNUxLo7+GNmn5bs066pIH/8/I3ZwK0C/re2qRD8VAgYyY97JGcdyc77cUmC73SQf+oRNFZ8+ZQE30K86WgODj7fSjf3LdXnxtPb/Me02P27z1stvtFq77wa8btLS7w3tfBuub5Zeb6UC3ZHqy7sG5VXHF04vYl/TJOuH33eounNFn3XzTmrRd+bc3i3XN+ihX7csppvcd+mNWw8vc1/TKvev/u8WfObzfquAvWrWd+bkXi3XN+svGd2fDROL+qblNstxkej9KY/pjmvX3zemOXtxnxH5de7Ld+XpnivVN+SrLkcw8dz/ua+NXmv7OLpbf5jWvT+3edtWt+e9++pKnbP+3emMd4t1zXr5JnPcc76RX2Thnbj9Cb+EQ26fvN5c7Y3m3M/8G1t9skj4durI9CsrX30+Td2388lwYXTm/hHfP/9o88boO8N8JuX/wcx85PBCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTM0MzAKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjQgPj4Kc3RyZWFtCnicPZDBEUMhCETvVrElgIBAPclkcvi//2tAk1xkHWD3qTuBkFGHM8Nn4smD07E0cG8VjGsIryP0CE0Ck8DEwZp4DAsBp2GRYy7fVZZVp5Wumo2e171jQdVplzUNbdqB8q2PP8I13qPwGuweQgexKHRuZVoLmVg8a5w7zKPM535O23c9GK2m1Kw3ctnXPTrL1FBeWvuEzmi0/SfXL7sxXh+FFDkICmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNzAgPj4Kc3RyZWFtCnicPZBLEsMgDEP3nEJHAP+A87TT6YLcf1vLmXSDFGPLL0RXdOyVh8fGlI33aGNPhC1c5XQaTlMZj4u7Zl2gy2Ey02+8mrnAVGGR1eyi+hi8ofOsZoevVTMxhDeZEhpgKndyD/X1pzjt25KQbFdh0J0apLMwzJH8PRBTc9BziJH8I19ya2HQmeYXFy2rGa1lTNHsYapsLQzqjUF3yvXUeq7zMBHv8wPfQT5kCmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NCA+PgpzdHJlYW0KeJxFkU1yBSEIhPeeoi/wquRXPc+kUllM7r8NzbwkK1qF5gPTAhNH8BJD7ImVEx8yfC/oMny3MjvwOtmZcE+4blzDZcMzYVvgOyrLO15Dd7ZSP52hqu8aOd4uUjV0ZWSfeqGaC8yQiK4RWXQrl3VA05TuUuEabFuCFPVKrCedoDToEcrwd5RrfHUTT6+x5FTNIVrNrRMairBseEHUySQRtQ2LJ5ZzIVH5qhurOi5gkyXi9IDcoJVmfHpSSREwg3ysyWjMAjbQk7tnF8aaSx5Fjlc0mLA7STXwgPfitr73NnGP8xf4hXff/ysOfdcCPn8AS/5dBgplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMyID4+CnN0cmVhbQp4nDVRSW7EMAy7+xX8wADW7rwnxaCH9v/XUsoUCEAltrglYmMjAi8x+DmI3PiSNaMmfmdyV/wsT4VHwq3gSRSBl+FedoLLG8ZlPw4zH7yXVs6kxpMMyEU2PTwRMtglEDowuwZ12Gbaib4h4bMjUs1GltPXEvTSKgTKU7bf6YISbav6c/usC2372hNOdnvqSeUTiOeWrMBl4xWTxVgGPVG5SzF9kOpsoSehvCifg2w+aohElyhn4InBwSjQDuy57WfiVSFoXd2nbWOoRkrH078NTU2SCPlECWe2NO4W/n/Pvb7X+w9OIVQRCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzEgPj4Kc3RyZWFtCnicNU85kgQhDMt5hT4wVRjbQL+np7Y22Pl/upKZTpDwIcnTEx2ZeJkjI7Bmx9taZCBm4FNMxb/2tA8TqvfgHiKUiwthhpFw1qzjbp6OF/92lc9YB+82+IpZXhDYwkzWVxZnLtsFY2mcxDnJboxdE7GNda2nU1hHMKEMhHS2w5Qgc1Sk9MmOMuboOJEnnovv9tssdjl+DusLNo0hFef4KnqCNoOi7HnvAhpyQf9d3fgeRbvoJSAbCRbWUWLunOWEX712dB61KBJzQppBLhMhzekqphCaUKyzo6BSUXCpPqforJ9/5V9cLQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM5NSA+PgpzdHJlYW0KeJw9UktuxUAI2+cUXKDS8JvPeVJV3bz7b2tDUqkqvIkxxjB9ypC55UtdEnGFybderls8pnwuW1qZeYi7i40lPrbcl+4htl10LrE4HUfyCzKdKkSozarRofhCloUHkE7woQvCfTn+4y+AwdewDbjhPTJBsCTmKULGblEZmhJBEWHnkRWopFCfWcLfUe7r9zIFam+MpQtjHPQJtAVCbUjEAupAAETslFStkI5nJBO/Fd1nYhxg59GyAa4ZVESWe+zHiKnOqIy8RMQ+T036KJZMLVbGblMZX/yUjNR8dAUqqTTylPLQVbPQC1iJeRL2OfxI+OfWbCGGOm7W8onlHzPFMhLOYEs5YKGX40fg21l1Ea4dubjOdIEfldZwTLTrfsj1T/5021rNdbxyCKJA5U1B8LsOrkaxxMQyPp2NKXqiLLAamrxGM8FhEBHW98PIAxr9crwQNKdrIrRYIpu1YkSNimxzPb0E1kzvxTnWwxPCbO+d1qGyMzMqIYLauoZq60B2s77zcLafPzPoom0KZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzNiA+PgpzdHJlYW0KeJxNj0EOAzEIA+95hZ9AIEB4z1ZVD9v/X0vYdtMLHsmAbFEGgSWHeIcb4dHbD99FNhVn45xfUiliIZhPcJ8wUxyNKXfyY4+AcZRqLKdoeF5Lzk3DFy13Ey2lrZeTGW+47pf3R5VtkQ1Fzy0LQtdskvkygQd8GJhHdeNppcfd9myv9vwAzmw0SQplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nE1RSYoDMAy75xX6QCFek7ynQ5lD5//Xyg6FOQQJr5KTlphYCw8xhB8sPfiRIXM3/Rt+otm7WXqSydn/mOciU1H4UqguYkJdiBvPoRHwPaFrElmxvfE5LKOZc74HH4W4BDOhAWN9STK5qOaVIRNODHUcDlqkwrhrYsPiWtE8jdxu+0ZmZSaEDY9kQtwYgIgg6wKyGCyUNjYTMlnOA+0NyQ1aYNepG1GLgiuU1gl0olbEqszgs+bWdjdDLfLgqH3x+mhWl2CF0Uv1WHhfhT6YqZl27pJCeuFNOyLMHgqkMjstK7V7xOpugfo/y1Lw/cn3+B2vD838XJwKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDk0ID4+CnN0cmVhbQp4nEWNwRHAIAgE/1RBCQoK2k8mk4f2/40QMnxg5w7uhAULtnlGHwWVJl4VWAdKY9xQj0C94XItydwFD3Anf9rQVJyW03dpkUlVKdykEnn/DmcmkKh50WOd9wtj+yM8CmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA1NCA+PgpzdHJlYW0KeJwzNjNUMFAwsVQwMjZRMDY0AmIThRRDLqAIiJXLBRPLAbNAqnK4oMpzYKpyuDK40gAFGA4yCmVuZHN0cmVhbQplbmRvYmoKMjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MiA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlxAvqmJuUIuF0gMxMoBswyAtCWcgohngJggbRDFIBZEsZmJGUQdnAGRy+BKAwAl2xbJCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0NyA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlyWEFYuF0wsB8wC0ZZwCiKewZUGALlnDScKZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIyID4+CnN0cmVhbQp4nDVRu23FMAzsNQUXMCB+Jc3jIEiRt3+bO9qpSNO8H1VeMqVcLnXJKllh8qVDdYqmfJ5mpvwO9ZDjmB7ZIbpT1pZ7GBaWiXlKHbGaLPdwCza+AJoScwvx9wjwK4BRwESgbvH3D7pZEkAaFPwU6JqrllhiAg2Lha3ZFeJW3SlYuKv4diS5BwlyMVnoUw5Fiim3wHwZLNmRWpzrclkK/259AhphhTjss4tE4HnAA0wk/mSAbM8+W+zq6kU2doY46dCAi4CbzSQBQVM4qz64Yftqu+bnmSgnODnWr6Ixvg1O5ktS3le5x8+gQd74Mzxnd45QDppQCPTdAiCH3cBGhD61z8AuA7ZJu3djSvmcZCm+BDYK9qhTHcrwYuzMVm/Y/MfoymZRbJCV9dHpDsrcoBNiHm9koVuytvs3D7N9/wFfGXtkCmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTggPj4Kc3RyZWFtCnicPVC5jQQxDMtdhRpYwHrtqWcWi0um//RI+fYi0RZFUio1mZIpL3WUJVlT3jp8lsQOeYblbmQ2JSpFL5OwJffQCvF9ieYU993VlrNDNJdoOX4LMyqqGx3TSzaacCoTuqDcwzP6DW10A1aHHrFbINCkYNe2IHLHDxgMwZkTiyIMSk0G/65yj59eixs+w/FDFJGSDuY1/1j98nMNr1OPJ5Fub77iXpypDgMRHJKavCNdWLEuEhFpNUFNz8BaLYC7t17+G7QjugxA9onEcZpSjqG/a3Clzy/lJ1PYCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MyA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDUxID4+CnN0cmVhbQp4nDM2tFAwUDA0MAeSRoZAlpGJQoohF0gAxMzlggnmgFkGQBqiOAeuJocrgysNAOG0DZgKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0MyA+PgpzdHJlYW0KeJxNUbutAzEM6z2FFjjA+tm+eS54eMVl/zaknASpREMUScnDU7pkymF9SkZIji4PbRpLbLo8N0JTh4qCqWuJ6pSrmabMUyxN0PPeWa7mGOB7VTfU3/SIXgKRUYJVYYEOkDu4YPjZayZsUQsiMYZQM4BpwgpzuBIxBBmMtWcYlCoMTtXPKlf7L6dl2CqweDCdIj+ymminX7oceOspB0LY3JW7eiFNCO6NBmPMLFx3qbKdABxMdJmJjFi8DcfTIQwNXpoGrHDWjZggsRsjpQ9eBxnTsHdFHnW3GPG+W8aUu9XPfVF95l3tHwjBGyf4ewHKG11eCmVuZHN0cmVhbQplbmRvYmoKMzcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjAgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzNCA+PgpzdHJlYW0KeJwtUktyxSAM23MKXaAz+AfkPOl0uni9/7aSk0VGDmD0MeWGiUp8WSC3o9bEt43MQIXhr6vMhc9I28g6iMuQi7iSLYV7RCzkMcQ8xILvq/EeHvmszMmzB8Yv2XcPK/bUhGUh48UZ2mEVx2EV5FiwdSGqe3hTpMOpJNjji/8+xXMtBC18RtCAX+Sfr47g+ZIWafeYbdOuerBMO6qksBxsT3NeJl9aZ7k6Hs8Hyfau2BFSuwIUhbkzznPhKNNWRrQWdjZIalxsb479WErQhW5cRoojkJ+pIjygpMnMJgrij5wecioDYeqarnRyG1Vxp57MNZuLtzNJZuu+SLGZwnldOLP+DFNmtXknz3Ki1KkI77FnS9DQOa6evZZZaHSbE7ykhM/GTk9Ovlcz6yE5FQmpYlpXwWkUmWIJ2xJfU1FTmnoZ/vvy7vE7fv4BLHN8cwplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzAgPj4Kc3RyZWFtCnicMzM2UzBQsDACEqamhgrmRpYKKYZcQD6IlcsFE8sBs8wszIEsIwuQlhwuQwtjMG1ibKRgZmIGZFkgMSC6MrjSAJiaEwMKZW5kc3RyZWFtCmVuZG9iago0MCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMyMCA+PgpzdHJlYW0KeJw1UktuBTEI288puECl8E/O86qqi777b2sTvRVMMGDjKS9Z0ku+1CXbpcPkWx/3JbFC3o/tmsxSxfcWsxTPLa9HzxG3LQoEURM9WJkvFSLUz/ToOqhwSp+BVwi3FBu8g0kAg2r4Bx6lMyBQ50DGu2IyUgOCJNhzaXEIiXImiX+kvJ7fJ62kofQ9WZnL35NLpdAdTU7oAcXKxUmgXUn5oJmYSkSSl+t9sUL0hsCSPD5HMcmA7DaJbaIFJucepSXMxBQ6sMcCvGaa1VXoYMIehymMVwuzqB5s8lsTlaQdreMZ2TDeyzBTYqHhsAXU5mJlgu7l4zWvwojtUZNdw3Duls13CNFo/hsWyuBjFZKAR6exEg1pOMCIwJ5eOMVe8xM5DsCIY52aLAxjaCaneo6JwNCes6VhxsceWvXzD1TpfIcKZW5kc3RyZWFtCmVuZG9iago0MSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE4ID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iago0MiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzMyA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjQzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzQwID4+CnN0cmVhbQp4nDVSOW4EMQzr/Qp9IIBu2+/ZIEiR/L8NqdkUA3F0UpQ7WlR2y4eFVLXsdPm0ldoSN+R3ZYXECcmrEu1ShkiovFYh1e+ZMq+3NWcEyFKlwuSk5HHJgj/DpacLx/m2sa/lyB2PHlgVI6FEwDLFxOgals7usGZbfpZpwI94hJwr1i3HWAVSG9047Yr3oXktsgaIvZmWigodVokWfkHxoEeNffYYVFgg0e0cSXCMiVCRgHaB2kgMOXssdlEf9DMoMRPo2htF3EGBJZKYOcW6dPTf+NCxoP7YjDe/OirpW1pZY9I+G+2Uxiwy6XpY9HTz1seDCzTvovzn1QwSNGWNksYHrdo5hqKZUVZ4t0OTDc0xxyHzDp7DGQlK+jwUv48lEx2UyN8ODaF/Xx6jjJw23gLmoj9tFQcO4rPDXrmBFUoXa5L3AalM6IHp/6/xtb7X1x8d7YDGCmVuZHN0cmVhbQplbmRvYmoKNDQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTEgPj4Kc3RyZWFtCnicLVFJcgNBCLvPK/SEZqffY5crh+T/1wjKBwYNi0B0WuKgjJ8gLFe85ZGraMPfMzGC3wWHfivXbVjkQFQgSWNQNaF28Xr0HthxmAnMk9awDGasD/yMKdzoxeExGWe312XUEOxdrz2ZQcmsXMQlExdM1WEjZw4/mTIutHM9NyDnRliXYZBuVhozEo40hUghhaqbpM4EQRKMrkaNNnIU+6Uvj3SGVY2oMexzLW1fz004a9DsWKzy5JQeXXEuJxcvrBz09TYDF1FprPJASMD9bg/1c7KT33hL584W0+N7zcnywlRgxZvXbkA21eLfvIjj+4yv5+f5/ANfYFuICmVuZHN0cmVhbQplbmRvYmoKNDUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNzQgPj4Kc3RyZWFtCnicTZBJDkMhDEP3nMIXqIQzwOc8v6q6aO+/rUMHdYH85CBwPDzQcSQudGTojI4rmxzjwLMgY+LROP/JuD7EMUHdoi1Yl3bH2cwSc8IyMQK2RsnZPKLAD8dcCBJklx++wCAiXY/5VvNZk/TPtzvdj7q0Zl89osCJ7AjFsAFXgP26x4FLwvle0+SXKiVjE4fygeoiUjY7oRC1VOxyqoqz3ZsrcBX0/NFD7u0FtSM83wplbmRzdHJlYW0KZW5kb2JqCjQ2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzUgPj4Kc3RyZWFtCnicM7U0UjBQMDYAEqZmRgqmJuYKKYZcQD6IlctlaGQKZuVwGVmaKVhYABkmZuZQIZiGHC5jU3OgAUBFxqZgGqo/hyuDKw0AlZAS7wplbmRzdHJlYW0KZW5kb2JqCjQ3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTQxID4+CnN0cmVhbQp4nD2PwQ7DMAhD7/kK/0Ck2CmhfE+naofu/68jS7sLegJjjIXQ0BuqmsOGYJvjxdIlVGv4FMVAJTfImWAOpaTSHUeRemI4GFwetBuO4rHo+hG7kmZ90MZCuiVogHusU2ncpnETxB01Beop6pyjvBC5n6ln2DSS3TSzknO4Db97z1PX/6ervMv5Bb13Lv4KZW5kc3RyZWFtCmVuZG9iago0OCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc2ID4+CnN0cmVhbQp4nD2MOw6AMAxD95zCR2h+JAdCiIHef6UptIv99CTbxdFgWpECt8DJ5D6p03LPJDt8EJsh5FcbWrWuytKaDIuajL8N391N1wumOBfACmVuZHN0cmVhbQplbmRvYmoKNDkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTUgPj4Kc3RyZWFtCnicNVE5DgMhDOz3Ff5AJIwveE+iKM3+v82M0VYewVyGtJQhmfJSk6gh5VM+epkunLrc18xqNOeWtC1zgLi2vC+tksCJZoiDwWmYuAGaPAFD19GoUUMXHtDUpVMosNwEPoq3bg/dY7WBl7Yh54kgYigZLEHNqUUTFm3PJ6Q1v16LG96X7d3IU6XGlhiBBgFWOBzX6NfwlT1PJtF0FTLUqzXLGAkTRSI8+Y6m1RPrWjTSMhLUxhGsagO8O/0wTgAAE3HLAmSfSpSz5MRvsfSzBlf6/gGfR1SWCmVuZHN0cmVhbQplbmRvYmoKMTUgMCBvYmoKPDwgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucyAvQ2hhclByb2NzIDE2IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ1IC9oeXBoZW4gL3BlcmlvZCA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXgKL3NldmVuIC9laWdodCAvbmluZSA2OCAvRCA4MCAvUCA5NyAvYSAvYiAvYyAvZCAvZSAvZiAxMDUgL2kgMTA4IC9sIDExMCAvbgovbyAxMTMgL3EgL3IgL3MgL3QgL3UgL3YgMTIxIC95IC96IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNCAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvQk1RUURWK0RlamFWdVNhbnMKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTMgMCBSID4+CmVuZG9iagoxNCAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zCi9JdGFsaWNBbmdsZSAwIC9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxMyAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNiAwIG9iago8PCAvRCAxNyAwIFIgL1AgMTggMCBSIC9hIDE5IDAgUiAvYiAyMCAwIFIgL2MgMjEgMCBSIC9kIDIyIDAgUiAvZSAyMyAwIFIKL2VpZ2h0IDI0IDAgUiAvZiAyNSAwIFIgL2ZpdmUgMjYgMCBSIC9mb3VyIDI3IDAgUiAvaHlwaGVuIDI4IDAgUiAvaSAyOSAwIFIKL2wgMzAgMCBSIC9uIDMxIDAgUiAvbmluZSAzMiAwIFIgL28gMzMgMCBSIC9vbmUgMzQgMCBSIC9wZXJpb2QgMzUgMCBSCi9xIDM2IDAgUiAvciAzNyAwIFIgL3MgMzggMCBSIC9zZXZlbiAzOSAwIFIgL3NpeCA0MCAwIFIgL3NwYWNlIDQxIDAgUgovdCA0MiAwIFIgL3RocmVlIDQzIDAgUiAvdHdvIDQ0IDAgUiAvdSA0NSAwIFIgL3YgNDYgMCBSIC95IDQ3IDAgUiAveiA0OCAwIFIKL3plcm8gNDkgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNSAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EzIDw8IC9DQSAwLjUgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMC41ID4+Ci9BNCA8PCAvQ0EgMC44IC9UeXBlIC9FeHRHU3RhdGUgL2NhIDAuOCA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCA+PgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKNTAgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxMCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCA1MQowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAyNDc0NCAwMDAwMCBuIAowMDAwMDI0NDY0IDAwMDAwIG4gCjAwMDAwMjQ0OTYgMDAwMDAgbiAKMDAwMDAyNDY4MSAwMDAwMCBuIAowMDAwMDI0NzAyIDAwMDAwIG4gCjAwMDAwMjQ3MjMgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQzIDAwMDAwIG4gCjAwMDAwMTM4NzAgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDEzODQ4IDAwMDAwIG4gCjAwMDAwMjMwMTUgMDAwMDAgbiAKMDAwMDAyMjgwOCAwMDAwMCBuIAowMDAwMDIyMzIwIDAwMDAwIG4gCjAwMDAwMjQwNjggMDAwMDAgbiAKMDAwMDAxMzg5MCAwMDAwMCBuIAowMDAwMDE0MTI3IDAwMDAwIG4gCjAwMDAwMTQzNzAgMDAwMDAgbiAKMDAwMDAxNDc1MCAwMDAwMCBuIAowMDAwMDE1MDY3IDAwMDAwIG4gCjAwMDAwMTUzNzIgMDAwMDAgbiAKMDAwMDAxNTY3NiAwMDAwMCBuIAowMDAwMDE1OTk4IDAwMDAwIG4gCjAwMDAwMTY0NjYgMDAwMDAgbiAKMDAwMDAxNjY3NSAwMDAwMCBuIAowMDAwMDE2OTk3IDAwMDAwIG4gCjAwMDAwMTcxNjMgMDAwMDAgbiAKMDAwMDAxNzI4OSAwMDAwMCBuIAowMDAwMDE3NDMzIDAwMDAwIG4gCjAwMDAwMTc1NTIgMDAwMDAgbiAKMDAwMDAxNzc4OCAwMDAwMCBuIAowMDAwMDE4MTgzIDAwMDAwIG4gCjAwMDAwMTg0NzQgMDAwMDAgbiAKMDAwMDAxODYyOSAwMDAwMCBuIAowMDAwMDE4NzUyIDAwMDAwIG4gCjAwMDAwMTkwNjggMDAwMDAgbiAKMDAwMDAxOTMwMSAwMDAwMCBuIAowMDAwMDE5NzA4IDAwMDAwIG4gCjAwMDAwMTk4NTAgMDAwMDAgbiAKMDAwMDAyMDI0MyAwMDAwMCBuIAowMDAwMDIwMzMzIDAwMDAwIG4gCjAwMDAwMjA1MzkgMDAwMDAgbiAKMDAwMDAyMDk1MiAwMDAwMCBuIAowMDAwMDIxMjc2IDAwMDAwIG4gCjAwMDAwMjE1MjMgMDAwMDAgbiAKMDAwMDAyMTY3MCAwMDAwMCBuIAowMDAwMDIxODg0IDAwMDAwIG4gCjAwMDAwMjIwMzIgMDAwMDAgbiAKMDAwMDAyNDgwNCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDUwIDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSA1MSA+PgpzdGFydHhyZWYKMjQ5NjEKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:10.029130\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["visualize_dequantization(quants=8, prior=np.array([0.075, 0.2, 0.4, 0.2, 0.075, 0.025, 0.0125, 0.0125]))"]}, {"cell_type": "markdown", "id": "717c85bd", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.016286, "end_time": "2022-05-12T12:19:10.334098", "exception": false, "start_time": "2022-05-12T12:19:10.317812", "status": "completed"}, "tags": []}, "source": ["Transforming such a probability into a Gaussian is a difficult task, especially with such hard borders.\n", "Dequantization has therefore been extended to more sophisticated, learnable distributions beyond uniform in a variational framework.\n", "In particular, if we remember the learning objective\n", "$\\log p(x) = \\log \\mathbb{E}_{u}\\left[\\frac{p(x+u)}{q(u|x)} \\right]$,\n", "the uniform distribution can be replaced by a learned distribution $q_{\\theta}(u|x)$ with support over $u\\in[0,1)^D$.\n", "This approach is called Variational Dequantization and has been proposed by Ho et al.\n", "[3].\n", "How can we learn such a distribution?\n", "We can use a second normalizing flow that takes $x$ as external input and learns a flexible distribution over $u$.\n", "To ensure a support over $[0,1)^D$, we can apply a sigmoid activation function as final flow transformation.\n", "\n", "Inheriting the original dequantization class, we can implement variational dequantization as follows:"]}, {"cell_type": "code", "execution_count": 11, "id": "ea583bfb", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:10.370526Z", "iopub.status.busy": "2022-05-12T12:19:10.370052Z", "iopub.status.idle": "2022-05-12T12:19:10.376986Z", "shell.execute_reply": "2022-05-12T12:19:10.376278Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.027603, "end_time": "2022-05-12T12:19:10.378528", "exception": false, "start_time": "2022-05-12T12:19:10.350925", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class VariationalDequantization(Dequantization):\n", " def __init__(self, var_flows, alpha=1e-5):\n", " \"\"\"\n", " Args:\n", " var_flows: A list of flow transformations to use for modeling q(u|x)\n", " alpha: Small constant, see Dequantization for details\n", " \"\"\"\n", " super().__init__(alpha=alpha)\n", " self.flows = nn.ModuleList(var_flows)\n", "\n", " def dequant(self, z, ldj):\n", " z = z.to(torch.float32)\n", " img = (z / 255.0) * 2 - 1 # We condition the flows on x, i.e. the original image\n", "\n", " # Prior of u is a uniform distribution as before\n", " # As most flow transformations are defined on [-infinity,+infinity], we apply an inverse sigmoid first.\n", " deq_noise = torch.rand_like(z).detach()\n", " deq_noise, ldj = self.sigmoid(deq_noise, ldj, reverse=True)\n", " for flow in self.flows:\n", " deq_noise, ldj = flow(deq_noise, ldj, reverse=False, orig_img=img)\n", " deq_noise, ldj = self.sigmoid(deq_noise, ldj, reverse=False)\n", "\n", " # After the flows, apply u as in standard dequantization\n", " z = (z + deq_noise) / 256.0\n", " ldj -= np.log(256.0) * np.prod(z.shape[1:])\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "97d1bdd1", "metadata": {"papermill": {"duration": 0.01692, "end_time": "2022-05-12T12:19:10.412762", "exception": false, "start_time": "2022-05-12T12:19:10.395842", "status": "completed"}, "tags": []}, "source": ["Variational dequantization can be used as a substitute for dequantization.\n", "We will compare dequantization and variational dequantization in later experiments."]}, {"cell_type": "markdown", "id": "3944ac74", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.017196, "end_time": "2022-05-12T12:19:10.446855", "exception": false, "start_time": "2022-05-12T12:19:10.429659", "status": "completed"}, "tags": []}, "source": ["### Coupling layers\n", "\n", "
\n", "\n", "Next, we look at possible transformations to apply inside the flow.\n", "A recent popular flow layer, which works well in combination with deep neural networks,\n", "is the coupling layer introduced by Dinh et al.\n", "[1].\n", "The input $z$ is arbitrarily split into two parts, $z_{1:j}$ and $z_{j+1:d}$, of which the first remains unchanged by the flow.\n", "Yet, $z_{1:j}$ is used to parameterize the transformation for the second part, $z_{j+1:d}$.\n", "Various transformations have been proposed in recent time [3,4], but here we will settle for the simplest and most efficient one: affine coupling.\n", "In this coupling layer, we apply an affine transformation by shifting the input by a bias $\\mu$ and scale it by $\\sigma$.\n", "In other words, our transformation looks as follows:\n", "\n", "$$z'_{j+1:d} = \\mu_{\\theta}(z_{1:j}) + \\sigma_{\\theta}(z_{1:j}) \\odot z_{j+1:d}$$\n", "\n", "The functions $\\mu$ and $\\sigma$ are implemented as a shared neural network,\n", "and the sum and multiplication are performed element-wise.\n", "The LDJ is thereby the sum of the logs of the scaling factors: $\\sum_i \\left[\\log \\sigma_{\\theta}(z_{1:j})\\right]_i$.\n", "Inverting the layer can as simply be done as subtracting the bias and dividing by the scale:\n", "\n", "$$z_{j+1:d} = \\left(z'_{j+1:d} - \\mu_{\\theta}(z_{1:j})\\right) / \\sigma_{\\theta}(z_{1:j})$$\n", "\n", "We can also visualize the coupling layer in form of a computation graph,\n", "where $z_1$ represents $z_{1:j}$, and $z_2$ represents $z_{j+1:d}$:\n", "\n", "
\n", "\n", "In our implementation, we will realize the splitting of variables as masking.\n", "The variables to be transformed, $z_{j+1:d}$, are masked when passing $z$ to the shared network to predict the transformation parameters.\n", "When applying the transformation, we mask the parameters for $z_{1:j}$\n", "so that we have an identity operation for those variables:"]}, {"cell_type": "code", "execution_count": 12, "id": "de9d1fc7", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:10.482629Z", "iopub.status.busy": "2022-05-12T12:19:10.481896Z", "iopub.status.idle": "2022-05-12T12:19:10.489980Z", "shell.execute_reply": "2022-05-12T12:19:10.489308Z"}, "papermill": {"duration": 0.027711, "end_time": "2022-05-12T12:19:10.491507", "exception": false, "start_time": "2022-05-12T12:19:10.463796", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class CouplingLayer(nn.Module):\n", " def __init__(self, network, mask, c_in):\n", " \"\"\"Coupling layer inside a normalizing flow.\n", "\n", " Args:\n", " network: A PyTorch nn.Module constituting the deep neural network for mu and sigma.\n", " Output shape should be twice the channel size as the input.\n", " mask: Binary mask (0 or 1) where 0 denotes that the element should be transformed,\n", " while 1 means the latent will be used as input to the NN.\n", " c_in: Number of input channels\n", " \"\"\"\n", " super().__init__()\n", " self.network = network\n", " self.scaling_factor = nn.Parameter(torch.zeros(c_in))\n", " # Register mask as buffer as it is a tensor which is not a parameter,\n", " # but should be part of the modules state.\n", " self.register_buffer(\"mask\", mask)\n", "\n", " def forward(self, z, ldj, reverse=False, orig_img=None):\n", " \"\"\"\n", " Args:\n", " z: Latent input to the flow\n", " ldj: The current ldj of the previous flows.\n", " The ldj of this layer will be added to this tensor.\n", " reverse: If True, we apply the inverse of the layer.\n", " orig_img (optional): Only needed in VarDeq. Allows external\n", " input to condition the flow on (e.g. original image)\n", " \"\"\"\n", " # Apply network to masked input\n", " z_in = z * self.mask\n", " if orig_img is None:\n", " nn_out = self.network(z_in)\n", " else:\n", " nn_out = self.network(torch.cat([z_in, orig_img], dim=1))\n", " s, t = nn_out.chunk(2, dim=1)\n", "\n", " # Stabilize scaling output\n", " s_fac = self.scaling_factor.exp().view(1, -1, 1, 1)\n", " s = torch.tanh(s / s_fac) * s_fac\n", "\n", " # Mask outputs (only transform the second part)\n", " s = s * (1 - self.mask)\n", " t = t * (1 - self.mask)\n", "\n", " # Affine transformation\n", " if not reverse:\n", " # Whether we first shift and then scale, or the other way round,\n", " # is a design choice, and usually does not have a big impact\n", " z = (z + t) * torch.exp(s)\n", " ldj += s.sum(dim=[1, 2, 3])\n", " else:\n", " z = (z * torch.exp(-s)) - t\n", " ldj -= s.sum(dim=[1, 2, 3])\n", "\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "723c8206", "metadata": {"papermill": {"duration": 0.01805, "end_time": "2022-05-12T12:19:10.583298", "exception": false, "start_time": "2022-05-12T12:19:10.565248", "status": "completed"}, "tags": []}, "source": ["For stabilization purposes, we apply a $\\tanh$ activation function on the scaling output.\n", "This prevents sudden large output values for the scaling that can destabilize training.\n", "To still allow scaling factors smaller or larger than -1 and 1 respectively,\n", "we have a learnable parameter per dimension, called `scaling_factor`.\n", "This scales the tanh to different limits.\n", "Below, we visualize the effect of the scaling factor on the output activation of the scaling terms:"]}, {"cell_type": "code", "execution_count": 13, "id": "c1e1275c", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:10.617526Z", "iopub.status.busy": "2022-05-12T12:19:10.617035Z", "iopub.status.idle": "2022-05-12T12:19:11.394850Z", "shell.execute_reply": "2022-05-12T12:19:11.394156Z"}, "papermill": {"duration": 0.797476, "end_time": "2022-05-12T12:19:11.397055", "exception": false, "start_time": "2022-05-12T12:19:10.599579", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovTWVkaWFCb3ggWyAwIDAgNzEwLjYxOTUzNjAzNTEgMjE1Ljk4NTYyNSBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJzNWsGSHLcNvc9X8GgfRBEkQIC52VGiKt9sqeJDKgfVRlKk0solKY5/Pw/o3Z3u0e56Z6ZnVqqaqu0nDkg8okHgDZ8+e/2/dxevf3n+Y/rri83T7dPFlw2l9/i8TSW9x+ePROk5Pm83BU+XG6WSOw1pHY8f5o+VJA+TXgV4WT7+Z7N5s3n6A8x8SSUP0s5axOyrBx6FRi9q6bOv4PliwOa+0ZtNq/hvU0lVc+PiM19uahmZ6qjaZ/CHOUyj5d6v8BsbCzSW/yndMgGWkysVqb2RWKLec6vp8+v0a/qYnv5Q3WFKP+HzHp8gc4duhh/VGo3FsrfoYiGbF5uf06drwyWTYKOubcfj8yt08wmbWNKTgv+CYek2mBuYGpmNYOzicvPjy/T075SI0ss3mwE/uo7u/pX08t+bf6bvJJfv07/Sy582f3sJ/0su5EbL7K+LS7fx5Nnr96/+8fuLVx+/PLl89/H3L+nZb+nnzc+x2pPQZoQlwNRyt7foCrR1zR28WaFKdS/eapZvlTf4kqkwDVoQN4NXYI6o5iawVqyz3ElduSGszAPtlN53uDmwnbr0fguv4b3UzAxrsKP6AO8X4XJC72uhXK3A2k6KvIHX8H7AnebWmpTxAO/lTHu/z/FwmOuaQWzL1Mto+rBM0b7VPHFjVLA//Wu2tvAxbEnJyK8PZqt+82xZzWpfs7WFj2FLLffycLbom2eLUD1K+ZquGX5wGuqZpzmKr+921mYH0HlTEDXLtd7i+BY/3vHGufCfOU5ndryjK7gl+c7w4x3Haa535OByWy45i+Pbk/Xyzu7jWMfNT9s/c7yt7XiW9AdaszqsNBRRolgjT00aagA89Os2jdMvz9OSoFmnM3ru16ExLA+VLg6qUK0MeqiAqyom1eExStMWcM8YQgbYPRgaZFKxLKMrmgHAKEJsqMPILc0J54RJUKNWnmDKozMxpeHJtBqHEWqZi7aCyrCAeuNxNZyzSWEsgFA+MYpdChi8jdYbOMNiiyiRBa5ZW7fCwBlLH9RH4CMSnTqOIR0P4VLFIGcf5mGym0gP+xWu4k9PnTgMQIEFMdg52GyEeKoeIAXcBT68EkUFlrzAr4bdifHNz11BJw28ZmSdiu11HJWrYW0cuLXCPewglVQUdc3xhtJe2gh2GkJv6JARuKo2CbfayM0YjCf3BPx1Cpwblt847LhJ0Bn0CCjENmrAjTE6lmk9GxuXaXjsWovmHeU6DS26xeUefPlaVY9bzH0lbTygSLxdFIDVW6vHe1/vPYvQvdNHgXMP7m+xedIU4SfqOsO4tjTLHXWZO15cvPrw7uPb9ObVxX9/+/y9px4so1jzf+m7v2CWRStzgOQziU57Sj9VsXhEqS37OkaVapULtQWxM3jB38zK1/KPR9DNgI7gpkPz5ea2wKumyG2tyE4obeFVhCC8dB15CG93oUeQgk5IYIMxpTJG3Xl3b+AVCGxI+01QYPThufn8mtApCexgahgO4yWBW3gNAmWgKIU11YKj5SBl6IQc4KjHwditt2Ue2cJrcIBMi5MT1tiPv4P0oVNywCgmlHXsKNJbeAUOGBUCijpYg7HDRKJTJuO9jpMDc7G0bCgm29Xyz68bnYfAuX40I/B4/WiHwEeQks5D4FxSmhF4vKS0Q+AjqEvnIXChMs1rv+NVJqwPHceNAwdqTWeiYa45zWk4XnPapeEw5elMNMwVqDkNxytQuzQcpkOdiYZ5Q3lXQ7QODYepUmvQsAltKh2sTc26r0i3zeUhlMRkWRs68ND7R5nUKe80RErnZPDff2x3oQh9MQ72GNqQbayOFijc8UwR8AA31fUXeOiiDltNxsjG3a05jK+aGIg2cVmjatxVYETsQG+TDIwUNPIjULzOGOODFS+2sAtHDccj5hPDYFe9iCbLUnNl7I4l7JJyvx7NWDQcG2kQeGfSMC09Yx+xBwiSWgcOkEANOR+VIRpKgWnlyZeOyCi1cOhs1IrKBFcXeHCWTMIZzFzhKAi0wnfgiMbW8GXH0bI2bdirULCwKLOAO6LDqutvLvWo+ESOY2tKNXYlrON4U+FYpBXsn7JfqKhoQdDBhbLVfF+DDNfEsMQ+mQf9OO/8d3/XxBCLtcZuuiJlES2EjZXKVYMwc0VGdMp94t+M4WBPWhnmyr64tNaucBe/QIKr9Ng7NpGY1vXL0kKXA27Sp1VyKb7T2hzX6EBDN+PiRvEW9MAVAVUDJ8yrPIQDH4id4vPiRcqlCcI/sj4h0EKvY3bfQRZPp4FHb8zLLpwy/l3hVkTvwfcQ2u4ose+SYGD51ur73ty1dxl/QH7cR3DzvDGJeWtIbfQYQhteEESoecabU4pXLsuOrvlhhi5Ym9m4XWa7GRCV6aonoOD1xDuoYxlEM3iFA1AqeBNj8vp8v5tD68hspyQQ+RhpB5l6SeAWXoNAVE6FexOkbeSw88tsJySwE5garfflbwAzeI2razgYFDm8s3gyO0hmOyUHqC9Qziyl7htwDf/ZSwhCP1sHPeQS0i0S2yn9x/nf+3Rrds7AFl6DA9SKI67m4si/W+y/V2M7ZSLZ5yQ5MI2gbJIbheMxJLbz8DeX2Lb8Ha+w7fD3CArbefibK2xb/o4X2Hb4ewSB7Tz8LQS2WdF3vL4mZWTbrv9Afe1MLMz1tRkLx8truywcJq+diYW5vDZj4Xh1bZeFw9S1M7Ew7x7vaIPWYeEwcW0NFo4V12Y9l7fXpXrzgE6CKwphQc5EGwDbZYQEgjo242/XMcQ1mE7kuo6I3/PBqsGMIr1aCyFC/GdmwtyALXNDYR2dpyuThrwE2+CttBZSlXTNKBQxBQrnLLWFnieKv9HRoH72UDMZdQTcEHfs33Rr2E4LGyDOLcN056wEHqbRaPVH6Fd+3QamNbzB4SKG+hwzWm44IkKfEZf34DD35JP7RlDA5rJNr5q05lG5xa0vGQUrQUmD0ZJ7LRZ3zQRBhqIftuLEqUST76O7kDYwxvzKF/LoNBpEID0VmlREv2oGuCNmMR9asIRZYG3ipHsCxkbXHrf1fE7fhY5Mj05NdLreRoKQCBwlF2hl0lDj2uiT2AfLuXMVv6jVKnzTaVK/luaqVQ35C3vbrAfuP1T1eLEYIabUNOCObanit8+4+W21OqbhLq+YuYrDfoMM0ex7gRmdanPpkeOCGo73wCUTPHHVTSjj67VO40F7UYQocGw6Qisk3e6F0PCamhC4IhZCZccQZDi/uQaC8BIhQiZcM7FWUIY3CsFAFjpt91DD+lxTQjCOQaRBcffsjvdnSnGEpcZNxI4gtVHU7XdsYAl5FjhCAueLN96AYWaiUhHSeMNdB0WYqt9hjB3B3iN4fRBhohYtgcNegWhsIOBRENThKjoSZBzyC4TI4IRFUr0H30MAvKP+v0scguXbWoP7suu+Hcb++Xsf8U/Ak3gquPpVZBUNcHm8bf4Pf4kqqgplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjI1ODQKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDEgPj4Kc3RyZWFtCnicNVI70ptBCOu/U+gCnlney3mcyaT4c/82AjsVLLBCAtICB5l4iSGqUa74JU8wXifwd708jZ/Hu5Ba8FSkH7g2beP9WLMmCpZGLIXZx74fJeR4avwbAj0XacKMTEYOJANxv9bnz3qTKYffgDRtTh8lSQ+iBbtbw44vCzJIelLDkp38sK4FVhehCXNjTSQjp1am5vnYM1zGE2MkqJoFJOkT96mCEWnGY+esJQ8yHE/14sWvt/Fa5jH1sqpAxjbBHGwnM+EURQTiF5QkN3EXTR3F0cxYc7vQUFLkvruHk5Ne95eTqMArIZzFWsIxQ09Z5mSnQQlUrZwAM6zXvjBO00YJd2q6vSv29fPMJIzbHHZWSqbBOQ7uZZM5gmSvOyZswuMQ8949gpGYN7+LLYIrlznXZPqxH0Ub6YPi+pyrKbMVJfxDlTyx4hr/n9/7+fP8/geMKH4jCmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoxOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzMiA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzMgPj4Kc3RyZWFtCnicMzY2VzBQMDQEkUZGBgqmQFaKIRdIwNDIRCGXCyQIYuWAWQZAGqI4B64mhysDzAZphagHsSDqjS2NoSoRLIhsBlcaAKfIF68KZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzNiA+PgpzdHJlYW0KeJxNj0EOAzEIA+95hZ9AIEB4z1ZVD9v/X0vYdtMLHsmAbFEGgSWHeIcb4dHbD99FNhVn45xfUiliIZhPcJ8wUxyNKXfyY4+AcZRqLKdoeF5Lzk3DFy13Ey2lrZeTGW+47pf3R5VtkQ1Fzy0LQtdskvkygQd8GJhHdeNppcfd9myv9vwAzmw0SQplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nE1RSYoDMAy75xX6QCFek7ynQ5lD5//Xyg6FOQQJr5KTlphYCw8xhB8sPfiRIXM3/Rt+otm7WXqSydn/mOciU1H4UqguYkJdiBvPoRHwPaFrElmxvfE5LKOZc74HH4W4BDOhAWN9STK5qOaVIRNODHUcDlqkwrhrYsPiWtE8jdxu+0ZmZSaEDY9kQtwYgIgg6wKyGCyUNjYTMlnOA+0NyQ1aYNepG1GLgiuU1gl0olbEqszgs+bWdjdDLfLgqH3x+mhWl2CF0Uv1WHhfhT6YqZl27pJCeuFNOyLMHgqkMjstK7V7xOpugfo/y1Lw/cn3+B2vD838XJwKZW5kc3RyZWFtCmVuZG9iagoyMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM0MSA+PgpzdHJlYW0KeJxFUktuRDEI279TcIFI4ZeQ87Squpjef1ubTNXN4AlgbHjLU6ZkyrC5JSMk15RPfSJDrKb8NHIkIqb4SQkFdpWPx2tLrI3skagUn9rx47H0RqbZFVr17tGlzaJRzcrIOcgQoZ4VurJ71A7Z8HpcSLrvlM0hHMv/UIEsZd1yCiVBW9B37BHfDx2ugiuCYbBrLoPtZTLU//qHFlzvffdixy6AFqznvsEOAKinE7QFyBna7jYpaABVuotJwqPyem52omyjVen5HAAzDjBywIglWx2+0d4Aln1d6EWNiv0rQFFZQPzI1XbB3jHJSHAW5gaOvXA8xZlwSzjGAkCKveIYevAl2OYvV66ImvAJdbpkL7zCntrm50KTCHetAA5eZMOtq6Oolu3pPIL2Z0VyRozUizg6IZJa0jmC4tKgHlrjXDex4m0jsblX3+4f4ZwvXPbrF0vshMQKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQ3ID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9CQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM5Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nOMyNDBTMDY1VcjlMjc2ArNywCwjcyMgCySLYEFkM7jSABXzCnwKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MyA+PgpzdHJlYW0KeJxFkDsSAyEMQ3tOoSP4IwM+z2YyKTb3b2PYbFLA01ggg7sTgtTagonogoe2Jd0F760EZ2P86TZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+UNw9V/1v2LdOZuJgcnKHQjN6lPc+TY7orq6yf6kx9ys134r7FVhaVlLywm3nbtmQAncUznaqz0/Hwo69gplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE4ID4+CnN0cmVhbQp4nD1QuY0EMQzLXYUaWMB67alnFotLpv/0SPn2ItEWRVIqNZmSKS91lCVZU946fJbEDnmG5W5kNiUqRS+TsCX30ArxfYnmFPfd1ZazQzSXaDl+CzMqqhsd00s2mnAqE7qg3MMz+g1tdANWhx6xWyDQpGDXtiByxw8YDMGZE4siDEpNBv+uco+fXosbPsPxQxSRkg7mNf9Y/fJzDa9TjyeRbm++4l6cqQ4DERySmrwjXVixLhIRaTVBTc/AWi2Au7de/hu0I7oMQPaJxHGaUo6hv2twpc8v5SdT2AplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODMgPj4Kc3RyZWFtCnicRYy7DcAwCER7pmAEfib2PlGUwt6/DRAlbrgn3T1cHQmZKW4zw0MGngwshl1xgfSWMAtcR1COneyjYdW+6gSN9aZS8+8PlJ7srOKG6wECQhpmCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA1MSA+PgpzdHJlYW0KeJwzNrRQMFAwNDAHkkaGQJaRiUKKIRdIAMTM5YIJ5oBZBkAaojgHriaHK4MrDQDhtA2YCmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjAgPj4Kc3RyZWFtCnicRZA5EgMxCARzvYInSFyC96zLtcH6/6kH1kei6QI0HLoWTcp6FGg+6bFGobrQa+gsSpJEwRaSHVCnY4g7KEhMSGOSSLYegyOaWLNdmJlUKrNS4bRpxcK/2VrVyESNcI38iekGVPxP6lyU8E2Dr5Ix+hhUvDuDjEn4XkXcWjHt/kQwsRn2CW9FJgWEibGp2b7PYIbM9wrXOMfzDUyCN+sKZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE4ID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzMyA+PgpzdHJlYW0KeJxFj0sOBCEIRPecoo7Axx/ncTLphXP/7YCdbhNjPYVUgbmCoT0uawOdFR8hGbbxt6mWjkVZPlR6UlYPyeCHrMbLIdygLPCCSSqGIVCLmBqRLWVut4DbNg2yspVTpY6wi6Mwj/a0bBUeX6JbInWSP4PEKi/c47odyKXWu96ii75/pAExCQplbmRzdHJlYW0KZW5kb2JqCjM0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzQwID4+CnN0cmVhbQp4nDVSOW4EMQzr/Qp9IIBu2+/ZIEiR/L8NqdkUA3F0UpQ7WlR2y4eFVLXsdPm0ldoSN+R3ZYXECcmrEu1ShkiovFYh1e+ZMq+3NWcEyFKlwuSk5HHJgj/DpacLx/m2sa/lyB2PHlgVI6FEwDLFxOgals7usGZbfpZpwI94hJwr1i3HWAVSG9047Yr3oXktsgaIvZmWigodVokWfkHxoEeNffYYVFgg0e0cSXCMiVCRgHaB2kgMOXssdlEf9DMoMRPo2htF3EGBJZKYOcW6dPTf+NCxoP7YjDe/OirpW1pZY9I+G+2Uxiwy6XpY9HTz1seDCzTvovzn1QwSNGWNksYHrdo5hqKZUVZ4t0OTDc0xxyHzDp7DGQlK+jwUv48lEx2UyN8ODaF/Xx6jjJw23gLmoj9tFQcO4rPDXrmBFUoXa5L3AalM6IHp/6/xtb7X1x8d7YDGCmVuZHN0cmVhbQplbmRvYmoKMzUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTEgPj4Kc3RyZWFtCnicLVFJcgNBCLvPK/SEZqffY5crh+T/1wjKBwYNi0B0WuKgjJ8gLFe85ZGraMPfMzGC3wWHfivXbVjkQFQgSWNQNaF28Xr0HthxmAnMk9awDGasD/yMKdzoxeExGWe312XUEOxdrz2ZQcmsXMQlExdM1WEjZw4/mTIutHM9NyDnRliXYZBuVhozEo40hUghhaqbpM4EQRKMrkaNNnIU+6Uvj3SGVY2oMexzLW1fz004a9DsWKzy5JQeXXEuJxcvrBz09TYDF1FprPJASMD9bg/1c7KT33hL584W0+N7zcnywlRgxZvXbkA21eLfvIjj+4yv5+f5/ANfYFuICmVuZHN0cmVhbQplbmRvYmoKMzYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTUgPj4Kc3RyZWFtCnicNVE5DgMhDOz3Ff5AJIwveE+iKM3+v82M0VYewVyGtJQhmfJSk6gh5VM+epkunLrc18xqNOeWtC1zgLi2vC+tksCJZoiDwWmYuAGaPAFD19GoUUMXHtDUpVMosNwEPoq3bg/dY7WBl7Yh54kgYigZLEHNqUUTFm3PJ6Q1v16LG96X7d3IU6XGlhiBBgFWOBzX6NfwlT1PJtF0FTLUqzXLGAkTRSI8+Y6m1RPrWjTSMhLUxhGsagO8O/0wTgAAE3HLAmSfSpSz5MRvsfSzBlf6/gGfR1SWCmVuZHN0cmVhbQplbmRvYmoKMTUgMCBvYmoKPDwgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucyAvQ2hhclByb2NzIDE2IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ2IC9wZXJpb2QgNDggL3plcm8gL29uZSAvdHdvIC90aHJlZSA1MyAvZml2ZSA1OCAvY29sb24gODMgL1MKOTcgL2EgOTkgL2MgMTAyIC9mIC9nIDEwNSAvaSAxMDggL2wgMTEwIC9uIC9vIDExNCAvciAxMTYgL3QgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDE0IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxMyAwIFIgPj4KZW5kb2JqCjE0IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvQk1RUURWK0RlamFWdVNhbnMKL0l0YWxpY0FuZ2xlIDAgL01heFdpZHRoIDEzNDIgL1N0ZW1WIDAgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9YSGVpZ2h0IDAgPj4KZW5kb2JqCjEzIDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE2IDAgb2JqCjw8IC9TIDE3IDAgUiAvYSAxOCAwIFIgL2MgMTkgMCBSIC9jb2xvbiAyMCAwIFIgL2YgMjEgMCBSIC9maXZlIDIyIDAgUgovZyAyMyAwIFIgL2kgMjQgMCBSIC9sIDI1IDAgUiAvbiAyNyAwIFIgL28gMjggMCBSIC9vbmUgMjkgMCBSCi9wZXJpb2QgMzAgMCBSIC9yIDMxIDAgUiAvc3BhY2UgMzIgMCBSIC90IDMzIDAgUiAvdGhyZWUgMzQgMCBSIC90d28gMzUgMCBSCi96ZXJvIDM2IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTUgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9GMS1EZWphVnVTYW5zLW1pbnVzIDI2IDAgUiA+PgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMzcgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxMSswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCAzOAowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxMDE5MSAwMDAwMCBuIAowMDAwMDA5OTY5IDAwMDAwIG4gCjAwMDAwMTAwMDEgMDAwMDAgbiAKMDAwMDAxMDEwMCAwMDAwMCBuIAowMDAwMDEwMTIxIDAwMDAwIG4gCjAwMDAwMTAxNDIgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQ4IDAwMDAwIG4gCjAwMDAwMDMwMjggMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAzMDA3IDAwMDAwIG4gCjAwMDAwMDg2NzcgMDAwMDAgbiAKMDAwMDAwODQ3MCAwMDAwMCBuIAowMDAwMDA4MDMxIDAwMDAwIG4gCjAwMDAwMDk3MzAgMDAwMDAgbiAKMDAwMDAwMzA0OCAwMDAwMCBuIAowMDAwMDAzNDYyIDAwMDAwIG4gCjAwMDAwMDM4NDIgMDAwMDAgbiAKMDAwMDAwNDE0NyAwMDAwMCBuIAowMDAwMDA0MjkyIDAwMDAwIG4gCjAwMDAwMDQ1MDEgMDAwMDAgbiAKMDAwMDAwNDgyMyAwMDAwMCBuIAowMDAwMDA1MjM3IDAwMDAwIG4gCjAwMDAwMDUzODEgMDAwMDAgbiAKMDAwMDAwNTUwMCAwMDAwMCBuIAowMDAwMDA1NjcyIDAwMDAwIG4gCjAwMDAwMDU5MDggMDAwMDAgbiAKMDAwMDAwNjE5OSAwMDAwMCBuIAowMDAwMDA2MzU0IDAwMDAwIG4gCjAwMDAwMDY0NzcgMDAwMDAgbiAKMDAwMDAwNjcxMCAwMDAwMCBuIAowMDAwMDA2ODAwIDAwMDAwIG4gCjAwMDAwMDcwMDYgMDAwMDAgbiAKMDAwMDAwNzQxOSAwMDAwMCBuIAowMDAwMDA3NzQzIDAwMDAwIG4gCjAwMDAwMTAyNTEgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAzNyAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMzggPj4Kc3RhcnR4cmVmCjEwNDA4CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:10.931761\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["with torch.no_grad():\n", " x = torch.arange(-5, 5, 0.01)\n", " scaling_factors = [0.5, 1, 2]\n", " sns.set()\n", " fig, ax = plt.subplots(1, 3, figsize=(12, 3))\n", " for i, scale in enumerate(scaling_factors):\n", " y = torch.tanh(x / scale) * scale\n", " ax[i].plot(x.numpy(), y.numpy())\n", " ax[i].set_title(\"Scaling factor: \" + str(scale))\n", " ax[i].set_ylim(-3, 3)\n", " plt.subplots_adjust(wspace=0.4)\n", " sns.reset_orig()\n", " plt.show()"]}, {"cell_type": "markdown", "id": "fbe411de", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.017149, "end_time": "2022-05-12T12:19:11.433168", "exception": false, "start_time": "2022-05-12T12:19:11.416019", "status": "completed"}, "tags": []}, "source": ["Coupling layers generalize to any masking technique we could think of.\n", "However, the most common approach for images is to split the input $z$ in half, using a checkerboard mask or channel mask.\n", "A checkerboard mask splits the variables across the height and width dimensions and assigns each other pixel to $z_{j+1:d}$.\n", "Thereby, the mask is shared across channels.\n", "In contrast, the channel mask assigns half of the channels to $z_{j+1:d}$, and the other half to $z_{1:j+1}$.\n", "Note that when we apply multiple coupling layers, we invert the masking for each other layer so that each variable is transformed a similar amount of times.\n", "\n", "Let's implement a function that creates a checkerboard mask and a channel mask for us:"]}, {"cell_type": "code", "execution_count": 14, "id": "d0e8846d", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:11.470735Z", "iopub.status.busy": "2022-05-12T12:19:11.470295Z", "iopub.status.idle": "2022-05-12T12:19:11.476167Z", "shell.execute_reply": "2022-05-12T12:19:11.475559Z"}, "papermill": {"duration": 0.026902, "end_time": "2022-05-12T12:19:11.477658", "exception": false, "start_time": "2022-05-12T12:19:11.450756", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def create_checkerboard_mask(h, w, invert=False):\n", " x, y = torch.arange(h, dtype=torch.int32), torch.arange(w, dtype=torch.int32)\n", " xx, yy = torch.meshgrid(x, y)\n", " mask = torch.fmod(xx + yy, 2)\n", " mask = mask.to(torch.float32).view(1, 1, h, w)\n", " if invert:\n", " mask = 1 - mask\n", " return mask\n", "\n", "\n", "def create_channel_mask(c_in, invert=False):\n", " mask = torch.cat([torch.ones(c_in // 2, dtype=torch.float32), torch.zeros(c_in - c_in // 2, dtype=torch.float32)])\n", " mask = mask.view(1, c_in, 1, 1)\n", " if invert:\n", " mask = 1 - mask\n", " return mask"]}, {"cell_type": "markdown", "id": "082a68c8", "metadata": {"papermill": {"duration": 0.017254, "end_time": "2022-05-12T12:19:11.512191", "exception": false, "start_time": "2022-05-12T12:19:11.494937", "status": "completed"}, "tags": []}, "source": ["We can also visualize the corresponding masks for an image of size $8\\times 8\\times 2$ (2 channels):"]}, {"cell_type": "code", "execution_count": 15, "id": "55168db2", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:11.548462Z", "iopub.status.busy": "2022-05-12T12:19:11.547815Z", "iopub.status.idle": "2022-05-12T12:19:11.669378Z", "shell.execute_reply": "2022-05-12T12:19:11.668705Z"}, "papermill": {"duration": 0.141383, "end_time": "2022-05-12T12:19:11.670995", "exception": false, "start_time": "2022-05-12T12:19:11.529612", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMTY2Ljg2IDExMi42ODUgXQovUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicVU/LTsMwELzvV8wxPeB47XqTHGkLEdwKkTggDiV106aEKolEf59tJB61NNKM7ZmdTVfx61DHp3KB5TOlf6oeidEqGli0ijMYpaIhq6ojFjG5KP34oczOSB70wv7je6Id9ciMm8DBmbkg94YFQ8QLPpHeauyo2a3irJElrpv0k03muIxWa3C/eXWH9IGxOmFNa/T63lwFqaZe17C40VZwXmNs4bOAQowX9dOiQnrPYIdqNy1XbekVyXIf6+MMPhgO4gKSOLyfNsMMLKZwmc395SDZotuM+vEN1SPdVaQ96BvN3kySCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMjMwCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjM1ID4+CnN0cmVhbQp4nDVRSW4AMQi75xX+QKWwJ++Zquqh/f+1hlEvAwPY2CTvwUYkPsSQ7ihXfMrqNMvwO1nkxc9K4eS9iAqkKsIKaQfPclYzDJ4bmQKXM/FZZj6ZFjsWUE3EcXbkNINBiGlcR8vpMNM86Am5PhhxY6dZrmJI691Svb7X8p8qykfW3Sy3TtnUSt2iZ+xJXHZeT21pXxh1FDcFkQ4fO7wH+SLmLC46kW72mymHlaQhOC2AH4mhVM8OrxEmfmYkeMqeTu+jNLz2QdP1vXtBR24mZCq3UEYqnqw0xoyh+o1oJqnv/4Ge9b2+/gBDTVS5CmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NCA+PgpzdHJlYW0KeJxFkU1yBSEIhPeeoi/wquRXPc+kUllM7r8NzbwkK1qF5gPTAhNH8BJD7ImVEx8yfC/oMny3MjvwOtmZcE+4blzDZcMzYVvgOyrLO15Dd7ZSP52hqu8aOd4uUjV0ZWSfeqGaC8yQiK4RWXQrl3VA05TuUuEabFuCFPVKrCedoDToEcrwd5RrfHUTT6+x5FTNIVrNrRMairBseEHUySQRtQ2LJ5ZzIVH5qhurOi5gkyXi9IDcoJVmfHpSSREwg3ysyWjMAjbQk7tnF8aaSx5Fjlc0mLA7STXwgPfitr73NnGP8xf4hXff/ysOfdcCPn8AS/5dBgplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMyID4+CnN0cmVhbQp4nDVRSW7EMAy7+xX8wADW7rwnxaCH9v/XUsoUCEAltrglYmMjAi8x+DmI3PiSNaMmfmdyV/wsT4VHwq3gSRSBl+FedoLLG8ZlPw4zH7yXVs6kxpMMyEU2PTwRMtglEDowuwZ12Gbaib4h4bMjUs1GltPXEvTSKgTKU7bf6YISbav6c/usC2372hNOdnvqSeUTiOeWrMBl4xWTxVgGPVG5SzF9kOpsoSehvCifg2w+aohElyhn4InBwSjQDuy57WfiVSFoXd2nbWOoRkrH078NTU2SCPlECWe2NO4W/n/Pvb7X+w9OIVQRCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzEgPj4Kc3RyZWFtCnicNU85kgQhDMt5hT4wVRjbQL+np7Y22Pl/upKZTpDwIcnTEx2ZeJkjI7Bmx9taZCBm4FNMxb/2tA8TqvfgHiKUiwthhpFw1qzjbp6OF/92lc9YB+82+IpZXhDYwkzWVxZnLtsFY2mcxDnJboxdE7GNda2nU1hHMKEMhHS2w5Qgc1Sk9MmOMuboOJEnnovv9tssdjl+DusLNo0hFef4KnqCNoOi7HnvAhpyQf9d3fgeRbvoJSAbCRbWUWLunOWEX712dB61KBJzQppBLhMhzekqphCaUKyzo6BSUXCpPqforJ9/5V9cLQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2NCA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgzID4+CnN0cmVhbQp4nD3MORKAMAgF0J5T/COEyCL3cRyLeP9WMNEGHqt6oCE4g7rBreFgyrp0E+9T49XGnBIJqHhKTZa6C3rUtL7Uvmjgu+vmS9WJP83PF50Pux0Z3QplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjU4ID4+CnN0cmVhbQp4nEWRS3IEIAhE956CI4D85DyTSmUxuf82Dc5kNnaXqP2ESiOmEiznFHkwfcnyzWS26Xc5VjsbBRRFKJjJVeixAqs7U8SZa4lq62Nl5LjTOwbFG85dOalkcaOMdVR1KnBMz5X1Ud35dlmUfUcOZQrYrHMcbODKbcMYJ0abre4O94kgTydTR8XtINnwByeNfZWrK3CdbPbRSzAOBP1CE5jki0DrDIHGzVP05BLs4+N254Fgb3kRSNkQyJEhGB2Cdp1c/+LW+b3/cYY7z7UZrhzv4neY1nbHX2KSFXMBi9wpqOdrLlrXGTrekzPH5Kb7hs65YJe7g0zv+T/Wz/r+Ax4pZvoKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxOCA+PgpzdHJlYW0KeJw9ULmNBDEMy12FGljAeu2pZxaLS6b/9Ej59iLRFkVSKjWZkikvdZQlWVPeOnyWxA55huVuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rnKPn16LGz7D8UMUkZIO5jX/WP3ycw2vU48nkW5vvuJenKkOAxEckpq8I11YsS4SEWk1QU3PwFotgLu3Xv4btCO6DED2icRxmlKOob9rcKXPL+UnU9gKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM0ID4+CnN0cmVhbQp4nC1SS3LFIAzbcwpdoDP4B+Q86XS6eL3/tpKTRUYOYPQx5YaJSnxZILej1sS3jcxAheGvq8yFz0jbyDqIy5CLuJIthXtELOQxxDzEgu+r8R4e+azMybMHxi/Zdw8r9tSEZSHjxRnaYRXHYRXkWLB1Iap7eFOkw6kk2OOL/z7Fcy0ELXxG0IBf5J+vjuD5khZp95ht0656sEw7qqSwHGxPc14mX1pnuToezwfJ9q7YEVK7AhSFuTPOc+Eo01ZGtBZ2NkhqXGxvjv1YStCFblxGiiOQn6kiPKCkycwmCuKPnB5yKgNh6pqudHIbVXGnnsw1m4u3M0lm675IsZnCeV04s/4MU2a1eSfPcqLUqQjvsWdL0NA5rp69lllodJsTvKSEz8ZOT06+VzPrITkVCaliWlfBaRSZYgnbEl9TUVOaehn++/Lu8Tt+/gEsc3xzCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxOCA+PgpzdHJlYW0KeJwzNrRQMIDDFEOuNAAd5gNSCmVuZHN0cmVhbQplbmRvYmoKMTYgMCBvYmoKPDwgL0Jhc2VGb250IC9CTVFRRFYrRGVqYVZ1U2FucyAvQ2hhclByb2NzIDE3IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDY3IC9DIDk3IC9hIC9iIC9jIC9kIC9lIDEwNCAvaCAxMDcgL2sgMTA5IC9tIDExMSAvbyAxMTQgL3IgL3MKXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDE1IDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9CTVFRRFYrRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxNCAwIFIgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvQk1RUURWK0RlamFWdVNhbnMKL0l0YWxpY0FuZ2xlIDAgL01heFdpZHRoIDEzNDIgL1N0ZW1WIDAgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9YSGVpZ2h0IDAgPj4KZW5kb2JqCjE0IDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE3IDAgb2JqCjw8IC9DIDE4IDAgUiAvYSAxOSAwIFIgL2IgMjAgMCBSIC9jIDIxIDAgUiAvZCAyMiAwIFIgL2UgMjMgMCBSIC9oIDI0IDAgUgovayAyNSAwIFIgL20gMjYgMCBSIC9vIDI3IDAgUiAvciAyOCAwIFIgL3MgMjkgMCBSIC9zcGFjZSAzMCAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE2IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL0NBIDEgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvQml0c1BlckNvbXBvbmVudCA4IC9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP///39/fwAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgMjEyIC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCAxMTYgL0xlbmd0aCAzMSAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCAyMTIgPj4Kc3RyZWFtCnic7dAxTkAxDETBD/c/NJ1pXCxSENnPvNKS5Uyejxf2/PUDfiOolqBagmoJqiWolqBagmoJqqV/g/qcnumHo+/ZdjTbzEbbASgoKCgoKKgLUCdOzWhDZZvh90FBQUFBQUHdiTpxakYbKtsMvw8KCgoKCuoFqBOnZrShss3w+6CgoKCgoKDuRJ04NaMNlW2G3wcFBQUFBQV1J+rEqRltqGwz/D4oKCgoKKgXoE6cmtGGyjbD74OCgoKCgoK6E3Xi1Iw2VLYZfh8UFBQUFBTUnaj6oFqCagmqJaiWoFqCagmqJaiWoFr6AjRXYBEKZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iagoyMzUKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjMyIDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMjA1MTIxNDE5MTErMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My41LjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My41LjIpID4+CmVuZG9iagp4cmVmCjAgMzMKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDY4MzIgMDAwMDAgbiAKMDAwMDAwNjE1NyAwMDAwMCBuIAowMDAwMDA2MTg5IDAwMDAwIG4gCjAwMDAwMDYyNDkgMDAwMDAgbiAKMDAwMDAwNjI3MCAwMDAwMCBuIAowMDAwMDA2MjkxIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDMzNyAwMDAwMCBuIAowMDAwMDAwNjYyIDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDY0MiAwMDAwMCBuIAowMDAwMDA2MzIzIDAwMDAwIG4gCjAwMDAwMDQ5NDggMDAwMDAgbiAKMDAwMDAwNDc0MSAwMDAwMCBuIAowMDAwMDA0MzYyIDAwMDAwIG4gCjAwMDAwMDYwMDEgMDAwMDAgbiAKMDAwMDAwMDY4MiAwMDAwMCBuIAowMDAwMDAwOTkwIDAwMDAwIG4gCjAwMDAwMDEzNzAgMDAwMDAgbiAKMDAwMDAwMTY4NyAwMDAwMCBuIAowMDAwMDAxOTkyIDAwMDAwIG4gCjAwMDAwMDIyOTYgMDAwMDAgbiAKMDAwMDAwMjYxOCAwMDAwMCBuIAowMDAwMDAyODU1IDAwMDAwIG4gCjAwMDAwMDMwMTAgMDAwMDAgbiAKMDAwMDAwMzM0MSAwMDAwMCBuIAowMDAwMDAzNjMyIDAwMDAwIG4gCjAwMDAwMDM4NjUgMDAwMDAgbiAKMDAwMDAwNDI3MiAwMDAwMCBuIAowMDAwMDA2ODEyIDAwMDAwIG4gCjAwMDAwMDY4OTIgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAzMiAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMzMgPj4Kc3RhcnR4cmVmCjcwNDkKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:11.575759\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMTY2Ljg2IDExMi42ODUgXQovUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicVY87b8JAEIT7/RVThoK728O3mJKnFTqSk1JEKZAxBoONjCX4+1ks5UEx0syd5ttduyhux7x4y2aYv5P9S3lHjEpVwqFS3cHIVCU5TTWxiElF7fnHMnsjadAH988fiPbUYmx8Lw7eJIJ0ZFhwLfCBBnaq2E7ZlequyAzPm7R9TRI8Rms1+F9eXsO+MhYXbGiDVv/LJ5BmavUMh6FuhcSZcUjZB0zEjETrNIuwKwZ7xH1/W9zRJ17mh23TFGfU2+40wBfimpaRdAh9A+8lRP4KZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoyMDgKZW5kb2JqCjEwIDAgb2JqClsgXQplbmRvYmoKMTggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzUgPj4Kc3RyZWFtCnicNVFJbgAxCLvnFf5ApbAn75mq6qH9/7WGUS8DA9jYJO/BRiQ+xJDuKFd8yuo0y/A7WeTFz0rh5L2ICqQqwgppB89yVjMMnhuZApcz8VlmPpkWOxZQTcRxduQ0g0GIaVxHy+kw0zzoCbk+GHFjp1muYkjr3VK9vtfynyrKR9bdLLdO2dRK3aJn7Elcdl5PbWlfGHUUNwWRDh87vAf5IuYsLjqRbvabKYeVpCE4LYAfiaFUzw6vESZ+ZiR4yp5O76M0vPZB0/W9e0FHbiZkKrdQRiqerDTGjKH6jWgmqe//gZ71vb7+AENNVLkKZW5kc3RyZWFtCmVuZG9iagoxOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMwNyA+PgpzdHJlYW0KeJw9kktuAzEMQ/c+hS4QwPrZnvOkKLqY3n/bJyXpihzZFkVqlrpMWVMekDSThH/p8HCxnfI7bM9mZuBaopeJ5ZTn0BVi7qJ82cxGXVknxeqEZjq36FE5Fwc2Taqfqyyl3S54Dtcmnlv2ET+80KAe1DUuCTd0V6NlKTRjqvt/0nv8jDLgakxdbFKrex88XkRV6OgHR4kiY5cX5+NBCelKwmhaiJV3RQNB7vK0ynsJ7tveasiyB6mYzjspZrDrdFIubheHIR7I8qjw5aPYa0LP+LArJfRI2IYzcifuaMbm1MjikP7ejQRLj65oIfPgr27WLmC8UzpFYmROcqxpi1VO91AU07nDvQwQ9WxFQylzkdXqX8POC2uWbBZ4SvoFHqPdJksOVtnbqE7vrTzZ0PcfWtd0HwplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2NCA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgzID4+CnN0cmVhbQp4nD3MORKAMAgF0J5T/COEyCL3cRyLeP9WMNEGHqt6oCE4g7rBreFgyrp0E+9T49XGnBIJqHhKTZa6C3rUtL7Uvmjgu+vmS9WJP83PF50Pux0Z3QplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDcgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZclhBWLhdMLAfMAtGWcAoinsGVBgC5Zw0nCmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTggPj4Kc3RyZWFtCnicRZFLcgQgCET3noIjgPzkPJNKZTG5/zYNzmQ2dpeo/YRKI6YSLOcUeTB9yfLNZLbpdzlWOxsFFEUomMlV6LECqztTxJlriWrrY2XkuNM7BsUbzl05qWRxo4x1VHUqcEzPlfVR3fl2WZR9Rw5lCtiscxxs4MptwxgnRput7g73iSBPJ1NHxe0g2fAHJ419lasrcJ1s9tFLMA4E/UITmOSLQOsMgcbNU/TkEuzj43bngWBveRFI2RDIkSEYHYJ2nVz/4tb5vf9xhjvPtRmuHO/id5jWdsdfYpIVcwGL3Cmo52suWtcZOt6TM8fkpvuGzrlgl7uDTO/5P9bP+v4DHilm+gplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYzID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzQgPj4Kc3RyZWFtCnicLVJLcsUgDNtzCl2gM/gH5DzpdLp4vf+2kpNFRg5g9DHlholKfFkgt6PWxLeNzECF4a+rzIXPSNvIOojLkIu4ki2Fe0Qs5DHEPMSC76vxHh75rMzJswfGL9l3Dyv21IRlIePFGdphFcdhFeRYsHUhqnt4U6TDqSTY44v/PsVzLQQtfEbQgF/kn6+O4PmSFmn3mG3TrnqwTDuqpLAcbE9zXiZfWme5Oh7PB8n2rtgRUrsCFIW5M85z4SjTVka0FnY2SGpcbG+O/VhK0IVuXEaKI5CfqSI8oKTJzCYK4o+cHnIqA2Hqmq50chtVcaeezDWbi7czSWbrvkixmcJ5XTiz/gxTZrV5J89yotSpCO+xZ0vQ0Dmunr2WWWh0mxO8pITPxk5PTr5XM+shORUJqWJaV8FpFJliCdsSX1NRU5p6Gf778u7xO37+ASxzfHMKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE4ID4+CnN0cmVhbQp4nDM2tFAwgMMUQ640AB3mA1IKZW5kc3RyZWFtCmVuZG9iagoxNiAwIG9iago8PCAvQmFzZUZvbnQgL0JNUVFEVitEZWphVnVTYW5zIC9DaGFyUHJvY3MgMTcgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNjcgL0MgOTcgL2EgMTAxIC9lIDEwNCAvaCAxMDcgL2sgL2wgL20gL24gMTE1IC9zIF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNSAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvQk1RUURWK0RlamFWdVNhbnMKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTQgMCBSID4+CmVuZG9iagoxNSAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE5hbWUgL0JNUVFEVitEZWphVnVTYW5zCi9JdGFsaWNBbmdsZSAwIC9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxNCAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNyAwIG9iago8PCAvQyAxOCAwIFIgL2EgMTkgMCBSIC9lIDIwIDAgUiAvaCAyMSAwIFIgL2sgMjIgMCBSIC9sIDIzIDAgUiAvbSAyNCAwIFIKL24gMjUgMCBSIC9zIDI2IDAgUiAvc3BhY2UgMjcgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNiAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOCAvQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyICj///9/f38AAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDIxMiAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTE2IC9MZW5ndGggMjggMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMjEyID4+CnN0cmVhbQp4nO3PwRGAMBDDwIP+i6YBnmEYJasG7J1rw+bvA18EVQmqElQlqEpQlaAqQVWCqnQMalb2NnqvDAoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKqovKB1UJqhJUJahKUJWgKkFVgqoEVekBUBpgEQplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjE0NgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMjkgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxMSswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCAzMAowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNTU4MyAwMDAwMCBuIAowMDAwMDA0OTk3IDAwMDAwIG4gCjAwMDAwMDUwMjkgMDAwMDAgbiAKMDAwMDAwNTA4OSAwMDAwMCBuIAowMDAwMDA1MTEwIDAwMDAwIG4gCjAwMDAwMDUxMzEgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzM3IDAwMDAwIG4gCjAwMDAwMDA2NDAgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNjIwIDAwMDAwIG4gCjAwMDAwMDUxNjMgMDAwMDAgbiAKMDAwMDAwMzgxOCAwMDAwMCBuIAowMDAwMDAzNjExIDAwMDAwIG4gCjAwMDAwMDMyNDUgMDAwMDAgbiAKMDAwMDAwNDg3MSAwMDAwMCBuIAowMDAwMDAwNjYwIDAwMDAwIG4gCjAwMDAwMDA5NjggMDAwMDAgbiAKMDAwMDAwMTM0OCAwMDAwMCBuIAowMDAwMDAxNjcwIDAwMDAwIG4gCjAwMDAwMDE5MDcgMDAwMDAgbiAKMDAwMDAwMjA2MiAwMDAwMCBuIAowMDAwMDAyMTgxIDAwMDAwIG4gCjAwMDAwMDI1MTIgMDAwMDAgbiAKMDAwMDAwMjc0OCAwMDAwMCBuIAowMDAwMDAzMTU1IDAwMDAwIG4gCjAwMDAwMDU1NjMgMDAwMDAgbiAKMDAwMDAwNTY0MyAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDI5IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAzMCA+PgpzdGFydHhyZWYKNTgwMAolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:11.634536\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["checkerboard_mask = create_checkerboard_mask(h=8, w=8).expand(-1, 2, -1, -1)\n", "channel_mask = create_channel_mask(c_in=2).expand(-1, -1, 8, 8)\n", "\n", "show_imgs(checkerboard_mask.transpose(0, 1), \"Checkerboard mask\")\n", "show_imgs(channel_mask.transpose(0, 1), \"Channel mask\")"]}, {"cell_type": "markdown", "id": "01da453b", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.018032, "end_time": "2022-05-12T12:19:11.709109", "exception": false, "start_time": "2022-05-12T12:19:11.691077", "status": "completed"}, "tags": []}, "source": ["As a last aspect of coupling layers, we need to decide for the deep neural network we want to apply in the coupling layers.\n", "The input to the layers is an image, and hence we stick with a CNN.\n", "Because the input to a transformation depends on all transformations before,\n", "it is crucial to ensure a good gradient flow through the CNN back to the input,\n", "which can be optimally achieved by a ResNet-like architecture.\n", "Specifically, we use a Gated ResNet that adds a $\\sigma$-gate to the skip connection,\n", "similarly to the input gate in LSTMs.\n", "The details are not necessarily important here, and the network is\n", "strongly inspired from Flow++ [3] in case you are interested in building\n", "even stronger models."]}, {"cell_type": "code", "execution_count": 16, "id": "9c427445", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:11.748218Z", "iopub.status.busy": "2022-05-12T12:19:11.747617Z", "iopub.status.idle": "2022-05-12T12:19:11.757827Z", "shell.execute_reply": "2022-05-12T12:19:11.757184Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.031488, "end_time": "2022-05-12T12:19:11.759391", "exception": false, "start_time": "2022-05-12T12:19:11.727903", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ConcatELU(nn.Module):\n", " \"\"\"Activation function that applies ELU in both direction (inverted and plain).\n", "\n", " Allows non-linearity while providing strong gradients for any input (important for final convolution)\n", " \"\"\"\n", "\n", " def forward(self, x):\n", " return torch.cat([F.elu(x), F.elu(-x)], dim=1)\n", "\n", "\n", "class LayerNormChannels(nn.Module):\n", " def __init__(self, c_in):\n", " \"\"\"This module applies layer norm across channels in an image.\n", "\n", " Has been shown to work well with ResNet connections.\n", " Args:\n", " c_in: Number of channels of the input\n", " \"\"\"\n", " super().__init__()\n", " self.layer_norm = nn.LayerNorm(c_in)\n", "\n", " def forward(self, x):\n", " x = x.permute(0, 2, 3, 1)\n", " x = self.layer_norm(x)\n", " x = x.permute(0, 3, 1, 2)\n", " return x\n", "\n", "\n", "class GatedConv(nn.Module):\n", " def __init__(self, c_in, c_hidden):\n", " \"\"\"\n", " This module applies a two-layer convolutional ResNet block with input gate\n", " Args:\n", " c_in: Number of channels of the input\n", " c_hidden: Number of hidden dimensions we want to model (usually similar to c_in)\n", " \"\"\"\n", " super().__init__()\n", " self.net = nn.Sequential(\n", " nn.Conv2d(c_in, c_hidden, kernel_size=3, padding=1),\n", " ConcatELU(),\n", " nn.Conv2d(2 * c_hidden, 2 * c_in, kernel_size=1),\n", " )\n", "\n", " def forward(self, x):\n", " out = self.net(x)\n", " val, gate = out.chunk(2, dim=1)\n", " return x + val * torch.sigmoid(gate)\n", "\n", "\n", "class GatedConvNet(nn.Module):\n", " def __init__(self, c_in, c_hidden=32, c_out=-1, num_layers=3):\n", " \"\"\"Module that summarizes the previous blocks to a full convolutional neural network.\n", "\n", " Args:\n", " c_in: Number of input channels\n", " c_hidden: Number of hidden dimensions to use within the network\n", " c_out: Number of output channels. If -1, 2 times the input channels are used (affine coupling)\n", " num_layers: Number of gated ResNet blocks to apply\n", " \"\"\"\n", " super().__init__()\n", " c_out = c_out if c_out > 0 else 2 * c_in\n", " layers = []\n", " layers += [nn.Conv2d(c_in, c_hidden, kernel_size=3, padding=1)]\n", " for layer_index in range(num_layers):\n", " layers += [GatedConv(c_hidden, c_hidden), LayerNormChannels(c_hidden)]\n", " layers += [ConcatELU(), nn.Conv2d(2 * c_hidden, c_out, kernel_size=3, padding=1)]\n", " self.nn = nn.Sequential(*layers)\n", "\n", " self.nn[-1].weight.data.zero_()\n", " self.nn[-1].bias.data.zero_()\n", "\n", " def forward(self, x):\n", " return self.nn(x)"]}, {"cell_type": "markdown", "id": "cbaa014d", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.018089, "end_time": "2022-05-12T12:19:11.797248", "exception": false, "start_time": "2022-05-12T12:19:11.779159", "status": "completed"}, "tags": []}, "source": ["### Training loop\n", "\n", "Finally, we can add Dequantization, Variational Dequantization and Coupling Layers together to build our full normalizing flow on MNIST images.\n", "We apply 8 coupling layers in the main flow, and 4 for variational dequantization if applied.\n", "We apply a checkerboard mask throughout the network as with a single channel (black-white images),\n", "we cannot apply channel mask.\n", "The overall architecture is visualized below.\n", "\n", "\n", "
"]}, {"cell_type": "code", "execution_count": 17, "id": "4baabfa9", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:11.839983Z", "iopub.status.busy": "2022-05-12T12:19:11.839549Z", "iopub.status.idle": "2022-05-12T12:19:11.846018Z", "shell.execute_reply": "2022-05-12T12:19:11.845330Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.028068, "end_time": "2022-05-12T12:19:11.847757", "exception": false, "start_time": "2022-05-12T12:19:11.819689", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def create_simple_flow(use_vardeq=True):\n", " flow_layers = []\n", " if use_vardeq:\n", " vardeq_layers = [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=2, c_out=2, c_hidden=16),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " for i in range(4)\n", " ]\n", " flow_layers += [VariationalDequantization(var_flows=vardeq_layers)]\n", " else:\n", " flow_layers += [Dequantization()]\n", "\n", " for i in range(8):\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=1, c_hidden=32),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " ]\n", "\n", " flow_model = ImageFlow(flow_layers).to(device)\n", " return flow_model"]}, {"cell_type": "markdown", "id": "6355c7e0", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.019906, "end_time": "2022-05-12T12:19:11.889126", "exception": false, "start_time": "2022-05-12T12:19:11.869220", "status": "completed"}, "tags": []}, "source": ["For implementing the training loop, we use the framework of PyTorch Lightning and reduce the code overhead.\n", "If interested, you can take a look at the generated tensorboard file,\n", "in particularly the graph to see an overview of flow transformations that are applied.\n", "Note that we again provide pre-trained models (see later on in the notebook)\n", "as normalizing flows are particularly expensive to train.\n", "We have also run validation and testing as this can take some time as well with the added importance sampling."]}, {"cell_type": "code", "execution_count": 18, "id": "304541b2", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:11.928154Z", "iopub.status.busy": "2022-05-12T12:19:11.927713Z", "iopub.status.idle": "2022-05-12T12:19:11.935492Z", "shell.execute_reply": "2022-05-12T12:19:11.934876Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.029315, "end_time": "2022-05-12T12:19:11.936963", "exception": false, "start_time": "2022-05-12T12:19:11.907648", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def train_flow(flow, model_name=\"MNISTFlow\"):\n", " # Create a PyTorch Lightning trainer\n", " trainer = pl.Trainer(\n", " default_root_dir=os.path.join(CHECKPOINT_PATH, model_name),\n", " gpus=1 if torch.cuda.is_available() else 0,\n", " max_epochs=200,\n", " gradient_clip_val=1.0,\n", " callbacks=[\n", " ModelCheckpoint(save_weights_only=True, mode=\"min\", monitor=\"val_bpd\"),\n", " LearningRateMonitor(\"epoch\"),\n", " ],\n", " )\n", " trainer.logger._log_graph = True\n", " trainer.logger._default_hp_metric = None # Optional logging argument that we don't need\n", "\n", " train_data_loader = data.DataLoader(\n", " train_set, batch_size=128, shuffle=True, drop_last=True, pin_memory=True, num_workers=8\n", " )\n", " result = None\n", "\n", " # Check whether pretrained model exists. If yes, load it and skip training\n", " pretrained_filename = os.path.join(CHECKPOINT_PATH, model_name + \".ckpt\")\n", " if os.path.isfile(pretrained_filename):\n", " print(\"Found pretrained model, loading...\")\n", " ckpt = torch.load(pretrained_filename, map_location=device)\n", " flow.load_state_dict(ckpt[\"state_dict\"])\n", " result = ckpt.get(\"result\", None)\n", " else:\n", " print(\"Start training\", model_name)\n", " trainer.fit(flow, train_data_loader, val_loader)\n", "\n", " # Test best model on validation and test set if no result has been found\n", " # Testing can be expensive due to the importance sampling.\n", " if result is None:\n", " val_result = trainer.test(flow, dataloaders=val_loader, verbose=False)\n", " start_time = time.time()\n", " test_result = trainer.test(flow, dataloaders=test_loader, verbose=False)\n", " duration = time.time() - start_time\n", " result = {\"test\": test_result, \"val\": val_result, \"time\": duration / len(test_loader) / flow.import_samples}\n", "\n", " return flow, result"]}, {"cell_type": "markdown", "id": "8c41b4f7", "metadata": {"papermill": {"duration": 0.01931, "end_time": "2022-05-12T12:19:11.975188", "exception": false, "start_time": "2022-05-12T12:19:11.955878", "status": "completed"}, "tags": []}, "source": ["## Multi-scale architecture\n", "\n", "
\n", "\n", "One disadvantage of normalizing flows is that they operate on the exact same dimensions as the input.\n", "If the input is high-dimensional, so is the latent space, which requires larger computational cost to learn suitable transformations.\n", "However, particularly in the image domain, many pixels contain less information in the sense\n", "that we could remove them without loosing the semantical information of the image.\n", "\n", "Based on this intuition, deep normalizing flows on images commonly apply a multi-scale architecture [1].\n", "After the first $N$ flow transformations, we split off half of the latent dimensions and directly evaluate them on the prior.\n", "The other half is run through $N$ more flow transformations, and depending on the size of the input,\n", "we split it again in half or stop overall at this position.\n", "The two operations involved in this setup is `Squeeze` and `Split` which\n", "we will review more closely and implement below."]}, {"cell_type": "markdown", "id": "bf4c83d8", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.018063, "end_time": "2022-05-12T12:19:12.011683", "exception": false, "start_time": "2022-05-12T12:19:11.993620", "status": "completed"}, "tags": []}, "source": ["### Squeeze and Split\n", "\n", "When we want to remove half of the pixels in an image, we have the problem of deciding which variables to cut,\n", "and how to rearrange the image.\n", "Thus, the squeezing operation is commonly used before split, which divides the image into subsquares\n", "of shape $2\\times 2\\times C$, and reshapes them into $1\\times 1\\times 4C$ blocks.\n", "Effectively, we reduce the height and width of the image by a factor of 2 while scaling the number of channels by 4.\n", "Afterwards, we can perform the split operation over channels without the need of rearranging the pixels.\n", "The smaller scale also makes the overall architecture more efficient.\n", "Visually, the squeeze operation should transform the input as follows:\n", "\n", "
\n", "\n", "The input of $4\\times 4\\times 1$ is scaled to $2\\times 2\\times 4$ following\n", "the idea of grouping the pixels in $2\\times 2\\times 1$ subsquares.\n", "Next, let's try to implement this layer:"]}, {"cell_type": "code", "execution_count": 19, "id": "21bd22ac", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:12.050335Z", "iopub.status.busy": "2022-05-12T12:19:12.049864Z", "iopub.status.idle": "2022-05-12T12:19:12.055719Z", "shell.execute_reply": "2022-05-12T12:19:12.055088Z"}, "papermill": {"duration": 0.027489, "end_time": "2022-05-12T12:19:12.057265", "exception": false, "start_time": "2022-05-12T12:19:12.029776", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class SqueezeFlow(nn.Module):\n", " def forward(self, z, ldj, reverse=False):\n", " B, C, H, W = z.shape\n", " if not reverse:\n", " # Forward direction: H x W x C => H/2 x W/2 x 4C\n", " z = z.reshape(B, C, H // 2, 2, W // 2, 2)\n", " z = z.permute(0, 1, 3, 5, 2, 4)\n", " z = z.reshape(B, 4 * C, H // 2, W // 2)\n", " else:\n", " # Reverse direction: H/2 x W/2 x 4C => H x W x C\n", " z = z.reshape(B, C // 4, 2, 2, H, W)\n", " z = z.permute(0, 1, 4, 2, 5, 3)\n", " z = z.reshape(B, C // 4, H * 2, W * 2)\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "af45308a", "metadata": {"papermill": {"duration": 0.018839, "end_time": "2022-05-12T12:19:12.095432", "exception": false, "start_time": "2022-05-12T12:19:12.076593", "status": "completed"}, "tags": []}, "source": ["Before moving on, we can verify our implementation by comparing our output with the example figure above:"]}, {"cell_type": "code", "execution_count": 20, "id": "35170bd0", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:12.135426Z", "iopub.status.busy": "2022-05-12T12:19:12.134980Z", "iopub.status.idle": "2022-05-12T12:19:12.141717Z", "shell.execute_reply": "2022-05-12T12:19:12.140991Z"}, "papermill": {"duration": 0.028309, "end_time": "2022-05-12T12:19:12.143258", "exception": false, "start_time": "2022-05-12T12:19:12.114949", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Image (before)\n", " tensor([[[[ 1, 2, 3, 4],\n", " [ 5, 6, 7, 8],\n", " [ 9, 10, 11, 12],\n", " [13, 14, 15, 16]]]])\n", "\n", "Image (forward)\n", " tensor([[[[ 1, 2, 5, 6],\n", " [ 3, 4, 7, 8]],\n", "\n", " [[ 9, 10, 13, 14],\n", " [11, 12, 15, 16]]]])\n", "\n", "Image (reverse)\n", " tensor([[[[ 1, 2, 3, 4],\n", " [ 5, 6, 7, 8],\n", " [ 9, 10, 11, 12],\n", " [13, 14, 15, 16]]]])\n"]}], "source": ["sq_flow = SqueezeFlow()\n", "rand_img = torch.arange(1, 17).view(1, 1, 4, 4)\n", "print(\"Image (before)\\n\", rand_img)\n", "forward_img, _ = sq_flow(rand_img, ldj=None, reverse=False)\n", "print(\"\\nImage (forward)\\n\", forward_img.permute(0, 2, 3, 1)) # Permute for readability\n", "reconst_img, _ = sq_flow(forward_img, ldj=None, reverse=True)\n", "print(\"\\nImage (reverse)\\n\", reconst_img)"]}, {"cell_type": "markdown", "id": "38bdebf0", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.018912, "end_time": "2022-05-12T12:19:12.181515", "exception": false, "start_time": "2022-05-12T12:19:12.162603", "status": "completed"}, "tags": []}, "source": ["The split operation divides the input into two parts, and evaluates one part directly on the prior.\n", "So that our flow operation fits to the implementation of the previous layers,\n", "we will return the prior probability of the first part as the log determinant jacobian of the layer.\n", "It has the same effect as if we would combine all variable splits at the\n", "end of the flow, and evaluate them together on the prior."]}, {"cell_type": "code", "execution_count": 21, "id": "3c093208", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:12.221775Z", "iopub.status.busy": "2022-05-12T12:19:12.221330Z", "iopub.status.idle": "2022-05-12T12:19:12.227327Z", "shell.execute_reply": "2022-05-12T12:19:12.226649Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.028576, "end_time": "2022-05-12T12:19:12.228774", "exception": false, "start_time": "2022-05-12T12:19:12.200198", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class SplitFlow(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.prior = torch.distributions.normal.Normal(loc=0.0, scale=1.0)\n", "\n", " def forward(self, z, ldj, reverse=False):\n", " if not reverse:\n", " z, z_split = z.chunk(2, dim=1)\n", " ldj += self.prior.log_prob(z_split).sum(dim=[1, 2, 3])\n", " else:\n", " z_split = self.prior.sample(sample_shape=z.shape).to(device)\n", " z = torch.cat([z, z_split], dim=1)\n", " ldj -= self.prior.log_prob(z_split).sum(dim=[1, 2, 3])\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "892077ee", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.019532, "end_time": "2022-05-12T12:19:12.266987", "exception": false, "start_time": "2022-05-12T12:19:12.247455", "status": "completed"}, "tags": []}, "source": ["### Building a multi-scale flow\n", "\n", "After defining the squeeze and split operation, we are finally able to build our own multi-scale flow.\n", "Deep normalizing flows such as Glow and Flow++ [2,3] often apply a split operation directly after squeezing.\n", "However, with shallow flows, we need to be more thoughtful about where to place the split operation as we need at least a minimum amount of transformations on each variable.\n", "Our setup is inspired by the original RealNVP architecture [1] which is shallower than other,\n", "more recent state-of-the-art architectures.\n", "\n", "Hence, for the MNIST dataset, we will apply the first squeeze operation after two coupling layers, but don't apply a split operation yet.\n", "Because we have only used two coupling layers and each the variable has been only transformed once, a split operation would be too early.\n", "We apply two more coupling layers before finally applying a split flow and squeeze again.\n", "The last four coupling layers operate on a scale of $7\\times 7\\times 8$.\n", "The full flow architecture is shown below.\n", "\n", "
\n", "\n", "Note that while the feature maps inside the coupling layers reduce with the height and width of the input,\n", "the increased number of channels is not directly considered.\n", "To counteract this, we increase the hidden dimensions for the coupling layers on the squeezed input.\n", "The dimensions are often scaled by 2 as this approximately increases the computation cost by 4 canceling with the squeezing operation.\n", "However, we will choose the hidden dimensionalities $32, 48, 64$ for the\n", "three scales respectively to keep the number of parameters reasonable\n", "and show the efficiency of multi-scale architectures."]}, {"cell_type": "code", "execution_count": 22, "id": "c89d5d05", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:12.306654Z", "iopub.status.busy": "2022-05-12T12:19:12.306215Z", "iopub.status.idle": "2022-05-12T12:19:12.313447Z", "shell.execute_reply": "2022-05-12T12:19:12.312759Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.029036, "end_time": "2022-05-12T12:19:12.314986", "exception": false, "start_time": "2022-05-12T12:19:12.285950", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def create_multiscale_flow():\n", " flow_layers = []\n", "\n", " vardeq_layers = [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=2, c_out=2, c_hidden=16),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " for i in range(4)\n", " ]\n", " flow_layers += [VariationalDequantization(vardeq_layers)]\n", "\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=1, c_hidden=32),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " for i in range(2)\n", " ]\n", " flow_layers += [SqueezeFlow()]\n", " for i in range(2):\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=4, c_hidden=48), mask=create_channel_mask(c_in=4, invert=(i % 2 == 1)), c_in=4\n", " )\n", " ]\n", " flow_layers += [SplitFlow(), SqueezeFlow()]\n", " for i in range(4):\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=8, c_hidden=64), mask=create_channel_mask(c_in=8, invert=(i % 2 == 1)), c_in=8\n", " )\n", " ]\n", "\n", " flow_model = ImageFlow(flow_layers).to(device)\n", " return flow_model"]}, {"cell_type": "markdown", "id": "72452657", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.018848, "end_time": "2022-05-12T12:19:12.352716", "exception": false, "start_time": "2022-05-12T12:19:12.333868", "status": "completed"}, "tags": []}, "source": ["We can show the difference in number of parameters below:"]}, {"cell_type": "code", "execution_count": 23, "id": "770b9376", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:12.393479Z", "iopub.status.busy": "2022-05-12T12:19:12.393142Z", "iopub.status.idle": "2022-05-12T12:19:16.320589Z", "shell.execute_reply": "2022-05-12T12:19:16.319851Z"}, "papermill": {"duration": 3.94926, "end_time": "2022-05-12T12:19:16.322228", "exception": false, "start_time": "2022-05-12T12:19:12.372968", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Number of parameters: 335,128\n", "Number of parameters: 379,556\n", "Number of parameters: 1,062,090\n"]}], "source": ["def print_num_params(model):\n", " num_params = sum(np.prod(p.shape) for p in model.parameters())\n", " print(f\"Number of parameters: {num_params:,}\")\n", "\n", "\n", "print_num_params(create_simple_flow(use_vardeq=False))\n", "print_num_params(create_simple_flow(use_vardeq=True))\n", "print_num_params(create_multiscale_flow())"]}, {"cell_type": "markdown", "id": "cbc80813", "metadata": {"papermill": {"duration": 0.018612, "end_time": "2022-05-12T12:19:16.360971", "exception": false, "start_time": "2022-05-12T12:19:16.342359", "status": "completed"}, "tags": []}, "source": ["Although the multi-scale flow has almost 3 times the parameters of the single scale flow,\n", "it is not necessarily more computationally expensive than its counterpart.\n", "We will compare the runtime in the following experiments as well."]}, {"cell_type": "markdown", "id": "c7456b66", "metadata": {"papermill": {"duration": 0.018628, "end_time": "2022-05-12T12:19:16.399001", "exception": false, "start_time": "2022-05-12T12:19:16.380373", "status": "completed"}, "tags": []}, "source": ["## Analysing the flows\n", "\n", "In the last part of the notebook, we will train all the models we have implemented above,\n", "and try to analyze the effect of the multi-scale architecture and variational dequantization.\n", "\n", "### Training flow variants\n", "\n", "Before we can analyse the flow models, we need to train them first.\n", "We provide pre-trained models that contain the validation and test performance, and run-time information.\n", "As flow models are computationally expensive, we advice you to rely on\n", "those pretrained models for a first run through the notebook."]}, {"cell_type": "code", "execution_count": 24, "id": "f7543171", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:16.439375Z", "iopub.status.busy": "2022-05-12T12:19:16.438832Z", "iopub.status.idle": "2022-05-12T12:19:16.736183Z", "shell.execute_reply": "2022-05-12T12:19:16.735384Z"}, "papermill": {"duration": 0.320435, "end_time": "2022-05-12T12:19:16.738208", "exception": false, "start_time": "2022-05-12T12:19:16.417773", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model, loading...\n", "Found pretrained model, loading...\n", "Found pretrained model, loading...\n"]}], "source": ["flow_dict = {\"simple\": {}, \"vardeq\": {}, \"multiscale\": {}}\n", "flow_dict[\"simple\"][\"model\"], flow_dict[\"simple\"][\"result\"] = train_flow(\n", " create_simple_flow(use_vardeq=False), model_name=\"MNISTFlow_simple\"\n", ")\n", "flow_dict[\"vardeq\"][\"model\"], flow_dict[\"vardeq\"][\"result\"] = train_flow(\n", " create_simple_flow(use_vardeq=True), model_name=\"MNISTFlow_vardeq\"\n", ")\n", "flow_dict[\"multiscale\"][\"model\"], flow_dict[\"multiscale\"][\"result\"] = train_flow(\n", " create_multiscale_flow(), model_name=\"MNISTFlow_multiscale\"\n", ")"]}, {"cell_type": "markdown", "id": "5d40ee0c", "metadata": {"papermill": {"duration": 0.018656, "end_time": "2022-05-12T12:19:16.777476", "exception": false, "start_time": "2022-05-12T12:19:16.758820", "status": "completed"}, "tags": []}, "source": ["### Density modeling and sampling\n", "\n", "Firstly, we can compare the models on their quantitative results.\n", "The following table shows all important statistics.\n", "The inference time specifies the time needed to determine the\n", "probability for a batch of 64 images for each model, and the sampling\n", "time the duration it took to sample a batch of 64 images."]}, {"cell_type": "code", "execution_count": 25, "id": "cdee5b41", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:16.817610Z", "iopub.status.busy": "2022-05-12T12:19:16.817150Z", "iopub.status.idle": "2022-05-12T12:19:16.822903Z", "shell.execute_reply": "2022-05-12T12:19:16.822265Z"}, "papermill": {"duration": 0.027622, "end_time": "2022-05-12T12:19:16.824400", "exception": false, "start_time": "2022-05-12T12:19:16.796778", "status": "completed"}, "tags": []}, "outputs": [{"data": {"text/html": ["\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["%%html\n", "\n", ""]}, {"cell_type": "code", "execution_count": 26, "id": "1558a7b3", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:16.863872Z", "iopub.status.busy": "2022-05-12T12:19:16.863266Z", "iopub.status.idle": "2022-05-12T12:19:16.882407Z", "shell.execute_reply": "2022-05-12T12:19:16.881767Z"}, "papermill": {"duration": 0.040403, "end_time": "2022-05-12T12:19:16.883868", "exception": false, "start_time": "2022-05-12T12:19:16.843465", "status": "completed"}, "tags": []}, "outputs": [{"data": {"text/html": ["\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Model Validation Bpd Test Bpd Inference time Sampling time Num Parameters
simple 1.109 bpd 1.107 bpd 51 ms 50 ms 335,128
vardeq 1.068 bpd 1.066 bpd 69 ms 50 ms 379,556
multiscale1.029 bpd 1.026 bpd 40 ms 22 ms 1,062,090
"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["\n", "table = [\n", " [\n", " key,\n", " \"%4.3f bpd\" % flow_dict[key][\"result\"][\"val\"][0][\"test_bpd\"],\n", " \"%4.3f bpd\" % flow_dict[key][\"result\"][\"test\"][0][\"test_bpd\"],\n", " \"%2.0f ms\" % (1000 * flow_dict[key][\"result\"][\"time\"]),\n", " \"%2.0f ms\" % (1000 * flow_dict[key][\"result\"].get(\"samp_time\", 0)),\n", " \"{:,}\".format(sum(np.prod(p.shape) for p in flow_dict[key][\"model\"].parameters())),\n", " ]\n", " for key in flow_dict\n", "]\n", "display(\n", " HTML(\n", " tabulate.tabulate(\n", " table,\n", " tablefmt=\"html\",\n", " headers=[\"Model\", \"Validation Bpd\", \"Test Bpd\", \"Inference time\", \"Sampling time\", \"Num Parameters\"],\n", " )\n", " )\n", ")"]}, {"cell_type": "markdown", "id": "ebfd4803", "metadata": {"papermill": {"duration": 0.019133, "end_time": "2022-05-12T12:19:16.922751", "exception": false, "start_time": "2022-05-12T12:19:16.903618", "status": "completed"}, "tags": []}, "source": ["As we have intially expected, using variational dequantization improves upon standard dequantization in terms of bits per dimension.\n", "Although the difference with 0.04bpd doesn't seem impressive first, it is a considerably step for generative models\n", "(most state-of-the-art models improve upon previous models in a range of 0.02-0.1bpd on CIFAR with three times as high bpd).\n", "While it takes longer to evaluate the probability of an image due to the variational dequantization,\n", "which also leads to a longer training time, it does not have an effect on the sampling time.\n", "This is because inverting variational dequantization is the same as dequantization: finding the next lower integer.\n", "\n", "When we compare the two models to multi-scale architecture, we can see that the bits per dimension score again dropped by about 0.04bpd.\n", "Additionally, the inference time and sampling time improved notably despite having more parameters.\n", "Thus, we see that the multi-scale flow is not only stronger for density modeling, but also more efficient.\n", "\n", "Next, we can test the sampling quality of the models.\n", "We should note that the samples for variational dequantization and standard dequantization are very similar,\n", "and hence we visualize here only the ones for variational dequantization and the multi-scale model.\n", "However, feel free to also test out the `\"simple\"` model.\n", "The seeds are set to obtain reproducable generations and are not cherry picked."]}, {"cell_type": "code", "execution_count": 27, "id": "897b3aa5", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:16.962505Z", "iopub.status.busy": "2022-05-12T12:19:16.961837Z", "iopub.status.idle": "2022-05-12T12:19:17.145249Z", "shell.execute_reply": "2022-05-12T12:19:17.144514Z"}, "papermill": {"duration": 0.205115, "end_time": "2022-05-12T12:19:17.146940", "exception": false, "start_time": "2022-05-12T12:19:16.941825", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 44\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzQ3LjA0IDM0Ny4wNCBdCi9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUiAvVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxVjs0OgkAMhO99inmC/aEbOKskG4/ogQfYLOgGNEgir281LoTDl5lJ2ml1Hd/3EC/+iNOV9JbCTBZJ6GGQhAUWXujJSBqJXaWMEztk+5dBBlZ7I+poQqWKH8yFKl2WV0SLB/RBWmepTsIijR77R6a88L3MzIrLtTGM0GeL+omGGvoATL0uPQplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjEzNgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjUzICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDQ2MiAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgNDYzIC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggNDYyID4+CnN0cmVhbQp4nO2de7wVVaHHPR4hAqFIjoKFZXLJRxlaISqWlZCv6yuRq5YvjDC93bTSCrlSkIZl3TSp0KwbmZnKLb2hktdHikJZaHiNA4WRVEhxgAC9eDifz/39cIYze1gzs9bMXot9Nr/vP4e95/GbNd/hs9fMrMcu/y6aiF129AGIeiKdTYV0NhXS2VRIZ1MhnU2FdDYV23R2BWNHBe4MRZROj4nBA6XTZ2LwQOn0mRg8UDp9JgYPlE6ficEDpdNnYvBA6fSZGDxQOn0mBg+UTp+JwQOl02di8EDp9JmYs8YvweGHH74LqMehSWeAxJw1pLNqYEMVUTqrBjZUEXdynatXr2bRdwWXgg3ANbBhitgJPg5aW1t3A6eBnH0de+yxtwGrQOn0gnQWIJ02gdLpBenM4dPgyCOPROlbd40YDp566imnwIYp4krwZoDyvAqcCTLWvB4MGjToKrAFtANcCtmB0ukF6cxBOptC5/PgStAXoOgo16BDwQDQAvbdd98HQGKDVeBi8DLYPrBhivgQeC3AZYkfxmNz9jMaoJhUyY+oQKyO/mkOlE4vSKcB6ZRO6axc1pfAFnNOdnTOGtjVF8HrQWt0c3bwwQcvA1y6GPC2Dbeh48F68Bdw8803fwzMBObAhtFJLgS4JK8AndvXbl4EN4CB4PTTT7cOlE4vSGcS6ZRO6ayHTh7MuwGfqJ4BrKNT3/4J3AE+AHB18FlBa8R+YPny5YZd/QjEayGff64B5sCG0nkOGDly5KmANaPU0vcB1vas/49Ip0+kUzoTSKd01kUnYU3kM2APYB2d+pZHH5vBP1n9oVJUgQ5+Bhj2c8kll+wOYvHYiLv9P2AObCidET8EvcAh4HbwwAMPrAN7Aj6ddw2UTi9Ip3RKp3TWBrroRF3lRvA3YB2TE536loVq7YY6+fcgYNjHC6B3797xG21eC7NmzcoPbESd5FHwOcBC9OvXbwJ4F2Dd0jVQOr0gndIZUOcfwJlnnsnTdxGwjsmJTn37VhC3BELKvonXfAauBtHKvA6mg+yVG1wn4VPvqPlQS4Jnwdq1ax8BnwWsSIwfP55tjcyB0ukF6ZRO6ZTO2kBbnfNBnz59qPNa4Fw+Q3TqWzYp5Z3XqwHuJgeD+4Fh+8cBjqUPVLLCdBmwCmxknWQBwGk+APAS5aXKxk+4TD8B4ttr/FkKzIHS6QXplE7plM7aQBudrAZdAKKb+7nAuXyG6IylrP9s2LDhOvA1YFjjCBAV7gvAOrDRdSZgg1q6Y2upb3/722xonKgmfh6YA6XTC9IZI51d0mkTKJ1bb2/3B9ETcf7hx0vAB0GpWpFVWTeDJ0Hiqz8C1nviV6GXX365U2AP0sl3nseBt4HJkyfHvXL4CvTuu+82PF6RTp9IZwrplM78wJ1c5xPgJBDfwI4YMYI9K3iby96JzNhrr73YmYRtZOOt5s2bNwtkvHZ2KitvojnizD1g7dq17MwZv6t+A1iyZIndyel5OglL/gtw1FFHsekQ//kbsH1TqESgdHpBOg1IZ910dnR0/DPoDeBsL3Y1aG9vT+9lzZo1bNJyArgTsEMeJL8GsE97vcoKlWvxc8HLK34ReBew3r5n6uSIAGyvidJ+B1gFSqcXpDMH6ZROQ1ntE6vScDpvvfVW3t/x1L0XZO+I3dnYtZ1r8tki/uBOaXJxtGWpuuE98FDAJFYMNm3axBekvP89G9xxxx35gZaJK1eu/B1YAZwPMZVYentWg1ihHAZwp13cL1A6s5BO6TQnlt5eOqUzTydg9z4+oOXLt+IjYL92qsXJ5fkuji7epQEeD3fPVqdnnHEG/8l2NdFX+YEFiex7w2fR/fv3HwtQF+wodYiJxFLb8m0mn96wsyPf+K5fv946UDq3IZ1JpHP7xFLbSqd0Wuj8FjgQuB3NkiVL1ucfQZWysi0q/bGaNmTIkClgI2BdwdzjPhFYkMjWSVDZf9SoUaUOzZDovuHSpUtZtYt7t7LRsXm0cHOgdG5DOk1IZ1dj6pwD/gm4H5RVdKlt+TveBvjC1aqcicCcRI5pTZW8cZ43b16pQzMkum+IW+t9wG4RrBNYjV0gnUmkszzSmY10Wgfu5DpZo1kE8ncUv5bjrRJbU1pHF69oYBpgeyF2K7feKF/nbwEngWAfw38Bzz//fKlDMySW2pbzU7DjCG+tWS1yCZRO6axBOrMTS20rnXaBO7lOC3D3znmHeIKPBnYblQ3kxcIXm7y/Zvdy6w1zijh9+nQ+G2WVg0qjGOdDy0isvB/XQOmUzm6kMzex8n5cA6VTOmvgnTcn4+FDB6do56TnQPxsmi16rTfMKOJjoF+/fnEXnImAQ1VvP0uSO9JZjHQWBkqndNYgndmJlffjGlhd523gcut+7Ylo5yT2Io11fhe4BiYTZ4No0l72qOEAqpzolqOw9u7dm932nQ/PkFhpH2UCpVM6a5DO7MRK+ygT2NBTmadgp4Yvg+PBr4BrYDKRd8lsiDNt2rTl0Wx1/N3kg2dcMNLpHu28nXQWBkqndNYn2nk76SwM7Ek6qwbuDEWUTo+JwQOl02di8EDp9JkYPFA6fSYGD5ROn4nBA6XTZ2LwQOn0mRg8sFunaAaks6mQzqZCOpsK6WwqpLOpkM6mQvedHhODB0qnz8TggdLpMzF4oHT6TAwe2JN0ckhLTrHEidXKBPaAIlYNlE4vSGcx0lkYKJ1ekM5i5gO2jXVqY5sItEjkFTNmzBj2Efw6cD7EVGLp7csGSmcN0lk62nk76SwMlM4apLN0tPN2ZwEOZs7pC8oEFiTyKhkNWlpaBgN2tXc+xFRi6e3LBkrnNqSzUrTzdtJZGCid25DOStE5a8yZM+cY0A9wSFJOibF69eq3AM6wy4rKl8CLL77oFJiT2A7YbZSDuR199NF/B05FykistI8ygdK5FemsHJ2zhnSWDCypk7MKPQlwgo8ApaIzli4DAwYMaI2guyHg5Zdf5lL2eJ8HVgHXwIzETYAztlPlj0Hx/orHgJbOGOksg3TGSKd0No/OnwPObs7Xjn379r0WlIrOWPoUiEZq41o8waeDwsm8iwMzEnlpcnwbzo5evK9vgHcDq8SCfVUpkzlQOqVTOqVzu+iMpdJZhhI6V65cycKwyvB28GHw4IMPlo7OWMop0aHzahB/taViuZ2vWAMcQe4zgPM6/BlYJeascT3Arm4BfI+bvztWAPcG2VecdLohnVlIZ5d0VkA60/CM7rnnnu8BPNF2UwFmkB/I89be3l4pwRzoppOP4TlJ36ngiCOO4IipnENwCbBOzFnjPwH+X/ApCQdE5xuHZ4F55U6wC/gDyA2Uziyk0xLp7JLOCkinicWLF3Ni+OIVC8gO/JvVLOwp+Mw+/6F4KZ1sZ9vSTTwXEmslxRsXB7I6Fb0vj/kUMK+8EewLssOlswDptEQ6u3a0Tt4NXnbZZZzSqLgwBWQH0kz0T86hzmejd4D83bEFJU9297YZgW462UCJu+WJ3meffdiaiD9hdhvbBa5Zs+Z34BowALDDxgsvvOB0lKlA6cxCOt2iDUukUzqls8tZ5w8AynYDKBVniM5Z46abbnoT4NncC/AVaPbKOClr3gkGDx7MZj/ZgW46GfoFwNe7HR0dTtu6txX6CGCDpdmzZ/8WuKV1SWch0ukWnbOGdLoinQU0tU7Sp08fPrn8JXCOM0RnLGUdaOjQofE0umyRxDOav0vU0i6D+4zp1EvprIRzIN97Ri2L9wSPgjKB0ukF6YyRTumUTnedV155JU8y39Q5RWVEG5bwMQKHaUOhegGmUWdx7/m/AtyEnwCyAxtV53OANb/WbpItpVwCpdML0kmkUzqJdLrrHD16NG/sLwV/AU5xhmjDkqh/CutA/EOlVwG73T7zzDMZrW0aXOdc0FrL90CZQOn0gnQS6Qyoc/fdd981gp0bnOIM0YYlD4GBIGqXMxKUTtk+sFF1ss9jPG7Aaaed9hXQ4fqUWDp9Ip1EOqXTOlA6a4h+ptl4hs37PwqcIlPRhiUbwKEAe58InAuVH9ioOu8DVMmrN9H1kc/GX47G37ENlE4vSCeRTum0DpTOGqZOnToTsBN9G+ARHHjggbh7f8busFPRGUvXgoULF1btQG8IbFSdhwOezESTYmpkK6kLgEugdHpBOmOks0s6bQKl0wSjOHIbe3f36tUru4N3QbTbRmbWA+tAtyJWOCqHIi4HVLkHWAmir9kbpj/YH7gESqcJ6XRAOruRziTSWRtYQuc/wEWAjxMuvvhi6+3S0RlL/wRuv/32bwEOS5qzn/POO4+PNd4KrALTiaxtHXbYYem177nnHtZCzgQ/BHaDIZgTC9Zat24dRzygztTgDhwfnf9dJgCXQOmUTuksogfpZOd19l17DVixYoX1dulow5JHQPTmLx78hYX6EMBJZqdE/qr9BHA8rugV4feBVaApcdKkSek73DVr1sSjzvBvqcFx7M4pktgkikV9P0h0Z9wPDAccAs0lUDqlUzrLFzGVJJ3SmR1YQudi8DowAlhvZIg2LEHdYB07NUStklqj9rbRi1beiL0PJMZU49jPvAasAk2JQ4cOZU+IhwHH6PwpGDJkCPceN/TlV08//XS9ipgElSxeLJzEKb5qUQU6A7CIVwDXQOmUTuksX8Qk0imduYGNN7sua1qLFi2Kz2isc9du4o9tbW13AutAU+Khhx7KU8duh+cDVLjuGTt2bOIa2po0atSo2eBngFfc70GVIqa4G/AgEoGs6Tm98ZXOrUhnZaSzFumUzowiNpzOCFa4OADNVwHHUsP9Piep50c+pOfixx9/3CnQlLhgwYLzwE2Ak953f7tgIUAV6OlPg4EDB8YjfH8TFA+z5nROWQfjsxB2/Dn77LPvB/ZDrdYGSqd0VkQ6a5FO6TQH1kenW/qOun7KJ65du5Yv7TkiRL0m3qgz0umAdBZHVwl0QzoDRAcP3BmKKJ0eE4MHSqfPxOCB0ukzMXigdPpMDB4onT4TgwdKp8/E4IHS6TMxeKB0+kwMHiidPhODB0qnz8TggdLpMzF4YLdO0QxIZ1MhnU2FdDYV0tlUSGdTIZ1Nhe47PSYGD5ROn4nBA6XTZ2LwQOn0mRg8UDp9JgYPlE6ficEDpdNnYvBA6fSZGDxQOn0mBg+UTp+JwQOl02di8EDp9JkYPNBF58uvUK/oyvtxDaycyP6dF4IDwZIlS4oTqwbaI53uSGdOdOX9uAZKZw3z58+/FHDsjvHgFlC80apVqzKiyxx1OSrrXLZsGQcySYw8tuuMGTOKE0sHOiOdDkhnjHQGQTodaEqdAwYMSBaIQzHn14w4dc/JJ5+cEZ369o/gecDpWOs4FWsi0P3sctBKTpa622678dA4vjbH1WTpp0yZUpxY+oidkU4LpDOJdAZDOi1oWp0ctDLSeDD4Hige6fk/AM4EqlDzDdHJrx577LF4ZoRXg+45vR1YsWLFe4BhKOjSOsmjIDGPEh8lcI6nSZMm5WwkndJZBem0RDpjpDMYDjpZIXgtiMZbtp8ybyKAfw6fbIhOfrVo0SJeJV8HjJg2bZpbaX4N9ttvP050wGfkqaWVdKbg4wTGjB07NmetqoE8x5wIMfrISZhOAycbKpY1gdLpiHRKpzPSKZ2WOslhAGY4ZftzgF9xvoI5c+ZwCvmMjd4OULH5MzBEp1bmAOTtYCh46KGH7Of46QBjQDRJECttGWW1218+XwOczu/UU0/NWatqIC8aThgYfXwDYFXxdJAbKJ2OSKd0OuNbJ6dtiSZH5RTmnBl1ENh77715r7cGpH5S8Wu4qA8w/AxmB3KCGF4tKZU0jfL9D+CLVs77xie7WPILcCyIbonfBJ4CGWXNL6Id4wCj8n/eqwZeACaA6OMegCo5n11uoHQ6Ip0Z0YYU6YyQTul00EkOOOAAPq0cAAYDZpx00kl8xsry0W5US/pvwLaoo0F2tH3hZoK2tja+PmXt6o2AlZ5DDjlkOIjfv+LQngU5ZbVPzOd2gAs4Z42qgZzOfCWIPhZPrSSd5ZFOQ7R94aRTOg1ltU/Mpzl0JngJsPUM6ih0x4lSaZqVFSx9C+AJvhxkR9slbQCsgGHv8Syo7CjDGgkuFj7gZRIrXQ8//HBBWZ2KWImqgdPBi8A1UDq9IJ0ZSKdLoHR6oYfpTLEcjAU8u2PGjKFZ1lU+B7Kj7XbNpxOtrzzAWAaSi2bNmkWVlM2+M9n76Hk6ywZKpxekMwPpdAmUTi/0bJ0xswFOMK3ywfga8722U+CXAXXiCkk3UrrllluYxKXvB9n7kM5ySGdtYvBA6fRCc+jk+7j29vYTwJMgP7p4dy8AtgLiL+RZZ52VXjxz5sz4ee2rwPnnn/8p8A9gDpROR6SzNjF4oHR6QToNSKdrYP11vuMd77CLzllj8+bNbGvEN6txZeeuu+5KrzVhwoTXA7b4/ymYPHkyzX4cmAOl0xHprE0MHiidXpDOJNJZMrC+Ov8Khg0bZhedsfQBcNFFF0U9Tlpp6Fxg6LKyYMGC5DvBl156aX/ADcyB0umIdNYmBg+UTi9IZ4x0Vgisr07e9x933HFsIZozzHh2IHuCDgHRwwFWg2YADv5ilb8KPA7MgT1BJ65ajgbAtlhlAqXTC9JJpFM6E0inm87V4McgZ5XNmzdzXAPe4C8xz05hDuS4CG8DURXoCNAPGLprOtNDdH4ArFu37ovAeYIT6fSJdEpneJ0ciYq/avk91KiT3QzYMNNwv2gOZNvK+PXlKaecshFwKJZEt7jS9BCd7Og4d+5c+yF4tg+UTi9Ip3RKpx3SmcFnwHWgeE02iuWIMZ2dnRnRqW8XAN5oHgKWLl36WcAWQP8GrA4th3rq5GNpXG2n5O+uVCAv6N69e3MAN2el0lkO6cyITn0rndJph3RmbMJ6ClvwnAOuAOeeey5PP/umOEenvmWHFlQE5v4GLF68uA3wcQJHXk33BHSlnjqvB+yCOH369OJEt11zGJ9evXodA/4OnLaVznJIZ0Z06lvplE6nQOncxtOgb9++HLGNcRzNmkNoX3jhhayrsH7gHJ2zxowZM3jhxO2F7gX2u88MrJtO1thQGypOdN89/quwxHxx7LSddJZDOjOic9aQTuksDJTObdwEUAE4G/AjO1HS4aZNm9wPuTjwzjvv5GA2HESVr6+zX+byVQBHjus0PK3YLrAuOjm+OS+vaPaR/ET33V999dXUyWYCTttJZzmkMyM6Zw3p9KyT4OdzPXA/RnN0/kp8z/knkL8Wp4XljepDoDCwvM4tr2C/QelA/DKz1sAWpk7bSacD0lkcnb+SdDptJ50ONKXOelHHQA7xxskbOHNhYWDPLKJToHR6QTpLIZ21gdLpBemsRH5FpSmKaBUonV6QzkpIp3R6RDr9B+4MRZROj4nBA6XTZ2LwQOn0mRg8sFunaAaks6mQzqZCOpsK6WwqpLOpkM6mQvedHhODB0qnz8TggdLpMzF4oHT6TAweKJ0+E4MHSqfPxOCB0ukzMXigdPpMDB4onT4TgwdKp8/E4IHS6TMxeKB0+kwMHiidPhODB0qnz8TggdLpMzF4oHT6TAweKJ0+E4MHSqfPxOCB0ukzMXjgzq5zy5Yt+wJ2yv89MKzBceIWLlx4IuCsTRwQex2wSqzHwdshnVuRznpFBw+UTp/RwQOl02e020Zlps1LBWYkxoOocv6F8wG+oi/OKM4JnT4GomnV4wkMXwe2nzDdkOh+qCimYejunwFOG1UYKJ3SWRHpzEY6rQOl0wtWgTyjj4A/Aw4NjrP4LlAlMCfxzYAzT7BKNHr06MGAY3jT23QAyceDseC1gGr79u17OtgAyhbRxMSJE18Pkl9B40iwHyguonQS6SyPdGYjndaB0umF4sD77ruPFRPOrsvhwf8X4AzyfH8TlA3MSZwGuPuownMA+BR4AKTWfAZwcVQr6g+e234SjlLnlE8o+vXrdzhIfn3iiSfuAW4GxUWUTiKd5ZFOM02n878AT+OgQYPiKeWolTMeRef6G6BsYN2K+BdwzTXXxLehS4A50W23HwQoIgo/6PPgI+A0gBNwLMjfVjrLI52GaMMS6ZRO60Dp9EJ2IO+vWmvhfR5v8G677TaeuveALwKW88Ybb+Qo05ytkHMIZj/V9VHEe++9l8fDc78cWBcxB9xzTnxlotdTfg44ZTwjRo0a9QeQv610VkM6U9GGJdIpndLZ1aA6rwC7vUJcDWL7nGgp7/P58UuAWs855xzecu8NjgIjRox4HmQH1rWIqJzwXF8LshPddvkoaGtriz/yYQVrf3Pnzi3eVjqrIZ2paMMS6ZRO6eyqo05Oos7XhZeCZ4F1dMbSaMIitpj5DjA8BU/At6GzABv2DB8+nP/MDqyrzk9+8pNvAjNBdqLbLlnyBQsWLAN7Aqr8LLDaVjqrIZ2p6Iyl0rlDdU4C/BXpBb4KrKPLBrrjqXrAFp284c1OdN/nuHHj3gh4Ql8DngJW20lnZaQzGV020B3ptEc6sxPd97ljdXZ0dLBCQJXfBfmzoqaiSwWWwpPO4kS3jR4D/fv3j1+j8hGua6B0+kx020g67ZBOS6QzN9Ftox2vs7Oz8xzABve/AKmlD4OBAwemK0jOge3goIMOYt+GUkfZA3TyYQxb00YtpIYDu2cytYHS6QXpzEE6bQOl0wuNqJMVm+JTuCnCsOhsgOMbAQzRdsfJgxgCWlpavgLi4+ET+vvvvx933eP4GvQLIGxboXycA/8VRFUgtipG0e5PLC22IJ0+kc4MpLNLOmuRTjIf8FzdBn4H7A8uYgLA8bGpbAdIRdvt4/uAhcT9NfuncKQ1duB4J4haFPE1L6sOiYSMsjoXwMDHQRsYP358zlpOgb8ELRGR0rZEmyFwNGDHmMJA6XREOqUzn0A6HwQ46vHsMP5RYHdwCd4KouN7AaSii7f/KWDBuANscCuYCvoAeox+Zy4GXFRcVucCGGC/Cz6izm8u6RR4HEjp5DACvwKnApSWS44EhYHS6Yh0Smc+0imd5kCbZ7bsRHA9MC/NeMf5G8ADiw4z2WnOqqx/B/sDCrsA4OP7AM8kezG8BRx//PGfAPn7SZY1ew2e0RsBm/+Y12BNJB6A5hJglVhwXBs3brwKtNTSK0HUK5L//AEoLqJ0EumMkc7sxILjkk7pzC9i5Xa2aZ04IayYDIoGkwFnARz5RvuyYuUPABaOnQI5utmwYcP4SIOPbzPatmZTXMTZ4A3gvWDs2LHsvb9LxKEAS/k44yDAplEZw2Run5izBkcEPeGEE1IqW6JqUHTmtra1veGGG+4DdkWUTiKdMdKZnZizhnRKZ3ER696Z/rrrrksOJBPpNETn7OORRx6Jq1HR4/e49el5wPmQ7ItIpXDHEVf5VP+HILqmWJa+4E5gnZizxo9A2mVL92QQ4N2A/eudiiidSaQzjXR2SWct0tkQOn8Nxo0bF2nc2mJ0ypQpa4EhOmc/7e3trALFBWM5UQX6NDAM1F2MXRE5sAIfWuBaYvUurnCx/ypqQ2eCn4AXgXViQVpCY2Lkcr5h4JwQxfMQGooonTHSmUY6axML0hpd5+4g0shj433aE088kRGds5/Ozs5jAMe3GQN+DJYuXVr6sOpRRLu+jqnEnDVWgZEjR/LZ84cAHxZ/GOCsPQmcD0863ZBO6cxZQzpLH5Z0urElgq1fp06d+jQojq4S6EY9r1inxOCB0ukzMXigdPpMDB4onT4Tgwc21mjTfgN3hiJKp8fE4IHS6TMxeKB0+kwMHiidPhODB0qnz8TggdLpMzF4oHT6TAweKJ0+E4MHSqfPxOCB0ukzMXhgt07RDEhnUyGdTYV0NhXS2VRIZ1MhnU2FdDYV/w9K0wIJCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKNzM2MwplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxNyswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwOTE1MCAwMDAwMCBuIAowMDAwMDAwNTg3IDAwMDAwIG4gCjAwMDAwMDA2MDggMDAwMDAgbiAKMDAwMDAwMDY2OCAwMDAwMCBuIAowMDAwMDAwNjg5IDAwMDAwIG4gCjAwMDAwMDA3MTAgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzM2IDAwMDAwIG4gCjAwMDAwMDA1NjcgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNTQ3IDAwMDAwIG4gCjAwMDAwMDA3NDIgMDAwMDAgbiAKMDAwMDAwOTEyOSAwMDAwMCBuIAowMDAwMDA5MjEwIDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo5MzY3CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:17.071922\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["pl.seed_everything(44)\n", "samples = flow_dict[\"vardeq\"][\"model\"].sample(img_shape=[16, 1, 28, 28])\n", "show_imgs(samples.cpu())"]}, {"cell_type": "code", "execution_count": 28, "id": "99b4f488", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:17.191393Z", "iopub.status.busy": "2022-05-12T12:19:17.190645Z", "iopub.status.idle": "2022-05-12T12:19:17.331569Z", "shell.execute_reply": "2022-05-12T12:19:17.330846Z"}, "papermill": {"duration": 0.164542, "end_time": "2022-05-12T12:19:17.333213", "exception": false, "start_time": "2022-05-12T12:19:17.168671", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 44\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzQ3LjA0IDM0Ny4wNCBdCi9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUiAvVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxVjs0OgkAMhO99inmC/aEbOKskG4/ogQfYLOgGNEgir281LoTDl5lJ2ml1Hd/3EC/+iNOV9JbCTBZJ6GGQhAUWXujJSBqJXaWMEztk+5dBBlZ7I+poQqWKH8yFKl2WV0SLB/RBWmepTsIijR77R6a88L3MzIrLtTGM0GeL+omGGvoATL0uPQplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjEzNgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjU0ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDQ2MiAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgNDYzIC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggNDYyID4+CnN0cmVhbQp4nO2df7wWVZ3HufwQlNL4peCaQmLkpmQahD9wTVP8gSiIoKGyS4hpVpYuui6ykZFkoK6SldqWpi6CkKVkrIXXsjUSERRMRSxaBU1SC1G4uPfV58M9B+bOnTk/nufOAI+f9z/I88zMZ77zHnzOzJw5p81/iBqizfbeAdGaSGdNIZ01hXTWFNJZU0hnTSGdNcVWnY2lsb0C3wslSmeBiaUHSmeRiaUHSmeRiaUHSmeRiaUHSmeRiaUHSmeRiaUHSmeRiaUHSmeRiaUHSmeRiaUHSmeRiaUH7ow6XwKPgtjAnajESgOlsxCkMxzpzA2UzkKQzjD+H3QGHwWrV69+DQQH7iQlVhMonYUgnWFIpzNQOgthJ9W5YcMGNko+D34PngHB0RUFkh6gHairq/s6CA6s6ug+BH4C3gXBiWGbfg7cD24BjzzySHvQFqC8ujbg+OOP/wHgiewNlM5ApNMTXVEgkc7cwAp0sqB7wAdAx44dzwOjwd3gbdDQ0LAetFatGRwGTL29gH+FqnU+/vjjXcCVAOU1BCf6F/wb4Om5G9gFtM2kE5gCvIHSGYJ0tkatOUhnbqB0hlDLOr8L+BN9BFi4cGH667Vr1x4KXgDu6OBAyx/ByYBl7w6gsxvwr1iVzl+CPfbYg0eVjZRvguBE/4Ivgv0BN8/mHf74EPgCuBjcBYxSHnJvoHT6kE7plE4/0ukNjNGJ69gvAzYKRoJ8YR8HvwbuaH9gkpUrV54LrgCbwA8BdLIF4V+3Ip1s81wA2OIzR5Q6rwfBiWFJVHoHMCnXAvvVY8B8zDvV3kDpzEM6LdIpnR6ks7V1Llq0iCp5Fe9Y6tZbb+Vv+tPAHe0PTHLfffd9DLwJeDP6OwA6OwL/utE6N4ATgW2gIOVSsCvoDYITgwMbxwLjjfdj7MefBu2a+B7wBkpnFtK5FelslE4v0tnaOi+//HJu9+cgZwke7O7dux8LXgfuaH+g5Q+ga9euLwP7EW9Io/R/Av71o3V+DbwPfBE8CF566SU+uubJswdYB4ISgwMbebg+DHB85wH78QcBPmIbDG3Bld5A6cxCOrcgnc0TgwO3h05c+PD/7atBxrevAD6X69GjBy+kHNuJqvU6wCN73nnn2Y/4jJB1Yl+uAf5txOnESbMXmAD+AsynvHXLk5k3jPnfQYlhgYQPkf8doD3wAOBHs8D7gfn9ngTy+wtJZx7SSaSzZWJYIJFO6WwZGKMThbDv53KQ+uZVcAZgreeee65nO0GBiwFbIvsApmLLRwKq5AuBbJW0adOG3VP9Ox6nc8iQIQy8CCQ+ZXOMLRJeeQcnhgVaeDsY7j4C2Gl5X2AufM0Vadv8fyfSmYd0Smd2YligRTqls3lgjM7JkydzmxlP/Lg+f67ZdWfRokWe7QQF/hiwewwTDwR33HHHAPArwJWpE9/wvoV/x8NL5LU6mnO8V8AjmvjmSUDLR4HgRP+CSXgCIYEVszzTdcjqZKuI57gzUDqbIZ12Y9LZMtG/YBLplM7mgTE6cYC5TRY1HawBb7/9Ni9+2QRi91D3O4hRteKsWHQ84BO/zaCxaQCav4J/ANyJXXfdlS2j4MCAEnmXH4ePNylWAfvx+vXr2euVB5ktvuBE/4JJLgTmnkGSAw44gA8clgFvoHQ2QzqlMz/Rv2AS6ZTO5oGR76icAOiNMZ8A/fv3Z3uFfVH5hn/YbgcF8g44Xx1NnCF8e5SdUOsMWCLq/Ak8up07d+bW+wD7yPiSSy7pDv4XvAOCE/0L8qix8xXvHXQ2zzgM+wE+nsdJHBwonS2QTunMTvQvuH11WjjiDO9gomw+jORv54TYo+tYYsGCBR0AO8/QofmUPfy7Ah5snlPeHjupwMASp06dync3fgP4OJVKGxoaeL3J8KhxxjxL4deQN2YTF5gWviOYqDw4UDpbIJ3SmZ3oWUo6pdMZWKFOjtrG2IkTJ64FvCbiHpwFgqMdS6BtkOxtSnB12xPUmZuat4Pgva2oRMJ2Dy96J0+ezGAORBOV6FiCbzXiijLR9Enq5EAFnuF8MgOl04V0Smci0bGEdEqnN7BCnbyTmuo1w23wjbn8l+hT0Tnfsjk1bdo0Wwz/Ogq0bRoR1Orkn2PHjg3b24p12h0YPHgwR1rz969NJTqW4BE0I53aphCfFqMBaO0eBIL3UjoDkc6w6JxvpVM6gwKlM48/g9QF9VuAhidMmBAWnfPtfaBNmzafBLyVwDv8dNf09kuPvoCD0fCO+F577fUn4N/binXyRc6ZoGPHjqeC8BXdgRwL3PSDsqcoH7TyCF500UXWLu/IO4bCywyUThfSKZ2JxJxvdzidGfAajTc2/XMnuAMfB4MGDbLdWHBtdsBxINUblIOK4dtzgH/X/CVyFDN2H+GrC4mP2TuURxc/ahyWy5+USsz4hu8Y2oHEcLbyoeZPgf363XffnWw6wLJ3aGygdG5BOtNIZ8vEjG+kUzqDA1tPJyd543PP5u8COKLdC7GhVQ8ci0ydOpWvO/h3zV8i2188hGcD89GzgA0vfowD7E/JSMz4ZgGwF8647uTQ3amJkzieNb89BMQGSucWpDONdLZMzPhGOvOQzhaBraPzCdDe8NRTT4VFVxO4hRUrVrBDj3/BsBLZ5ZWvLODgPg843QSbQd8CQV2gMhIzvuHjYfaeNTdr+Uc/wN5JPKdmzJixh5kXYjCIDZTOrUinRTqzEzO+kc48pLNliVXr5A85X9Gz98qXLFkSFl1pYDzhJfJ9/YULF3KkaZ6aHAOag+xUmpjz7RuAt/R79erFE4ZJ9r67ge1KvnAZGyidzZBO6cxPzPlWOqtCOj1wABi2FWbNmsVXPrkH1MmjsWnTprDoqMCqCC8R5+IS09WVpyhnbKgm0bMUmj7fAHa8HQMfOKTuLAQHSmczpFM68xM9S+0wOs8HbZpghx6+KTcR8HotODoqsCriSmxoaGCvnaj38XISK16/0kDpbIF0Smd2YsXrVxoonS14D+psjejSA98LJUpngYmlB0pnkYmlB0pnkYmlB0pnkYmlB0pnkYmlB0pnkYmlB0pnkYmlB27TKWoB6awppLOmkM6aQjprCumsKaSzptB1Z4GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQdKZ5GJpQfuTDr51uUQUGlgZSX+EAwcOJCzm/8IcHzWj4OhQ4c65kOUTj/SmY90BiKdYdHR60lnPjubTjMjPccUrzSwshI5Hf2+++57NOgE7Igxu+++uz+xosCKkM5ApDMsOm4l6XRSvU6+bX777bdzgCrOOcQRsjjn2qOPPjoLcNqenB+X6EAOy9KnTx/+fP0RRO9p1Wfs0qVLWcsFgDqPBe7pYKXTgXR6kc4IpFM6dyKdcPbox0Db5tjJUptGUm7HKUdvuummnOjwNM4DiM19BETtZSqw6qP7ADgTrAANDQ3+xPBNTwOXAjMa3nDwWcARsIPWl854pFM6w5DOcKTTwVUgoTA5TrLRaa3yq7333rvqWr8CKr2FkAisXOfGjRt5rDnpEWf34yx/QYn+LXMupZNBezOt+SGHHMKpIZjUEewGxo8f77g73DxQOkOQTumUzhyksyCdnNSWUXaCgblz5z4CrgS8th4EcNyZzoYSl3riiScqrtXyGYDNsn1gP+IEgm+88Qbvj/vXr0on75Ycd9xxHOe1rWmocFq/oETPpjds2DAa8Oz/JFgO3nrrLfstR2FlYxLffgkEBUqnD+mUTul0IJ0F6bwW2HZP8ugaXge33nqrZyNRB/duYA7l18AmcArgNHpmWqXeYDPwBsbr5Kl6GDBzcdj2yo3AvWJY4IUXXmgL4J2SjCU48wcCLwNBgdLpQjqlM5Ho2fz21VlfX8+5ylnQWYATi/pXyo/2L8iJH/iQkzXDHY8ive0H6rbBaSLcV4LROnnS3AHYHmjXHE5+xBZEtSWi1TEXO98NLAU5S7HcgQMH8pc1KFA6s5DOCKSzUTqlMyAwRmfTRWXdBPAO8K/gjg5beE/A1EsvvbTBPGO8CJg7xOykxLupQYGBibho5qTitt1Dh5MmTfow4EcXA9q+55572Bb8Hci4o+oO5BXlIQBb/glw7AtfocA+jAJhJUpnGumMQzqlUzqDSgzUyctcNDnYXef/gHthD8EH95vAXr6bsvlO3u7ANIN+DIIDAxI5u+6QIUMSj3PbXgKg72HA/aDWKwDahPZJ7/vAihUrwkvkVObUOWPGDMe+cKl+AGVOBmElSmcS6YxHOqVTOoNKDNRpnmSyEPsRWwCQzCPgXjcn2r0Qt34koLQjAD66DdgeSdeDjRs3RgV6EteAoSChkg832fDCvtwCGGw/NkvYuwt9+/YNL5ENyb8B9/TzjwFzm2QxCCtROi3SKZ3SKZ3bVyeTeasYmx4I8AM+w/acAWwf7A3YbgmbzT3o4PIsMaF1fHV0zJgxDOVf+UD5KeBPStXqLvFfgDE0GLCBsyuYB7DEcYCl2vbPwQcf/CQ4EfCjzp07R5foYTZguRdccEF4idJpS5RO6dwJdRJ7BxPwqLLDvfkZS16jTQfOXjvJaPdCXwX2t4q/2DDL/+wCfg783fwzas1f4nJgZZ122mnPgBcB3+J/E+B3k8857RJk+PDh9M8Dwb8OGzYsukQHqwGv8XkE6uvrw0uUTiKd0imdyWj3QtJZpE4+5MTR3Bd8H9hhZiZNmsRibCOF4NuwaP9CVic7CZlOtROBv7jswJxEPtxkm4dm0KLpPG7cOH58IPgy4KVohya4BP80Ru115zlg3bp10SU6+BlgufuAoEfL0mmRTumUznS0fyHpLEonB0+bOXMmB0dJffML8G1gdeI/w6LdC00B3BxfJ+ffm9pddTl9c4ICsxLRzuHpyRS+5sdnjOPHj/86sF2KE+0fqrR/xR/8K9975D2ISkp08DmQuFsdXqJ0SmcS6ZRO6XRTtE43PCL2VsLNN98cFu1eiAOl8bV9FvUJYJoePIQXgoxD6A1MJ/K0uPfee+3TS776cjvAX21Dxyb27NmTf7DnDvsK0fRhhx2G03vmu9nPLKs5pq+AXUxXpLC+UI3S2SiduUhnJSXmIJ1hgdIZCMfptIdg9uzZYdH+zXIkBg6aZm8ntN02PhzbLpdccol7TIRUYDqR71UmBtVJ/GnbPITPAp588sn/Af6k6BIzuAbYMnm7JiZQOqWzBdJZaYkZbH+dfD5orzv9/TSjArkkN7vteWsSDtJ8AwgKTCeyr0/Tr3IzneY/+bgxbFCxaktM8vTTT3NgUFvff4KYQOmUzhZIZzUlJpFO6WyM1snh2e68886Mb/4AzgWJX++w6KDdbFwP+LoBGlu8uuVEQz0A37HHcWco3zpocE6ckFMi70Pvs88+CY0WDhMX/7JGy8T4FadMmWJVcqAAvoYYEyid0tkM6cxPjF9ROqWzMVon75H27t07+SI9PU6YMIE9X3lUOfUsn3v6H0ZWXGuKpUuXssstn1EGBWYlnnnmmdYh379hs+ree+8Nvnr3JMatxGe5OL3svwvOqRQbKJ3S2YR0ehPjVpJO6WweGHMbYfDgwbw3zbc1Dgd88mduHTCd98v/CoKjg3e1asJLbOXEuJV4YE0z6CRQ0SNd6SwwMW4l6QxDOrOQztjEuJVK1vn888+fDdi1lO9ysPlwzTXX8C4x5+cJ7/0qndm8Crp27cp/HxxyoaJXWKWzwMS4laQzKlA6C0E6s2F3JKjkDWNOFV9JoHQWmRi3knRGBe7oJUpnVOCOXqJ0RgW+F0qUzgITSw+UziITSw+UziITSw+UziITSw+UziITSw+UziITSw+UziITSw/cplPUAtJZU0hnTSGdNYV01hTSWVNIZ02h684CE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sPlM4iE0sP3Jl0vgM4YuYgwA0ED83VGiX+BbwFNoFfgfvvv38x+FeQn1hNYBzSGYd0eqKj15POfKQzjprU+SMwFYwD+4MePXocC/YCHwX8KjWpd3509C7PAx8Edjbx6dOnh61Ytc6lS5fuDf4RcIzOs8ALL7zgT/RvmQNpLwSoZfrHm3gEVDzjrHQGIJ3+6KjdJdLpDZTOEGpK55uAkaeeeiqPoh0i0/zJufo4jZ+d1XTEiBFh0f59fBvMBpyAsKGh4SbAOVM5rjV3ZNSoUXG1hi2dgEeVw8PgbGUgh4POmX81O9Gz1JQpUzgi3q7Azq9oJk2nXU5IG7yn0hmAdEpnItGzlHRKpzMwROdQYNzxSPLn+tPgePDMM89wooTV4GlwN8DldqvU2tg0qbed82flypW8fcBaeeJwf4466qi4WsOWNlBlYvamy0H4ymGBI0eO5Nb5j4HzEY4BaOkZo+3+GWwEMYHSmYd0Smcq0bOUdEqnMzBEZzfAI9qnT59fgPCi3NH+BTmPEoPvAevXr+dfuwJ+NADg/IkKDNfJ++2DgW3xHXnkkf65lDISPUudcsopTEhOD7xgwYKTAXV2BDyfrrvuuuBA6cxCOiOQTsP20Wl/q+bOnRtWTFi0f8E/g4cA/xu/ZLwQO8UcgqtBbGC4Tk5hbn+0h4F58+bF3UUNC8SFMy8wWWbiU04tew7gjXAqxZXpLBAUKJ1ZSGcE0mmQTul0BobqZGHDhw8PLSckOm6lzZs30+zugG0F/HVzbGB4ImduYsWHAd44xke8pP4SCAuuqMQU7JXE6W979epFA3cBb6B0ZiGdEUhnENLpDpTOLXwf8Lq2Q4cOZwLOGppagrcW60FypnNvdFBNW5k/fz7ngOW0hHeCqHWjddq71D8F5iP2sR0B0HwZ5VcaHMgWVl/ALki3gA0bNrD70HngImBuGPPofwjcBtwlSmcG0imdGYn+JOmUTkeJAToXAG6wR48eR4CR4LdgIujSpQu72/KKl02Hg8GiRYs8JUTr5OGDTvtsMGzC9JaBgYkPPPAAa2GjK3F6/h6wg+8u4GEQlOhJWr58+RcB60r1FeJ/8hZCu23woz3AypUrHSVKZxrplM7sRE+SdEpn1TpfBNwYimE/IZvKDrb9+/cfAo4EBwB+DLP8Ma+61hQI5WN0vibCP3Ft3evkk0/+CjgD8GBjL78DMjrCxum87LLLeIBHNe/GuwzYu/Ls++veRlggjhqbdgmNpsXZge8ArQfsLYXTiw/t+VX+NqUzD+mUzuxET9L20cmfrm8A89iTF0B83MgrscRSr4JrgZmMu+paE/BxH0pmT0Xuy1FgHwCjZ4PbAX/ozNE4EeTU6k/6AWjX9N7Gt0HimwfMTypvn7KTpvvxZ1jgfvvtl9DYjmrxyzw8dW8c1/SfAlyCvaWcgdLZDOm0SGd0idIpnc7AmJfpccXDVpFjCXZzadv0jqBjqSid7N/KF9lxDmV01+HbFJ8B9iVFcDHIDgxI5HsTpmGS0rkO8JqTSd2B++50WOC0adO4ST5GHQu+CVatWpVeqr6+nqK5T/mHXzqzkM4k0hlVonSmvpXO5oGtO4gbh1lDuUx33FiNCjwdsJL58+dnfMv+qPbS+6qrrsrpjxpW4h2A24JOvp+S+pal2et9/on9cXS9DStx06ZN/AfgWWrAgAEM5BA8HNbAGSidW5HONNLZPNFTm3SmkM7mga2rkwWeccYZbC6Y7qmOaP/m2LeVlXweZHw9ZsyYHoB35DlaTH4HnrAS7auj4BrAwQHmAN4JR1n3g7oEhx9++OOg2hI9vAzQDuQ+/RvIX1A600hnGumMLdGDdOYHSmckfNq6//778za1Y6mgwOsBq+Ad9ZebDxPOlzeuAPi2MxgP2FDxBnoS2ZwyOtkl6v2A/22GW7XPmamSo8a8/vrr1ZbogTcuDgQI5zBviT6/+YHSuRXpTCOdMSV62P46vwBQcm/gWMofiN9MHj4eySUg9e00wCOL38yw9xvCSuQxa9uSdk3PP+1feZN1unec62qPKQf86QPMk9BPgKBA6dyKdKaRzuAS3UinP1A6I2B7hUOrQcNM4I92LLF48WI2N74KEt1mOeg0R2fmUWWHpaVLl4btWliJfOyY0Ei77Gtr2kC2KeQfEyasRE5qsWzZsoxv1oLjgD2Xjj32WP8IctKZRjqTSGdUidIpnd7AUJ0HAXOAMx7RPQX4UiCvsU8//XT3psLaCSyIfXLYG4i3TR9++GG+n8JhrjnbkSciM9CTyLrYWbi+vv5ngCfOb4Dpp8MhBM4HYYO5uQN5ivI2Sa9evb4H+CI9P+bTzLFjx3YBtustn8EGTYQonWmk0yKd2Yk530qndIaVGKiT74fA1fvAoWAg6AeOOOII/pUda6mSjwz9Y9EEBa4EfJppmyBNr5FnvUfuJ6zEFLwD/hLgyOW4oH8ShA+C6g/kiXPMMcd8GJh/BKdzKAKjka+vTgZhcyo1SqcX6ZTOVKJjCemUTm9gqM4N4Morr+TBTfaYadME77mzQ89roDVqNfDwsa3AERncL8cEBbbKU4aoRM9SZ599Ng+i7brLP/fcc0+OIPcYqCRQOgtBOjOQztjA1n3eGRVdeuAOVyJOfvb35A+lfZTa/C3E6EDpLDLRs5R0VhO4w5UondUE7nAlSmc1ge+FEqWzwMTSA6WzyMTSA6WzyMTSA6WzyMTSA6WzyMTSA6WzyMTSA6WzyMTSA6WzyMTSA6WzyMTSA6WzyMTSA6WzyMTSA7fpFLWAdNYU0llTSGdNIZ01hXTWFNJZU+i6s8DE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOls8jE0gOlM5A/AQ7V+bp7eqPmiRUlcfgdDi/NpL+BoJWkMw7p9ESXHiidhSCdTra/zqVgEOAwZCNGjAiLriiJk6RykoMvg12a4Lia7pElqy5x3bp1HKeOo8VwnFBOOrRmzRrHChUHrl27lvOu2nHdOoJu3brtA34LvIHSGYJ0OpDOfKRTOhtbSyd+rb8BeETt7EDYAY5M5vgRryhw5syZnMKA/mytYBjgGKKTQP66VescOnRoInQLOJ0cK1QUyHPy6KOP5tZZamJqdcJRr72B0hmCdEqndDqQzgJ1ct6hm8BBBx3UNoUZVvxzwB0dnkZZ7du35zDldWaiW866dNVVV80Fdq54tFc8tUaVaLkRdOrUyZZHrbuDhx56yLFSxYGvvfbafwHOEvgXwImd2jZNT/go8AZKpw/plE7pdCCdBep8ECTbPtkcCb4Pqq31YZCY7oNHcwp4p4lLAS1zmqVVq1Z5ag0ukbwNLgBmZnpO88R5RTjP0yeBe+Vqzh/+U3kEcAIOTvGL/MtAUKB05iGd0llRiSlK1Plj0Bm4VdrfGE58lB/tT+MvyP7AzKfEu8L4yXrIzuyGfbGzLXGy1PxpS6N1bga8pLXFnHTSSevMT/OnwN5g48aNjg1UrBObvQgwlHXhR7vT1Vdf7V9POh1Ip3RWVGIa6ZTOxop0zp49O33n0sDWCJ9A8oLMXBjyjqM72h94NjBzELaZPHly+uvRo0fbWQr7A3+t/kTDt4CtbQhItLI4byqPwrPPPuvYQLROnkCcKf2ss86yl9bvB9yRoPWl04F0SmdGYnCgdEpnRmCoTvZUMb1lbEOHs3DfeOONC8Ar4NeAv9xmZ44C1dT6BcCHmx8AbIBlLHLbbbfZc2ks8NfqTiQ8qtOnT2cwN/1dsKZ5r6BjAb8aN26cYzvBgWzWfRbwcJn2D//gjZMfAf8GmgdKZzOkUzrzE/2B0rkV6WwRGKLzGdABJNo+I1r2pp0BzEX9geB3wB3t2Ll58+bxiHJbx4CMJS4EaP7YJ6vudznCdPKc5J0KHEmWyOecvMufWupEwNQ5c+Y4thWmc/HixaOAPag8xv369fshYLC9WxKEdKaRziTS2TLRHbiddD4H2jXB1A+BhSC11L7A7NeXgGNn/LWOHDmSh+yDIPXNcsDfGf5g4riH/UD5S+Qd2YGAe48L5rtBTgFfA1wq+1FuKtGxxH+Dvn37clPHAx7Qx4C/GGegdG5BOhemlpLOlomOJaSTSKczMETnX4HptPJR8BuQqnUasI0lNF3YzcYfnfPtSoBrTersBWzSxo0bJwJzZbvlRu38+fPjas3+dgNg0447zzLdB/WLgEvecMMN/sScb78NzEXmaaABhBXiDZRO6ZRO6WwRnfOtdFZChE4yExx66KEciiX1zR+AVcn32GAjLDrnW74uj4NKZ2x8HQD4Ivtuu+1Wl4A60XRhwyS81uxvbwbcJPtA/QC4t8UbrCwV1/oVl8hnwab7Ls+O6wBv1Xj+FTiRTot0JpHO6BKlUzrdROrMYf369fZJHbkCBEc7lujXr59pKqQVsjcS22J8ZQQf8blgcGBWIq7e7blIre7t8NEnG0tcesmSJRWXeDLgnnfv3t2+5MinFuzCO2vWLH89+YHSKZ3bkM5KSpRO6XRTvU4+Mu7WrZtVyU40UdGOJZYvX34l4DNqeuPSOHzJZ9TngbqmfjXBgenE58EJJ5zA3kg4irPcA8GRnwEmHgbcl/7uEs8BbFsuW7bsFkCN7c1gAYAnDJ9qswNA8CNs6ZTOZkinNzHn2x1OJ0u/BpiLQ44j8iQIW7maH2vLWcCUHxyYTmQ3JBw694hhSUab1ygco7I0TwzbLPkVYNfWVatWDQZMoVb8unYfNmyY36p0SudWpDMoMWyzRDqls2VgBTo3Ag74XNfUFOF77I5OtfnR4Suk4LnEcW7MpWhwYDqRLZvevXvzRQbPBp577rnrAQeh4Wt67sErE4n+XcuBfX4/A0wz81UQFCid0imd0tkiOnyFFNKZGxip83HAkc2MSvatedb9Wnl+dNxKCTgFELsR1TUN1BIcmE7cBAYNGkRRvwQcAZTDtCWW4JnLV0b69OnDijlvQs7LidmJMVW1wNyWruvZsycHzwwKlE7plM7AxJiqWiCd7kDp3AL7sKwGd911l509wD6CfPDBB+P3s9paxwOTz2kiggOzEtGqegHwoS23tycYM2YMR8DpC9jSMq0Rtr3cAz5nJIYtzNPqFYP56E3Ak4fJo0ePbtXbCNIpnf7o+BUN0plfonS+J3SiRo7MfRv4GDDjd9tn1RxDYFaV/VriVlqzZg3HU+X4NOzcw2Pfv3//l/NHPW0ZmJPIjsKUxbrMO6z2VVam8Gp+zpw5cS+SRJX4C2BbWuPGjeMYr3yzlR9xmPKgYOm0SKd0hiWGLVyGzi5durRtCadp4YgzYRN6u6PDV+BN2okTJ9Yl4IDPOApRgY5E/latAvXg/PPPPxfwCeTvQcZgNMGJYQvzJ5lDESSG8WaJfIj8IogJlM4tSOfrMeVlRYevIJ2hgdK5hZrXOWDAAG6XzY69AMdUu/jii9lsiKrLER2+AnV26tSJV4G8Sct22B9BbGB1t1CjiA5kH2K6mzp1KrsQ83lysMpEoHQWgnQ6kM7QQOkshB1OJy5e2S6IGvI4LjpupWuvvZbzDlU00NlOoLOVAqWzEKTTg3SGBEpnIexwOouPLj3wvVCidBaYWHqgdBaZWHqgdBaZWHqgdBaZWHqgdBaZWHqgdBaZWHrgNp2iFpDOmkI6awrprCmks6aQzppCOmsK6awp/g6fHi25CmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKODc3OQplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxNyswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxMDU2OSAwMDAwMCBuIAowMDAwMDAwNTg3IDAwMDAwIG4gCjAwMDAwMDA2MDggMDAwMDAgbiAKMDAwMDAwMDY2OCAwMDAwMCBuIAowMDAwMDAwNjg5IDAwMDAwIG4gCjAwMDAwMDA3MTAgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzM2IDAwMDAwIG4gCjAwMDAwMDA1NjcgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNTQ3IDAwMDAwIG4gCjAwMDAwMDA3NDIgMDAwMDAgbiAKMDAwMDAxMDU0OCAwMDAwMCBuIAowMDAwMDEwNjI5IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgoxMDc4NgolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:17.265579\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["pl.seed_everything(44)\n", "samples = flow_dict[\"multiscale\"][\"model\"].sample(img_shape=[16, 8, 7, 7])\n", "show_imgs(samples.cpu())"]}, {"cell_type": "markdown", "id": "a54b61c2", "metadata": {"papermill": {"duration": 0.020193, "end_time": "2022-05-12T12:19:17.375862", "exception": false, "start_time": "2022-05-12T12:19:17.355669", "status": "completed"}, "tags": []}, "source": ["From the few samples, we can see a clear difference between the simple and the multi-scale model.\n", "The single-scale model has only learned local, small correlations while the multi-scale model was able to learn full,\n", "global relations that form digits.\n", "This show-cases another benefit of the multi-scale model.\n", "In contrast to VAEs, the outputs are sharp as normalizing flows can naturally model complex,\n", "multi-modal distributions while VAEs have the independent decoder output noise.\n", "Nevertheless, the samples from this flow are far from perfect as not all samples show true digits."]}, {"cell_type": "markdown", "id": "66d1bbfd", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.020133, "end_time": "2022-05-12T12:19:17.417021", "exception": false, "start_time": "2022-05-12T12:19:17.396888", "status": "completed"}, "tags": []}, "source": ["### Interpolation in latent space\n", "\n", "Another popular test for the smoothness of the latent space of generative models is to interpolate between two training examples.\n", "As normalizing flows are strictly invertible, we can guarantee that any image is represented in the latent space.\n", "We again compare the variational dequantization model with the multi-scale model below."]}, {"cell_type": "code", "execution_count": 29, "id": "a7cac8db", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:17.458920Z", "iopub.status.busy": "2022-05-12T12:19:17.458467Z", "iopub.status.idle": "2022-05-12T12:19:17.500905Z", "shell.execute_reply": "2022-05-12T12:19:17.500200Z"}, "papermill": {"duration": 0.065572, "end_time": "2022-05-12T12:19:17.502663", "exception": false, "start_time": "2022-05-12T12:19:17.437091", "status": "completed"}, "tags": []}, "outputs": [], "source": ["@torch.no_grad()\n", "def interpolate(model, img1, img2, num_steps=8):\n", " \"\"\"\n", " Args:\n", " model: object of ImageFlow class that represents the (trained) flow model\n", " img1, img2: Image tensors of shape [1, 28, 28]. Images between which should be interpolated.\n", " num_steps: Number of interpolation steps. 8 interpolation steps mean 6 intermediate pictures besides img1 and img2\n", " \"\"\"\n", " imgs = torch.stack([img1, img2], dim=0).to(model.device)\n", " z, _ = model.encode(imgs)\n", " alpha = torch.linspace(0, 1, steps=num_steps, device=z.device).view(-1, 1, 1, 1)\n", " interpolations = z[0:1] * alpha + z[1:2] * (1 - alpha)\n", " interp_imgs = model.sample(interpolations.shape[:1] + imgs.shape[1:], z_init=interpolations)\n", " show_imgs(interp_imgs, row_size=8)\n", "\n", "\n", "exmp_imgs, _ = next(iter(train_loader))"]}, {"cell_type": "code", "execution_count": 30, "id": "52757b12", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:17.545009Z", "iopub.status.busy": "2022-05-12T12:19:17.544482Z", "iopub.status.idle": "2022-05-12T12:19:17.775782Z", "shell.execute_reply": "2022-05-12T12:19:17.775084Z"}, "papermill": {"duration": 0.253822, "end_time": "2022-05-12T12:19:17.777408", "exception": false, "start_time": "2022-05-12T12:19:17.523586", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgNjQzLjI5NzUgOTcuNTYgXQovUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicVY7LDoJADEX3/Yr7BfPCeS1VkolLdOEHTEaUgAZJ5PctLiAuTtLTtLeVdfk8cjmnA44XkpvliTQ6poVCx8zQSExLim0gt6uEid6y9JtEL6zjhlqrO9GNRnhhfjgTRFgmQyW0w7vgiifknoMnTu+YmSMT/n8ZeTEKE7Ac51Vr1sQ8QJ406hcaaugLxjwu9wplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0NAplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjQwICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcG/v7++vr68vLy6urq5ubm4uLi2tra1tbW0tLSzs7OysrKwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKgoKCfn5+enp6dnZ2bm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGPj4+Ojo6NjY2MjIyLi4uKioqIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19dXV1cXFxcXFxbW1taWlpYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDg3NCAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTE2IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggODc0ID4+CnN0cmVhbQp4nO2d+X8VVxnG6dyGIFsoBohEJQJSaRAFoiCLYWkoRVREKFpBSYEUaGkRi1KWLkSRApW1RbHsJcoqVo0iwVRB0daKUUMBZVOBsJo/wefNPbd3zpxzZstlaIb3+0uSmXvPM++Z95lPzpyZ9zR7imGYCGh2uw+AYe4M2GoMEwlsNYaJBLYaw0QCW41hIoGtxjCRkLJafRQ8dRvE4qoV28Ci1IpUjK3WRLViG1iUWmy1TIrFVSu2gUWpxVbLpFhctWIbWJRabLVMisVVK7aBRanFVsukWFy1YhtYlFpstUyKxVUrtoFFqcVWy6RYXLViG1iUWmy1TIrFVSu2gUWpxVbLpFhctWIbWJRabLVMisVVK7aBRanFVsukWFy1Gi9WVVU1FxQXF78I/geMYo3S+S84Cvr06XMXSCQSr4Jbo1V/6tSp3eAeAK1moLS01KjVOLFasH///jZAaA0ARjG2WhPVYqtpYatJmrFN/yi12Gpa2GqSZmzTP0ottpoWtpqk2XTSvwaMA/37918Abp3WP8C8efPeAyzL6gjeAG5a4cV+AB4AWVlZVpLO4JfAKBZaq37OnDkfB1YaWG0TyLAWXSkWg969e1syU6dONWqFD2wRuB/IWp8CRjG2mhtsNbaaHraaKtaoNthqbDU9d4zV8B/5t8GqVas8xRqlMx4kkjQHmzdvbpzWjRs3/gxoNDF//nw0t7miouJxcB+Qz9qnwdWrV41anmJXQGVl5ROgAxg6dOgwUFhYmA0shYdA+MDA38CTTz6ZB0aBkpISaBWi51Stj4HjILQW8RcwadKkZ8G0adM+B1QhAqO23wCtVpDswMm4eunSpW0AV94WQKt1z5o1axzfZKv5gK3GVkvBVvMSa5QOW42tloKt5iXWKB22GlstRTRWWweaNWs2VTvCtB3J9evXXdvxEyBlN01RPPLII/oPHACDBg2iyDAs9RRzPeCysjLSora2bt1KW2/evJn6wLJly1qDRBqjtT3jgsfIXz179iQt0RzNMom/1LOWBQ4dOmTUcu3Ew+ATwNZeUssMpX+IwOob7khsBL1AWstV7EPgBAiuJagGdAvCNSIBPnZVuWYFtNrbQL0JotVq5G0RthpbzQxbza7FVtMfMFvtHdhqLrDV2GoG2GpS+jcRq/0bfBSgoR8CfdgY2LcHvXv3plGjsS3PACsqKlKJN3nyZMfOkydPLgftgIisR48exqY8zxzdV7H10pYtW2geecmSJT8FpaCgoMDRk1APp0UXKz9p4WDMmDHqo4menQiDdgIBtei2z/z584MGBhYsWJA6adqLhgp9/HsguBZRXl5OU/36K5RAmP2uJI27LfLMM8+MBH4C2wDUBthqbDU7bDUTbDWLraaHrdZErYYEPykaWg9oE8x3FlRVVWGsUyaf1seAsS3PADdt2pRqaPr06dfAPoDUXwGGDBniiAxbzD3kqoUEngZsp37ixIl0tXDtyRdeeCGE1r+A/BigX5D9L4MAnfgfUFxcHEKL+gDxBwkMPAdatmyZasBLwaZ1vzLQ9sz+Y8eO0ZOoXja7G1DazJgxgyzSsWPHEFajSxxy/FugVatW5qBycnIqwZEkN21DfUWMrWax1VJ9wFZ7B7aaDFvNCVuNreYvQLaaBFtNIu5Wo3shotlcQPcLunTpotcdOHDgJWBsyzVAOkx59DlITJ+ZTlyipqbGqOV+5pCR6gkyd6jQU9Lel9YyoLblpUU/s7Ky6BG/AJ34ExBMK+UA+lx2dnaQwHbv3o1U6GKbRPOhlaI3CKBFLF++3DZjpwZGW4qKimCxGeIbr4GePXt+AwToRILy2C0SesxzLDh48KD5gGUxtpqArcZWs8FWk/uVreaErdaErUaao0GbNm0cavgffSigx7XEFrdxmq8AcfL0qUFbhw8f/gEgtuA/6FZHjx71FDPupzPimhcqL730UggteqrHeHWy3Mywfft29b9/106kN0u88l1/BDTCUd5rcQ/s+vXra4HxKSwRGLWckCCbqUUPPK2mXEZsdAWrwJUrV8wN+OxEwmg1evZq5cqVF4EfIZsYWy0AbDUJtpo/2GpsNfsRsNUk2GpsNbaaljhYLUVlZeXzoEcSmm/Yt28f7ZgIrIYJhhzkVOMCBLvAFDB48GBqMh+MHDmSpOvq6mwvMswGfsSM+2lQK+cE/WjevDkNvdsCW8qKFFm3bl0IrWNAW2NAVtdsGTdunFFLL/ZlYBQSTZuP4vTp00ECKy8vd31dxmG19JZvggCdSO/AzATqBJeIpnXr1vRq1Z/AjRs36NL2XVBRUUE/AlaNQJbRju5A1nov6Nu372mlk+h1oPPnz+v7yCbGVrPYailNthpbja2maLHVLLaapyZbTZeRmi1stQbuNKtpoZf4xZODGMJv9/q4H6vpoUfSlixZkiWKX6Mfalznr+s9sgSj5w8CGKsP+BqoqqoStyCoHjW5PJGe2xUsXLjQ9Sk3vdYfgXzW6Oq0fv16mp6Ws9L2g8ARGrX0Yl8AluTeRDoM/RahRb+eOXPGb2C0tIzrw4E2sbSC+K0lCFDwgLLMq2XKDLos4+qUKlwudk6YMCHIS3/79+9XtahCA73GePHixfUKtBLNiBEj6NcfAeMZY6ux1dhqNthqbDWHFlvtzrLaJwEdCv6NpWoBXh8Pb7XrYPjw4anQH374YXq5Z968eeeAq5h+586dO+k0LF26VN23BujzBoMOKk4cUGs+EGd+/PjxNOqkxf3qk5NgrnnarVs3o5ZGrLa2Vjw2arOapf5pzNojR474DewXwPXYvcRw4VziU4uu4sZ2PPVw0r4O/HaiajUM238GqGycsYSygF4oU0tgs9XYavb9bLUkbDW2mkOLrcZWc4GtxlZjq6mafkhZbdasWX4+Ht5qfweiX+khtGvXrtHWoqIis8NdtSZPnkyrhpw9e1bdZ5ubonP5HbBt2zZx1ijFAmjRLRG6/4IvY+Q8Qhy3gOaB5ISx/UU/u3fvbtTSiE2dOlV8O6FFNOvIVRtlZWWOQsBGrf5ATX1jDjqOAtDqOAcOHPCjJWojarT8WA3QnTv5joW5E3Nzcx3fxll/P2ijPgWsJScnx7ECK1uNrcZWU8XYamw1hxZbzWKrmWGrsdXYaqqmJ1u3bqWi3jQSTw7zPQlvNTqz6FfSE5XGaJFK/EWV31zF1B0HAbpN/yDe66+/TjOhIi0mAbojs2jRIurFe++9N6AWbaWGsrOzfwWwhV50ogf2ysvL0YFb5Rx0pH5xcbFRSyM2YMAAWzuW0p4jT1UXJBrujBxRxVStXDUlVSdYTjM4Dk+uJW7UohrnfrRckQuXmzvRa8UePxzQXkPYajJsNbaan+ZcYasJ2GpstSZktd8CsSLMGODzWyGtNnv2bPGyBv3XLjbSpBQ2igQ2i8kb6RWIIckyySuBvHMhEA9ZEp06daJleGhP8slCKr4XQIumaFIZ2aJFC2odZvgI0KdHYWEhFZCw0tnfrVu33wGtlkOMagiKwYTNWKrPBOpGseXnwCswqraXZesoN1wzV14aUt+Jhw8f9qMjtMxiLwKvTiTYamw1hxZbTafFVvMM0BW2mkaLrabTYqt5BugKW02jxVbTacXQaquBaHcn8PmtEFajCbXOnTsLLTrJYgeVEcCWgPNq5YAaKigoeFsuz7BixQqqd2dLzrRW/YgRI+hbEyZMCKBVP2rUKFtzthptIpqEzJ49e2jNVXkjVerWajnEbIuwyw1Y8lZL8Z/8OXoryyswX2toiiZthenUz8grzOo7sbKy0lML5w1XwYGUC2PHjlW1cKFrIS95G9JqaIfehtu4cSM9gktV29XlkfLy8qqAGhhbTcBWa4CtxlZrgK3GVtMQX6udO3eO6tOi2cHAde0ZmRBWs73Rkp+f/1cgdswC6NO3jVW6tFqpUszqqEuelCIOHTpEO+i1GXQffWvx4sUBtOo7duxoWYqjEsowSrBly5Z+wLYPedoTaLUcYu8DljRxZfa04sPUn0hZP4GNB57pnw5Tv59qCMh1CPSdWF1d7TqvRnTt2pUqPfQF2iFkL+CnEwnX92Tat2//fJIngP4zc+fO1XciW03AVmOrEWw1i60ma7HV0rDVFNhqbLUUbDU/hLol4h6gEaodJ8TmzJmT2nr58uUigPP4ReAqJm+k5espqUaPHm3bSleLBx98UKQGQYNfJAI9//gAEGNlZQVPV636tWvXpu6F3J3EmPO0Mzc315ZUtBU2O6gsS6nvRMfzlHKiGx1u0yLU+oJaLXozqJ2YVXXDpqCyA/jpRNA5fV8sBDh3dP/CTycS2uJ0frkPvPnmm/pOZKvVs9XYamnYahZbTdZiq/mEreYTthpb7Q62WklJCbWLDqAHEQN8MaDVKM1E8mWD3bt323Z+FWAESmXDXMXkjccBZUC/fv2ugNraWpqL/BKQE68U1NXVfR/Qnxhsfx4EuwVDr6YZveXIcvnXDwMqqKauDGPsxAsAR+loTk4EvWxeXl4ZoLs/Sr1tc/qrRQiM2GRpNZdHH330DUBXMZ9anwUh8r5Tp06LADomSCHxw4cPU3mDgFr5+fmvgatuy96w1dhqbDUbbDW2mnsnstUUmrrVzoBevXpR88OGDfP7Lc8AtWwGIhSaNgonJm+kd27IvchKWsqwQ4cOjvyzkoua0LsrGIfS5M88UF1dHVyLRpSUkVqL2QZNtkEcvXMzbdo0XAFqvbQcYpS5MCfFpljMju1I2rZtS0tDusZmPGH0FpWX22wR03EVFxdjILQ/uNZ5YJwz00OLYWpXhXHtRMFeEMBuj4Fdu3Z5diJbja3GVpNhq4UTkzey1dhq5k4UsNXCickb2WpsNXMnCt4lVsOJqbYahoH56Gy/3/IToMrLQJwyWs0lnJi88RSoALhMiCygH1nJGwpCi1L/MyAnJ4cKHd8AobRS0CNzCenOh03axpAhQ2iZdJ9aejF6H08kiWpv8RelPcW3YcMGLy33wHDVpUcPXd/wateuHa0YTyWiG6VF4FLSDKgiIkQqI/3KK6/sAfrFXRUtsxgVKHbzF2kdSHJNrlltFGOrsdXCBsZWY6vZxeSNbDW2mk2LrSZgq7losdXibjV65hBKVODN71cUTV9i586dGw6EE9QabX7F9DsvX75MA/ulS5emRNKQ1WjtEqV4XUgtOu1vvfXWdIDESyhibcFXQF1dXQAtcyeeOHGCRupUlcRKTyAnGubt+02ZMsWxTkr4wOqTd2P27t1Lld9sQT0Onn766R8D46x/cC2CJqPhpFcB3bzYsWPHPqBOvfvUCvTkRmjYamy1RgZWz1YLIsZWY6uFDayerRZELIjVqB6cqGpL6VhaWoqzeiK4pi8x/NNtO3lUWaGmpiaAlq8zB2gpm+4grUUhPgQuXLiQYS0a6dLrOSL9adBUWFiolln2o+UpRqUgVq9enSpDXFJS8k8QQMgmFvBb4bgNWmw1gq3mrsVWy4AWW41gq7lrsdUyoMVWI9hq7lpstQxovWut9nsgphVozH38+PFwmr7EMHan+s+UlR06dHCUZg4g5uejfwAFBQV0/YDFfg1unRY90EczbTNnznzWWEDBh1Zs0z9KLbYawVZz14pt+kepxVYj2GruWrFN/yi13rVWy5RmbM9clFqxDSxKLbZaJsXiqhXbwKLUYqtlUiyuWrENLEottlomxeKqFdvAotRiq2VSLK5asQ0sSi22WibF4qoV28Ci1GKrZVIsrlqxDSxKLbZaJsXiqhXbwKLUui1WYxjmlsJWY5hIYKsxTCSw1RgmEthqDBMJbDWGiQS2GsNEwv8BuztMkQplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjQzNDAKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMjA1MTIxNDE5MTcrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My41LjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My41LjIpID4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDYwOTcgMDAwMDAgbiAKMDAwMDAwMDU5NiAwMDAwMCBuIAowMDAwMDAwNjE3IDAwMDAwIG4gCjAwMDAwMDA2NzcgMDAwMDAgbiAKMDAwMDAwMDY5OCAwMDAwMCBuIAowMDAwMDAwNzE5IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDMzNyAwMDAwMCBuIAowMDAwMDAwNTc2IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU1NiAwMDAwMCBuIAowMDAwMDAwNzUxIDAwMDAwIG4gCjAwMDAwMDYwNzYgMDAwMDAgbiAKMDAwMDAwNjE1NyAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDE1IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAxNiA+PgpzdGFydHhyZWYKNjMxNAolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:17.639096\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgNjQzLjI5NzUgOTcuNTYgXQovUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicVY7LDoJADEX3/Yr7BfPCeS1VkolLdOEHTEaUgAZJ5PctLiAuTtLTtLeVdfk8cjmnA44XkpvliTQ6poVCx8zQSExLim0gt6uEid6y9JtEL6zjhlqrO9GNRnhhfjgTRFgmQyW0w7vgiifknoMnTu+YmSMT/n8ZeTEKE7Ac51Vr1sQ8QJ406hcaaugLxjwu9wplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0NAplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjQ0ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uYmJiXl5eWlpaVlZWUlJSTk5ORkZGPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRiYmJhYWFgYGBeXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj48PDw7Ozs6Ojo5OTk4ODg3Nzc1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDg3NCAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTE2IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggODc0ID4+CnN0cmVhbQp4nO2d+2MUVxmGc2sBCyJGiIBcFOoFBAGlChaFcElBbgKKpRKVqIiACEShoG2Ri5iqoeWmxVotGrC0IiUJoiIIFRARKOVSSyCaCrWEBKj7B/i+3bPubHbnnDO7wzQZv+eXtnuZZ7+Z805zZs6ck3W/IAgBkPVG/wBB+P9AoiYIgSBRE4RAkKgJQiBI1AQhECRqghAIsahFguD+N0AWVldoCwvSFahMotZCXaEtLEiXRM1PWVhdoS0sSJdEzU9ZWF2hLSxIl0TNT1lYXaEtLEiXRM1PWVhdoS0sSJdEzU9ZWF2hLSxIl0TNT1lYXaEtLEiXV1lNTc0wUFVVlYZMotZCXaEtLEiXRM1PWVhdoS0sSJdEzU9ZWF2hLSxIl0TNT1lYXaEtzOO3fgR69er1KPDu8iK7CkaNGpUNpkyZ4uUnStRatiu0hXn8lkRN52zOR67FuEJbmMdvSdR0zuZ85FqMK7SFefyWRE3nbM5HLmjXvn37ngLeXV5kN0BFRUUJWLly5WvAo8yDayNoD7Kysr4LPHzTo4tlrFq1qjXIycn5OvDuspXVgkKQHcVbrCVqzcMlUUvXJVGzcDb35h+kS6KWrkuiZuFsls3/ypUrR8DBgwd/Br4GZs+evQvcAteLYMmSJR8BrVq1ug3cCYqLi/8A6urqbFy2sj0AXYxRuVHg+ib4DThw4AAzaCOzdEXWr19/O6AIzX8nsP2mdxdOGitzoqD1/w54d9nKxgMVs46gsbHRg0ui1gSJmkZm6ZKo6WQSNYVETSOzdEnUdDKJmkKippFZuiRqOpnXqD0GevfurQqcDHhjA+3Ug9NW9hMwGkCETnZWjx49PgOeBB5klq7I0qVLVQshqmHmg63g2rVrfro+DZJcKgi3jRs3zsblpZUkF0batGkzD9TX1/tW2IQJE5QnJy8v768AaeMvWAH27t1rU5gH14RoK8wuKir6D7D9ZsTzTnwPUFHbBDyIHDKJmkKi5kNhEjWNTKKmkKj5UJhETSOTqCkkaj4UJlHTyKyjdvny5RmAdwzVoXIcuL59+1YCS6dR9m9w3333tQJxl7OVlJeX2xZo+hiPERtCQl1OF9m/f78fLsIz05tA8k5UdOvWjXdMTS4r2bFjx9iNT10YwsB/lJaW+lIYmThxYkwSTVsezh15Cpy2jgO/XJ8AKtZjx461/Vaiy2vU2rVrtx2kJ5OoSdRKfSmMSNQ0MomaRK3Ul8KIRE0js44aWgj3YCewceNGdVPDceDuAJZ/jRtl3wfRthAjufk/DmxkJtdR8GYQ33ITF/9z7ty513T9NdudeOHChQ8CR8aSXfjHn4HJZdVKHnjgAcdmVcNku+EoReWcP3+++wY8uE6AgQMHxvYeDtpPQUNDw0ugLcCLvwd+uHh+HAPg4n28iooKm28lu7xGrbCw0KPIIZOoSdQkaiYkajYyk0uiJlEzIlGzkZlcEjWJmpHgovZ30Lt3b+7BPwK8chpsjeIYYNezZ89/AZNTK+NAxHcCbO7d4AugqKiIfeAZM2Z0BsqFHmq7kydPGgvUFlZbW/sxwO2h/85LPtgqdEXjALrxjgjs1Z1HbA/cmjVrHJvk2Wn16tVfBe8H8Td+CUwuo+y3ANXEotahQ4dysHPnTl6b+BJQb2gvMXlokfcCFWa0g55nzpy5DvAGT1JvA3jjIPDDxafTlOtDwOYrKV1WsrVr1/Jsz6jhSHl3SdSIRE2iZkSiJlHTI1HT0GyjxmOEqhaC5Mcu+Ap+jdqZXwYmp1b2A5ATbYcbQOKbZ8EIoFrJrFmzjAVqC6uuro71LYYOHdrkTXSsugLVVkeCjFw8X6keEjONpujo/r0KBg8erNLG/pzJZWwlvYDaXjF4/vnnHW9+Bag3tTMFWLo4y4HqUncBTHn8zUNA9Q1tTsQmVwQnWJ6L2fTh4241fsXNZZTxDuegQYPUoKzsmpoa7y6JGpGoSdS0SNQkahI1iZpEzQWJmkRN70RhJ/ksSefOnd2rQkPhkcWP4fMhJqe77PDhw28FvFfyyCOPpP4MH83IiT48oR37ZnJxoOW4cePY2vqDixcvNvkAOvWfA6r7/T6QtousiV8T6QPiM8TxJhHvpKnbh4DP0phcWlkd4LlP+ariS6dUVlbyrmVeXIbf4r4hyxbJh5vU5oYAnhHx6q/A6tWreZFEndC+DTJ18WSsYkb+AfDiz8Fm8OCDD+K8vxYN6TAapclllHHyi+w4KmoVFRW/UDz99NPGHyxRk6hJ1EwuiZpETaImUZOouSBRk6jpnXMBD1ffvn3dN7Znzx519YDj3kxOd9ny5cu5nbeAlO+zVfL2pbosMmfOHPffZHIdALmvP/bfZn/0eTRekTl9+jQlvEVfUlKiiiLaW6U2B24JwOb4rBpvm+PMtAyUl5fz7BS7OJP7vwe9TC6t7E8gN446VXDyiLZt2+YkUlBQ4L4hyxbJ85DaHFskwzVixAieMRJd7wCZuiL9+vWLiQCPy6RJk1imeiUmGzhwoMlllHH+QEfUcG7/JIi7slu3bs1XtPe2JWoSNYmaySVRk6hJ1MIXtU8B1uE6VI4rgkyfPp2fGTBggBr3lmaBCxYs4HZSd/jg4d/6qvmwySLhrpsyHjnOJIztfACwhF27dnHMJfLAziD3YjxmhI8Npe0ipQCb5OjK94LEIOQmyDhJgcmllbGDre6rOUKsJIkuys6A9AuLvN76++VYwHPoM888k5Ergmw5Wn8Tg+MddPqrgc7lNWqudOnSxf3JJ4maRE2iZnJJ1CRqEjWJmkTNBYmaRE3jvHz5cjfAg+P6a74D1HHctm2b9ocbCxw+fDjvqfEGHlr/HFATH3b2wgsvsFutXMZFYkwu3n7Bdjiwk+uy5ESfnUcI+GxcbiIzZ878G0jbRXYDtDRnuNzAmWu6yWVsJVxwxqb5o6Hwll/6hQGcYweobZlc2XfddVdGrsiwYcPco3Z7FCXjwNWGhgZXl9eoFRQUdE6C02VkZWVxXGZNykGSEjWJmkTN5JKoSdQkahI1HRI1iZpELbXp0qVLscOffGu6rq5uPogtE6mLo22BaoJtFnju3Dne9F0Orl+/zvU/+vTpE2ujKC51Vcky7ft5CQwCDz/8cHfQpO2fOHEiIxfhfXI1dzdRzuScDRky5NVMh+/xMSs+FOeINHYZL0pg67yB/llQXFzMljJ69Oj9rhOl2xR248YNNMEC1dY5hvUhgBMLV/KsrKzcBzjN/LRp0/gB7N30XaRjx46xpo8DxsEG9fX1r8XHlBI+6qiGeV69etXV5SVqHwevvPJK8me4LmunTp34GTX4M7VMoiZRk6i5IlGTqEnUQha1SPS+Gn+x41bX+fPn+XAB/j51/JE8Rzsg0bLAefPmcVtcgmbr1q08VH3B3Xffrf4qj7WbLl26mFzGI8f2lZswhTJH8uXn58fyrFaE55/ip06dyshFlqrFEhOjpqrlEy/r1q2jz7TSiU0rwabWOY4Nh40+8cQTi0Hi545GcV8H0KYwnAUdrl+D1J+rrq5mqUVFRem7LoIOHTrEouY6YSBnVG7fvj3PminjkUbUXDu0/wTo4nPe79SPmUnUJGoSNa0sIlGTqEnUJGp6JGoSNYmaq+wbgE2jbdu2wxXoA/MVdLI5WHFwdFI1X6L27LPPcnIztkH4OKNZTjxgU6dOXQX4r+h6m1xWUVOuxOsSVPIpvbKyMiaAlw78WHae+yr5OkiPHj045pKzL2zZsoWv6BZrcbi0sqEgVgrgPUgImiQK+5JPzCVeT0ijMLX8jOIxkPwZTlfXtWtXPsh26NCh9F2cuV7dTGPzX7BgQfJn8H+C8/hNA7Gz14OUp5E0osZ7dMmxxSuFADFzX2VHoiZRk6hpZRGJmkRNohbGqDWCLwLHuiytW7emV/2hXBBN3o+B6YfbFMg/iR236lTU1KTJfOKFr95zzz0ml9WRU5tLhNnim2pEGpex88OV79iDcbBn+eYwoF45cuSIjUsrY59I7blYBDZs2HAM3Lhx41EwEbRp04azN2/evDmzwrCfVFeasDlcAKqFnzx5kifhtwP8GC64k5FrG4i7OAyL99XUm5dBeXl5bPYF/JDX3M4jtlGbCbLj7Nix42J0ajVumY9hjRkzhm/oplOTqEUiEjWJmgGJmkRNoiZR0yFRk6hJ1EzOyJUrV9QyoVvj64xcAhz0hlZ0Dpi2YVNgbGq4hQsXxpojYvZSFDZ77sWysjLjD7Yp7Pjx4xwtlzgBATPBCceiYwg5x5n7JGceXKuBmk3ZASSc3sBxccbtGf0mLq2MV3W4WdUi1TmLoyLVDnSQ8tKCl8JWrVrVAzTZ7OTJkznncadOnRwvfhhk5OLa9iUlJZxR0BHvxYsXc+lVx7QLfHPq1Kkml7HZo9VfGT9+vCNtXAhp9uzZ/L+P41XsA/eNSNQiEjWJmgGJmkRNoiZR0yFRk6hJ1EzO1DwJeBy18ygnOa1kK1eu5D5jx/3ll1/mK2o5US7jolt8sonMxsXBgapFtm/f/g6gVk/hK08Bv1wNDQ2LQKz5x3G8snv3bhuXUXYKjBw5MscVpS0tLc24MI79m6aeR3OloKBgt644DweM1+BU1FRbb+KaAWpra00uK9m1a9fuBNkpwdmLa+revHnTfQMStTgStQwLk6hJ1CRqEjWNqwVFjQ+BszJ04Gw+7qXAZcuWseGpBWD4N7p60N1msKVDZvNRNfEwQWdCLcrIo8cu1BHTjS5vLo4RTG7vOVEZf0Hi+vCuLitZZWVl8nIwjli/C2zatMmXwjj0UA1adZD9+izRuexGHT161KYwGxda9s1JIC6JUVhYWGbRk/eyE8EOwGmp4x7e+n0cVFRUGL8tUYsjUfOhMImaQSZRi0jUfClMomaQSdQiEjVfCpOoGWQZRO3s2bO8p8Y9eQuiplr/FID/mgxU80h+bF8rs/kob8rkRucgQMN0XK3gQ+zHgY8uPjuWeD3E8V9c17OxsdHGZXvEvgcc114cUevevfsJP2bdc1BVVXUv4Cx4+fn5HIyIM+VY4Fj43i9XPdi+fTvHzC5fvpyr1e4F6hKapSvN/8N4RKIWR6LmQ2FEoqaRSdQiEjVfCiMSNY1MohaRqPlSGJGoaWQZRO3YsWOxJnILooa+LbuenBoOh4qZVi1kC/Ags/noZqAmTEmc6purufjsqgOcaDv+SF4s0xMmTHgRWLpsjxjHlP4QLFq0iCMj0Y3n8pZcVyflrG2pZZYup9PjV9J2pYdEzYFETe+SqGWCRM2BRE3vkqhlgkQtkb8Ajqzr37+/6mrwgRb3NdJTyixdnDf388ARtXnz5jWkXAcvYxdZsWLFtwA8HwWu66O7ugJtJWF1SdQUEjWNK7TNP0iXRE0hUdO4Qtv8g3RJ1BQSNY0rtM0/SFdLiRqa4SzAx7uee+45D06vMoi48g2+yP6964RjqWUeXQp27C09Gbs8IlHzzyVRSxZJ1Jq6Qtv8g3RJ1JJFErWmrtA2/yBdLSVqaTtDe+SCdIW2sCBdEjU/ZWF1hbawIF0SNT9lYXWFtrAgXRI1P2VhdYW2sCBdEjU/ZWF1hbawIF0SNT9lYXWFtrAgXRI1P2VhdYW2sCBdEjU/ZWF1hbawIF1vSNQEQbilSNQEIRAkaoIQCBI1QQgEiZogBIJETRACQaImCIHwXx/NXzgKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago0MjM5CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagoxNSAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjIwNTEyMTQxOTE3KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNS4yLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNS4yKSA+PgplbmRvYmoKeHJlZgowIDE2CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA2MDA4IDAwMDAwIG4gCjAwMDAwMDA1OTYgMDAwMDAgbiAKMDAwMDAwMDYxNyAwMDAwMCBuIAowMDAwMDAwNjc3IDAwMDAwIG4gCjAwMDAwMDA2OTggMDAwMDAgbiAKMDAwMDAwMDcxOSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzMzcgMDAwMDAgbiAKMDAwMDAwMDU3NiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA1NTYgMDAwMDAgbiAKMDAwMDAwMDc1MSAwMDAwMCBuIAowMDAwMDA1OTg3IDAwMDAwIG4gCjAwMDAwMDYwNjggMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAxNSAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMTYgPj4Kc3RhcnR4cmVmCjYyMjUKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:17.736600\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["pl.seed_everything(42)\n", "for i in range(2):\n", " interpolate(flow_dict[\"vardeq\"][\"model\"], exmp_imgs[2 * i], exmp_imgs[2 * i + 1])"]}, {"cell_type": "code", "execution_count": 31, "id": "e91953f8", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:17.825477Z", "iopub.status.busy": "2022-05-12T12:19:17.824997Z", "iopub.status.idle": "2022-05-12T12:19:18.074474Z", "shell.execute_reply": "2022-05-12T12:19:18.073638Z"}, "papermill": {"duration": 0.275174, "end_time": "2022-05-12T12:19:18.076116", "exception": false, "start_time": "2022-05-12T12:19:17.800942", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgNjQzLjI5NzUgOTcuNTYgXQovUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicVY7LDoJADEX3/Yr7BfPCeS1VkolLdOEHTEaUgAZJ5PctLiAuTtLTtLeVdfk8cjmnA44XkpvliTQ6poVCx8zQSExLim0gt6uEid6y9JtEL6zjhlqrO9GNRnhhfjgTRFgmQyW0w7vgiifknoMnTu+YmSMT/n8ZeTEKE7Ac51Vr1sQ8QJ406hcaaugLxjwu9wplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0NAplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjQzICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7+8vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn58fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1bW1taWlpYWFhWVlZVVVVUVFRTU1NSUlJRUVFQUFBOTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj48PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDg3NCAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTE2IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggODc0ID4+CnN0cmVhbQp4nO2d+38UVxnG26UmMaDQLFJAS1BaxBqwQIvWWxEqQtVWoTFaLl6AQqQQuQha2xRRikYtUFFjUaoUJGqMYkFBEApyESoRTAHljkIpqIEm/4DP65797JnM7czsZCDT5/sLbWYz331n3mc+mZkzZ657hBASA9dd7S9AyKsDRo2QWGDUCIkFRo2QWGDUCIkFRo2QWMhGrTUOHrkKsqS6EltYnK5YZYxaB3UltrA4XYxalLKkuhJbWJwuRi1KWVJdiS0sThejFqUsqa7EFhani1GLUpZUV2ILi9PFqEUpS6orsYXF6WLUopQl1ZXYwuJ0MWpRypLqSmxhcboYtShlSXUltrA4XYxalLKkuhJbWJwuRi1KWVJdHauwV8CaNWsWgpdffrkdXVeuXPk2qASdOnV6ELS0tLi68pMdBXBdB1Kp1BvBn4GrjFHroK6OVRijxqh1WFfHKoxRY9Q6rKtjFcaoMWod1pWf7B8AjXICNDU1+crCiy5duvQCmAzQjuj+TosWLUIerrSD62ugoqJCJKkcTz/9tKsrvGwBGA5SFiYBVxmj1kFdjJoNRs3mTGz7x+li1GwwajZnYts/v5WgS94AZs2a9Ufg5woh+yc4derUV8FrAVrjelBWVnYOeMoCis6C2bNnLwP9+/dP2fgbiMRVX1+/GdTW1u4ExcAuw2Z1dQWRHQNbtmz5OBg4cGAhsLveA1xljNo142LUGLXIYNS8YNQYtchg1Lxg1Bi11h8C7JSPANfPtDjev3B0esq2gyEAZ8/uH3I8rXaW+X1Mzm9lKzou/DvAfqx9+OGHnU/lg7kE9Pqpn4ETJ078EixdunQe+DDo0qWL2mOfAn4uIxm+82nwfVBVVXUTSGWuTVg75AmQd2ECNtf7wVsBJBJkezuCbSBvl+y0UaNGaau1FybMmDHD1WUqOw7eBxyr0SgHrjJGjVFj1Pxg1Exkfh9j1Bg1Xxg1E5nfxxg1Rs2XeKImrTEIYEVfBPYPHAbIhdqPOHHc4rou3wIvXLggMZNv3adPnzbdffHixXVgMEBXynnprl27XFflu+f+CqZMmXIDeAfAxlQL5KDxHKiurpZzYLUVvwRCu4QzAG14O5Dz95kzZ7q2orSqn8u3S7aCYcOGtWlFq7JTBhkqGL4whQw9TKfT2tpd61sP8nL9CLwO+LW+fIO6ujpXl5Fs06ZNcrnKVaIlfJdjRzJqjBqjZiJj1Bg1bxi1jha1I0Bts58C+dG5c+f+BFCD3Ee4EeRk3wSu6/ItcMeOHdkChg4d+i+An+yYNm3aOJA5AdD/Gl+xYoX7FvJz/QGo1XUGOGWbD+bMmfNeoO47aZSC0C7hG0C1ttbljqcXA4Cfy1P2F9ADaK6ULWq5hU+B8IXhyLQfvAlYG7BNO5aUlPQHa9eulUdqwrmkG5955pmuwLX15eCJY/+i1atXyz72crnL5DCPM0851hYUFLi65NGZg+DFF1+Um6HOFywYNUaNUXMVMWqMGqPGqGVg1Nxg1Bg1b2cjeA1Qq34LQNNJvtRe005+8bHHgeu6fAvE6bK2wV4PtL1mP8neuHGj66p899z6nExKcO17hXRuaJcg10KURLan+k+Uade+Dfi5PGWyXAmyteGfIrBy5cp6kG1/gDN+ufQVvjAcDB2OGrkg3wq+C5qbmz3X4+s6efLkp4H9yo76z7eDmpoa56t3ji532XngHrBUSi7OfRm4jh61yxg1Ro1Rs8OoMWqMWmKjJsh9NfSEZpIdiNDdDXB2M0f9dOrUqfkVePny5Wpgr0z+yMefzXImqH4iA4AuXbrkK3Nd/l9w1113aV2i/pEgyJ02nIdm9YWFhUtAaJcgRyetPdT2VI1yg2Xz4uR0mp/LUyazN2l7ShgyZIjc89yzZ8+7gLZpp0+f7r4ik8JyA8r0o6Eq7LbbbvsEaPbLmYnr9OnTznlOp9PfAc8DP4nV5S57CVhbUKnLwOLFi+U6gqGLUWPUGDVGjVFj1Bxg1Bg1Ro1R85cZT3hw4cIF6YCPgaqqqkNAjVCUR0HwLaT18Zn8Cmz9f9ouN4GZM2fKObBcZmlsbJTJ1LDwB0BK7tatmwyDM5F5fmb//v3y3IOssnv37neCzOM7i/4Dxo8fn93MlZWVebtkEKXW/rkUqNlxs5SUlDgPpbO6PGVjgNWlpU5rV6Ski+MDLaaF4eA6Vfvu1isWIkOfyHQKcjXNcZYDU5fcUBs3bpy1+1MFBQU9wYABA5yfy/FzOcjQfnJIGgqsLrlIhzzL0NkAIk3GqDFqjJoGo8aotXExaoxaqAJzMGqMmgOvpqg58nOgnhuSh/gNnSFldXV1chtYdqDXzAttZCZr/h3IZDmL1qDDABomb5d0XEVFhaxSmn3QoEEy0LKoqCj7k8y1khtmz55t4vKUyZTaWpfIau0DAHBgkenjwhd28OBBHBdKtCsVffr0eTfAYau7+snNN988FsiOu++++8K7tAs9ChkoumrVKhlikU6nB4KJoKam5gvgnQBHTJmOz8vlINu7d6/1GpUg1wXl+NfS0iKjImdkkOPM5MmTR4LBgwd/D8gzgq4yRq2VUQtZGKPGqOkykzUzaqEKY9RijJpUJV+lX79+njNK+RZoAnpQXHLL6/Dhw74fz8uFfbhKDfxcCaJy4UTgANgI0CHyk+XLl3fJ3ZiShjl79qyJy1MmI/g+AFSIvw5qa2vbnLaNGDHC9wt7urZv364OE9k8P/vss7LgJMj1aVaL5Z7b0tUl81Vlx8Nm+AxQMZJV5m6N6uAnknscEFxdDrJbbrklZUOOxTKNMlza3rI75eGeBQsWOMsYNUcYNYPCGDVGTZeFczFqBoUxaoyaLgvnYtQMCmPUYoya3GWTLYmzcJOP5xc1tKGU8yEQQBbcI30qT4zBJWfXcout3Vwy81laTemGHSf3KA1dvrLfgJqaGllzbzBq1CjtdF922sSJE32/nqcLG0aL2ufA0aNH5S0zvwBFRUWqFbWEy/USHMQCulYD66UKGRL7WYBcyLxx9phpQGl/s4/7RiwpKbGvQ3aResjQXZSLWwOwyxg1G4yaTea8kFFj1HRZcA+jZpM5L2TUGDVdFtzDqNlkzgsZtbiitmzZMpnVTTZofX29yW+Ej5qMSywuLpZy6xxnhnaVBRS1tLRIz6jNlp2Orx1czc3Ne0Dfvn2z+2j06NEBXEayM2fOyKTksnbbbdnU/Pnz5Zk9E5nzQsvFjywyolMLl/2KxfTp0z3ncLMvkJGvvh3uhYxCMN2IakxBXiywXhph1Bxh1JxkzgsZNRcYNRMYNSeZ80JGzYVIo7Yb9OjRQ9Y72rBFPAv0ZN26dWp6PLmnZvpb4Vw4AchusLKysn+DdnDJ4/+VlZVtdtDSpUsDuIxkY8eObSNBJ2lTA88CniV6uo4fP+7caipcbwbl5eVyvPo8yC2Xl/IEcMnDVK5trQW5F5g0adJcYP2MfbI/941ocjqWcrqnptHmAS9GzRFGzUnmvJBRc4FRM4FRc5I5L2TUXGDUTGDUnGTOCxk1FyKN2reAWm+94TURzwI9USf2ONfeB0x/K7hLJjgrLi7ObrDly5e3g0suCchLadSBSmPKlCkBXL4yubmlpjiQtRcUFEAw5dixYzJ4UK4tZTbp9fPmzQtZWFNTk/PoQ3kQD8dHGe6pPio3+XLLj+de0WriqrQflqyUlpb+Cqi5CH4NtAszYJXtXp77RsxNIehAOp3+IOjbt69M/jEZaDPzKXB8OQ3sMkYtB6MWrDBGjVHTZQE8jFqwwhi1do/a+fPnZXKpVGaK4Qt+82SZFOjKb4Fqf5w7nTN58YdVFsAlmy6V+TNcZnFAJ7WDS96uJ+N80AvyYEjPnj0L1LvyTAZKtZptRHmi6ZNAS8H8+fPVwhPgo0AtWLhwYcjC9uzZo0VNi1wtUJ/BDtsJ5CUxuW5E3o8FcOGYt9w+CYFC5iLYtm2bXDxYA3BaV2S9NVZYWCjPwJhuxAEDBrhHDSeDK8ADDzwgL/G0v+9SEo5v4FwYo5aDUQtWGKPGqOmyAC5GLVhhjBqjpssCuBi1YIUxau0etYaGBrXen4AAvxgwamfBg0Cd4q5fvz7ItwxY2NatW9X2kp5pVPMRRO+SezzqpF068sCBA9moPfroowFcnjJZrkqRf+R/1bDDHTt23ARybVh49OjRkIXV1dVJt1kvjch/rgVnzpyR13f27t1bm71ZFt5///0BXXIbskLN62dDqhkxYoRzFOWhnc2bNwfZiM7vQM3hukx+WgPsz1wxalYYtcCFMWqMmi4z/TijFrgwRo1R02WmH2fUAhfGqLV71EaOHCnrLi0tPWwyS5xJgY7IFQRVhzyyf+TIkSDfMmBhmes8Qj9w0HFKs7xdGzZskOaUqGXeAbNtwoQJ6rB1fVVVVQCXp+zOXDWyZhmFuG/fPpkiACf22d4oKSlZDPIqTHvlTRusra+60+R1qM4L5S60Nc/OWg1k2vXdnu4bcefOndkZKALQo0cPOTq/AlxljJqCUQtTGKOWgVFj1Bg1L9c1HbWLQL1n5J577jH9Ld8CHdkPVCnyvvlwMtOP9+zZU3YgGlPeZNdOriVLlmR3TnV1tfxl37lzZ/k/mQLKb1pjq8tTJqvT2lHiZm1OeRrJ712QJoWtA37tL6dR8jyN35hST9cLIPfwjzv4EjeCu4HXqxM9N6JMPn0rMIwZjpYT0Km+hTFqCkYtTGGMGqPGqDFqjBqjxqgFdr0Ko/ZjkMrcsNi7d6/pb5kUaOcpILXccccdkvBwMpOPPgfUm04ee+wx53PbvF0y9LC4uDjbkd26dVOzOAgyTjCYyyhqnXLDE3NBkDt7poNJ/VyyrdQbgpShUw756fjx45vASyBfl2xBeX29FmvrP/Ie0cbGRu0xOT+Xu+z3wDqWUqsP/0gUnwcbNmwwneGPUWtl1MIXxqgxaowao+bvYtQUjJqHi1FLetRkOjPITF8H4+g0kq1duzY7rg2xDu4KErWhQG3Q1atXt5NL3mgzfPhwrQ/lakU6nX4CBIi3yUaU2d+0E3vJNFy3g/Ly8iMBxgKYFHby5ElpupkAByrp0fWgoaFBLvs0NzdH6YqKgAf9SGSMWiujll9hjJqZjFFrZdTyK4xRM5MFiZrMMKae+JCRfGPGjFGv/g7oNJJVV1dn/0ru2rVrAEkbmclHBwP1Z/jcuXPb0fXkk09mTzPuvffehSDArApWl1Fhu3fv3gTkptShQ4cCijRZiN/sEC5GTWDUvF2MWgQuRk1g1LxdjFoELkZNYNS8XYxaBK5rNmpLgJqaWh5iHzJkyFnDcXtWp5HsoYceys4G3atXrwCSNjKTj8rjXUjAV4DfPZK8XXlzNbokqS5GTWDUvF2Jbf84XYyawKh5uxLb/nG6rtmoReVM7J6L05XYwuJ0MWpRypLqSmxhcboYtShlSXUltrA4XYxalLKkuhJbWJwuRi1KWVJdiS0sThejFqUsqa7EFhani1GLUpZUV2ILi9PFqEUpS6orsYXF6boqUSOEtCuMGiGxwKgREguMGiGxwKgREguMGiGxwKgREgv/A05KLukKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iago0MTU4CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagoxNSAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjIwNTEyMTQxOTE3KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNS4yLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNS4yKSA+PgplbmRvYmoKeHJlZgowIDE2CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA1OTIxIDAwMDAwIG4gCjAwMDAwMDA1OTYgMDAwMDAgbiAKMDAwMDAwMDYxNyAwMDAwMCBuIAowMDAwMDAwNjc3IDAwMDAwIG4gCjAwMDAwMDA2OTggMDAwMDAgbiAKMDAwMDAwMDcxOSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzMzcgMDAwMDAgbiAKMDAwMDAwMDU3NiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA1NTYgMDAwMDAgbiAKMDAwMDAwMDc1MSAwMDAwMCBuIAowMDAwMDA1OTAwIDAwMDAwIG4gCjAwMDAwMDU5ODEgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAxNSAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMTYgPj4Kc3RhcnR4cmVmCjYxMzgKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:17.931335\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgNjQzLjI5NzUgOTcuNTYgXQovUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicVY7LDoJADEX3/Yr7BfPCeS1VkolLdOEHTEaUgAZJ5PctLiAuTtLTtLeVdfk8cjmnA44XkpvliTQ6poVCx8zQSExLim0gt6uEid6y9JtEL6zjhlqrO9GNRnhhfjgTRFgmQyW0w7vgiifknoMnTu+YmSMT/n8ZeTEKE7Ac51Vr1sQ8QJ406hcaaugLxjwu9wplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0NAplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjM3ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHPz8/Ozs7Nzc3MzMzLy8vKysrJycnHx8fGxsbExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCurq6tra2srKyrq6upqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+dnZ2cnJybm5uampqZmZmYmJiWlpaUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX16enp4eHh3d3d2dnZ1dXV0dHRzc3NxcXFwcHBvb29ubm5tbW1ra2tqamppaWloaGhnZ2dmZmZlZWVjY2NiYmJhYWFgYGBfX19dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NRUVFQUFBOTk5NTU1MTExLS0tKSkpISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDg3NCAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTE2IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggODc0ID4+CnN0cmVhbQp4nO2d+2MUVxmGs5sSCyHBkIJQjEBFQrl4SUQKAjFGvEDVErUqYpREkSpIKWK1ooWkUiFQWhSrYKnGC0ioSoBoBKJIAcM9EDUIKpcoCc3+D75v96yZze7MObO7mWTH7/ml7czsPHNmvneamTlzJuNxQRA8IKO/N0AQ/j+QqAmCJ0jUBMETJGqC4AkSNUHwBImaIHhCJGohL3i8H2R+dfm2YV66PJVJ1NLU5duGeemSqKVS5leXbxvmpUuilkqZX12+bZiXLolaKmV+dfm2YV66JGqplPnV5duGeemSqKVS5leXbxvmpUuilkqZX12+bZjLX10DtbW1LcC9y62svb19NqiqqnLzK4laert82zCXv5KoOTkH8pFLG5dvG+byVxI1J+dAPnJp4/Jtw1z+SqLm5BzIRy5tXAO5YWNBIBDYCvrWdeTIkbeBYDD4Z+De5UZ2HZSWlsIVPHPmjJutlKilt2sgN0yiFk8mUUtT10BumEQtnkyilqaugdwwiVo8mUQtTV1uZXV1db8Bt27dSkBmunhHRwcXD4Z5BvSd6+9g6dKlgTBtwL3LVIZ2dZQDJWtqanLhkqiluUuiJlHTO31b/l66JGoSNb3Tt+Xv8ldnwc2bN7uBe5ep7CDIBKr8N27c6GYTXTbs/PnzyhMcMmTIr0Dfuf4NJk6cyNLPysraD9y7TGU8UipmU0FXV5cLl0St/10StWRcEjUD50Aufy9dErVkXBI1A+dALn8vXRK1ZFy+jxqXzc7O5sHLyMhYCL4JTp065cJpKlsJhgIcuBywbNmyw4C72IXM0BV68skn7wZo15vBd8GxY8dMf+zSxRqcMWMGd2J+fv468DJw5zKVqedOwXBFZlVXV5tupUVmuvicOXMiUcMOdCNy7/ooQKPo2rRpU2IuU9n9QEXtpyAxmURNIVFzkJkuLlFzkEnUFBI1B5np4hI1B5lETSFRc5CZLi5Rc5C5iRpL/zUg+D9YLHcB1M33gKHTSLZo0aLBIBgtA7wu3b17t3YFLlwXwbBhw6I9mYMGDXojqAcpdBFczU8MV39A/YM+7MSTwNBlKhsHVMMmg9u3b5tupUVmuvjo0aNVrAN1dXVuRO5dOEnlw8MiuXr1amIuU9l4gP+zZKAk/gQSk0nUQhI1rcx0cYmag0yiFpKoaWWmi0vUHGTGUdu/fz//1p8GNmzY8DVgqUowBBw4cMDEqZX9EeTm5qJhg2aBUaNGZUaD3at9kGLo4rsR7wFY60yAC8J7gPKwbni12NDQkAoXeQvganGCKgLTp0/nuUrJ3gCOHz9u4jKSrVixgmunD42ynA33gLKyMu0KXLh+BHCxSxcvozo7O01+lZgL2brKqMH1DuBSFHIdNZ7eef6YPXu2e5dETSFRM5CZuCRqGplETaKml5m4JGoamURNoqaXmbgkahqZUdRuAFQIo9bQU3WNjY1fAu8CqjLz8vIcBwkzbeAPgdqLl8C8efNqQHV19TsBA4jymQRMGqhzfRlw21GKraC7u/tv4LPActtn9erVqXCFtm3b9lrAK+zy8vJXACbuBE8D9QCspKTExGVUJTggkZtKixcvVj7yfvDAAw/sBaloGLtxFgBUI9tmsmkJu0jkfIXDY3I7ztZlJFu7dm3kXk9FRYV7l0RNIVFLQcMkagYyiZpELfmGSdQMZEZR2wUg+xyInYk/yjvXrFkzOvxM5ZRTHy2TBn4VsLqnTJnyBxC7wPeBuv5wHLnIpGG4TmOXLybq4Ycfjp3/EcCZ48ePvwmScpHhw4dzuyeAy5cvW2awVrET7wqfQzqdLnRMqwSnh9WqQnipEd2djY9HR4wY8QPguBJD1y8BT4Hw3Qd0iyflCl25cmUkYMNwbZiUSyvj2Mm4gg+ocwjO+4nLJGoSNQckahI1iZrOJVHTuyRqEjWJmj2+jdodwIrDHuT9gvgLnT179r0AhXIa6Jz2MqyH/QPZqpqamvjL8A0UVULnzp2zXZX2yLGeKysrWfqI9RSkLnaZY0C9PHHbqfegyYH7GUC76OPpJHaBhx56SD1euxydw7guR9lfAe+/qFsibwVqDkrnFyAjvCX0tba22q/IsCJZIGo/vQ+oqSyd2traReALYN++fY5jO5hGraWlJXKbAk1UEzl83BZQUVGxGfweOK3DNGo7QKAHVXAnTpzgSYpvWpm8PiZRk6hJ1HQuiZpETaImUZOo2SBRk6g5O7cBHpFZs2bZr6y5uVkd1eSitnPnTrrmg57Hrb3g823VPfHChQv226Rz4cjvU+uJ/0pa+E7FGlX+/wAJuwgfv2M97NSJC/sr0TNZGrm5ucr1F6BzOcoOgGAP7OOp5jQ2NkamqsY79iU1rEiL64NATV0BVI2qmZ8AybpCBQUFkaiNGTOmC+zdu5dai4s7Mu5NvGiXVjYXqNXy7IRymA7y8vIi4Rs6dGgxeOSRR+xXIlGTqEnUdC6JmkRNoua/qL0ecJ3Lly+Pv8AtMGfOHDYuOzv7nNP1k7aBKHnuKL6iE3f+f8DXAVy8/nB80V3n+jTIfHWYgRnxF8BFzAjADZo8ebLJsy77jWloaFDlsTHu+MLsA6k6ko4cOfKfQOdyrBIcrOWW8v82UHMee+yxXlFz7AZpWJHqmRpZCjgJVWG5WlQ+7k/b86Np1MaNGxcpdJQ9n3xhzZYLqogP1XgeOLm0Mh6tgAHROzm+TKImUZOo2SJRk6hJ1CRqErV4SNQkatqoce8MA7YL/ASoPomoF8cN1zawqKiI1XbJrq9ZNeACWVlZ3wEmMtv5nwLYSU/H3Wj2eETNsFHsl6gbPld74Orq6tR2c7noT1z+FuD6fkxm+N04XHqbuByrhDcJLGn6F1BzLOP9qWg7vWBoVP6nT59GVWerorugolRTU6MknwT8xOfcuXM5BeWSuItMmjQpUuE5OTn83vz8+fP5X/y/QklJCbvqcuQITCkDd+7csXVpZZ8HlkRNmDBhLEDY+XZeHsjPz+fweJjHTsDcGFuZRE2iJlGzRaImUZOoSdSckKhJ1CRqtjKugXGLnfPEE08UAo44gj1YC3Qbrm3go48+yqPPjzRGz2BnxJUrV7K5XMDpSWgvme38jwOsrtcy3d3dfP77JqBKkf+arCv0wgsvqNUNB9fDHS55k+f555/noVLPrvmvhi7HKslWHwtSqK6H/I6P6oWpUsBbTLYdW40aBg4dOqTuDASnTp3KKYzbtGnTuBWWZ7tLlizhcraDwJtGTQ0Ux0fKTU1NHDClqqpqLeC7kqqTJYfAwTIMw40bN2xdWlkuUDH7CsABi13mGaB2Kd/bs5VJ1CRqEjVbJGoSNYma/6Km/uoNrFu3LjKptbX1x8By1MCHgG7DtQ3csmULXex5du3aNU5heaAcLS99UIY/znUu7ZE7FC4Rnik4QB082wGOWkZ0w+aAZF2ho0ePcl3w8REUOwc2NzezPAI9z4GCZkP1mlTJPKAq5O2Akzo7O3nGVOcqynC+esW2p6lpw0KvdoDlNSZlDz74IKeop1GWATL4gAvnbA57bPvM0DRqlmu1zZs383wV06U0xM9DqgMb970d06jxGlq52Gc27jJ8YwnnTw7C0AhsZRI1iVoyDQtJ1CRqEjWJmkTNdgGJmkTN0UV8G7VvAX5lBjsT17VLPgNwrcnDhYrkQwb1IRXGQ7fh2ga2tLREHtEsWLDgY0D1rmM7CgoKeI+G6p73r7Qy2/kcwQDX7pEqHzx4sPrEPSkHqiIZwGRdhHtOrZ378957743c5FEiwNsUhi5H2W6gPkFzN+CTrfr6eo5IZ4n24sWLdS7T8lefQQ2qZ6KvA0VFRTd67kjwRTWofw2SdTFq6rTFUfd4UyJ6Pr/ryUOZk5Ojc2lllnMWT4MdHR29FkDQ1RmTn2d1lEnUJGpJNiwkUZOoSdQkao6ugRU1wj+xw0/PIpXCcZFwrcNRABgG7OJNus5Lhg1ktNVIyYQPNkpKSn4O0E4ORcypuFLUuYyOHMqCnc5Yg8pXWFjIP8o5IFK4dDI5DlMqXBzFoLi4OBKszFc/fnMP9uP9QE3FZU+zoctRxgdpfMsoGIWSRLagurpa5zItf45QoSSRQ7R+/XrOuX79Os8fLJKysrKUDBGNC8JAD3wJCuXAFnOErmXLlvHuArdg1apVOpdWxrHNVKy5yhdffFENG0cfR6koLS3lTFwF23+PXqIWkqjpXBI1iZpETaImUXNAoiZRk6g5CtmG7du3Y++tZx0iY2rGNwCPHKLhOKCaywbypY+DBw82gWvq+Rpob2+nix/07Orq0q3D+MiF19x+9OjRl4Eaw/jDIGj2sNCNq62t7Qg4AS5evMhnhpjImwhs2NatW124tDLGduzYsZaukMFw0fAf7GyJvZuyhoF3A8tNF5xG2GtQnTR5c+bZZ59NiQvF+AGgosa15+XlLQCsT3g4lUPIJTtqBOFIiPg/iyXa7BKLlVs6R5I9e/ZoGyZRC0nUkm9YSKKmbZhELSRRS75hIYmatmEStZBELfmGhSRq2oYZRc0W3j3gQbQduS6uM0HZjh076GK8XcgSc7Fugj0jrfWdC2cS3phhwTz11FMuXKYyfiRmbRjWn7otkupzSCj8LZji4uKAJW49d2P4DmAKXXw4zpoPxGM2cBzPPuRuJyKyvE8X1xVADrlrnUYulKg5IlGLkpksKlHTyCRqcZGoRclMFpWoaWTJRa0ScE9u2LDBZPHkolZYWEgXv4rnQubew3GM2eENriLQp67Qrl27IpdRJ0+edOFKQMaekYFwt8sq4EJmarh06dIXQT5QopkzZzJmHbG9B5N1sTULFy5U41bwKSIHxcb+dBwbOtplKuPFNft5qnypcwgfN7/00kvaX0vU7JCoxchMDRI1B5lELQaJWozM1CBRc5BJ1GKQqMXITA0SNQdZElFra2vji1eUpviRUCxnQG5uLl2rnPq2xcrcuzj2sNqZlk9g9o0r9Nxzz0VuHZh07AwlsxNbgEpA39/vcU8/uDyVSdRikKjFyNy7EkCiZotELXmXRK1fXRK1WCRqvV0StRS40iVqx48fV++qZdbU1LhwJhg1VZEVwIXMvet3gKKsrKz4XxZJoSvU86JX0JN7S5WVlfeBVuBClpjLJRI1WyRqybskav3qkqjFIlHr7ZKopcCVLlFraGhgRXIQtL7pvmeBQ57BxTHHdgEXMveuc4CNMhlCOVlXqL6+fhQoBe3t7S5cvi1/L10StVgkar1dvi1/L10StVgkar1dvi1/L10StVgkar1dvi1/L13pErXDhw9zeDW+O2/7KfG4zgRkHNWsvLzcNGYWmXtXAvi9SvzqkqjFIlHr7fJtw7x0SdRikaj1dvm2YV660iVqCTt9e+S8dPm2YV66JGqplPnV5duGeemSqKVS5leXbxvmpUuilkqZX12+bZiXLolaKmV+dfm2YV66JGqplPnV5duGeemSqKVS5leXbxvmpUuilkqZX12+bZiXrn6JmiAIfYpETRA8QaImCJ4gURMET5CoCYInSNQEwRMkaoLgCf8FJ+cUygplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjQxMzYKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMjA1MTIxNDE5MTgrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My41LjIsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My41LjIpID4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDU4ODQgMDAwMDAgbiAKMDAwMDAwMDU5NiAwMDAwMCBuIAowMDAwMDAwNjE3IDAwMDAwIG4gCjAwMDAwMDA2NzcgMDAwMDAgbiAKMDAwMDAwMDY5OCAwMDAwMCBuIAowMDAwMDAwNzE5IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDMzNyAwMDAwMCBuIAowMDAwMDAwNTc2IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU1NiAwMDAwMCBuIAowMDAwMDAwNzUxIDAwMDAwIG4gCjAwMDAwMDU4NjMgMDAwMDAgbiAKMDAwMDAwNTk0NCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDE1IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAxNiA+PgpzdGFydHhyZWYKNjEwMQolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:18.034217\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["pl.seed_everything(42)\n", "for i in range(2):\n", " interpolate(flow_dict[\"multiscale\"][\"model\"], exmp_imgs[2 * i], exmp_imgs[2 * i + 1])"]}, {"cell_type": "markdown", "id": "3f492396", "metadata": {"papermill": {"duration": 0.02286, "end_time": "2022-05-12T12:19:18.124072", "exception": false, "start_time": "2022-05-12T12:19:18.101212", "status": "completed"}, "tags": []}, "source": ["The interpolations of the multi-scale model result in more realistic digits\n", "(first row $7\\leftrightarrow 8\\leftrightarrow 6$, second row $9\\leftrightarrow 4\\leftrightarrow 6$),\n", "while the variational dequantization model focuses on local patterns that globally do not form a digit.\n", "For the multi-scale model, we actually did not do the \"true\" interpolation between the two images\n", "as we did not consider the variables that were split along the flow (they have been sampled randomly for all samples).\n", "However, as we will see in the next experiment, the early variables do not effect the overall image much."]}, {"cell_type": "markdown", "id": "702b8862", "metadata": {"papermill": {"duration": 0.022287, "end_time": "2022-05-12T12:19:18.168450", "exception": false, "start_time": "2022-05-12T12:19:18.146163", "status": "completed"}, "tags": []}, "source": ["### Visualization of latents in different levels of multi-scale\n", "\n", "In the following we will focus more on the multi-scale flow.\n", "We want to analyse what information is being stored in the variables split at early layers,\n", "and what information for the final variables.\n", "For this, we sample 8 images where each of them share the same final latent variables,\n", "but differ in the other part of the latent variables.\n", "Below we visualize three examples of this:"]}, {"cell_type": "code", "execution_count": 32, "id": "f5575fa2", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:18.215227Z", "iopub.status.busy": "2022-05-12T12:19:18.214742Z", "iopub.status.idle": "2022-05-12T12:19:18.457363Z", "shell.execute_reply": "2022-05-12T12:19:18.456683Z"}, "papermill": {"duration": 0.267682, "end_time": "2022-05-12T12:19:18.459179", "exception": false, "start_time": "2022-05-12T12:19:18.191497", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 44\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzQxLjY3NDgzODcwOTcgMTgwLjcyIF0KL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSIC9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTIgMCBSID4+CnN0cmVhbQp4nFWOSw7CMAxE9z7FnCDfKkmXQKWIZWHBAaJQiCioVKLXx61AhcWzPJbHHtnk1zXlQ9xidyS5qjSSRmE6KBRmgkZkOlKserKVFs5XwdYsb79SByW84Zla2wvRmQZ4YRas4TpvB69qD+2csAbPjBPukBv+MvKrwkx8PeI/2LD4HeYgH+v3cOoh9xrNAy219AYPKzF0CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQ4CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyNDUgKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/b29vX19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Ofn5+bm5uXl5eTk5OPj4+Li4uHh4eDg4N/f397e3t3d3dzc3Nvb29ra2tnZ2djY2NfX19bW1tXV1dTU1NPT09LS0tDQ0M/Pz87Ozs3NzczMzMrKysjIyMfHx8bGxsXFxcTExMPDw8LCwsHBwcDAwL+/v76+vr29vby8vLu7u7q6urm5ube3t7a2trW1tbS0tLOzs7KysrGxsbCwsK6urq2traysrKurq6qqqqmpqaioqKenp6ampqWlpaSkpKOjo6KioqGhoaCgoJ+fn56enp2dnZycnJubm5qampmZmZiYmJeXl5aWlpWVlZSUlJOTk5KSkpGRkZCQkI+Pj46Ojo2NjYyMjIuLi4qKiomJiYiIiIeHh4aGhoWFhYSEhIODg4KCgoGBgYCAgH9/f35+fn19fXx8fHt7e3p6enl5eXh4eHV1dXR0dHNzc3JycnFxcXBwcG9vb25ubm1tbWxsbGtra2pqamlpaWdnZ2ZmZmVlZWRkZGNjY2JiYmFhYWBgYF9fX11dXVxcXFxcXFtbW1paWllZWVhYWFdXV1ZWVlVVVVRUVFNTU1JSUlFRUVBQUE9PT05OTk1NTUxMTEtLS0pKSklJSUhISEdHR0ZGRkVFRURERENDQ0JCQkFBQUBAQD8/Pz4+Pj09PTw8PDs7Ozo6Ojk5OTg4ODc3NzY2NjU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLi0tLSwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx4eHh0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBISEhERERAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNDU1IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCAyMzEgL0xlbmd0aCAxNCAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NTUgPj4Kc3RyZWFtCnic7Z17nBVVAcd3F5YVdnMBl4esgPIyozbRoFwhE3vJCgW7ZCAEGI8ww3DzQZoCUShE8tIeBEngFmaSUUEUvUhTkEcp1IYGLEsF8RT8v98v7v00Oztn5sy9cyXO/r7/oHfmzvee+Q3MOWfOOZP3ReECeef7B4hEUI5uoBzdQDm6gXJ0A+XoBukc33yrkC8nPuXohk85uuFTjm74lKMbPuXohk85uuFTjm74lKMbPuXohk85uuFTjm74lKMbPuXohk85uuFTjm74lKMbPuXohk85uuHLOMcTYPny5b8DPwcjwPjx4629sX2nwaFDh/4OXgSLFi2y+16mvrNnz74CasFd4NwnZ98AufEBlmsyuAEsW7bM7kvKMRzlaOWN7VOOoT7laMD1HM+AH4Ni0KpVq0fAMPAEaGho+Dc4CSK9sXyrQU9QWFh4B/gAWLJkCbdwj3+A5Hy8Xr4PunbtWgDen2LPnj3c8jo4BZLzHQdPAfhQvMIqwPLhr0i6fKk/wn3K0YdyVI6Z+JSjcvT4Yue4EiC/VleCrVu38uZ/FHgqANVgOwj1WvseB/ngIlBXV+fbynPbFhh1sX0LAAPs0aPHbsCP/gSOHDnC/2Rxp4Nt27Yl5LsN8HwWFRUhu+UBe+BvyrZIn3L0oRyVYya+tzxHFmHFihX8R3wG2AuCd+Q/70w71Bvt4+1hzpw5TOlh8DII/lHlYO3atYbDWPv+Be69917e93kTRtvRsOP1YPHixdn6cFe/E5SBD4GNGzcadhwyZAhv2OE+5ZhGOfpQjsrRg3JMkXCObBYOHjx4HjDv9Dy4FJhvzNa+5wDqNmxb7QPBO7Fl2QbU19cbDmPtewagfvM1YN7p64C1EnMT0tqHSg2PNAaYd/oWaN269Usg3Kcc0yhHD8pROfpQjm/mJMefgXNnzHTK0FZ+O/g4MB/G2jcNlJSUsP8xeAdu6Q7YvZuAbxJA+fhANXgH9quy92M0SMA3depUJNR6HTD7LgFDhw41nwTl6Ec5plGOcXznK0d2OqKchnsVh3nceOONReBRYD6Mta8StG/f3vCU6PTp0zyfzLEBJODrBTp16rQfBGxF824u4B2Nj+gS8FVVVV0DDL8+1Y97GXjttddCDqMcfShHohzj+pQjUY5NffY5suMxPz//BeDbwsEcXwHY4XYQfhhrHx844tL5EfBtaQQzZ84sBeGtr1g+ZNgJ5fs28G1h+W666Sb+oJEgIR+uUT5XTT/k9MCBKuPHj8cO7ccCK59yTKMclWMmPuWoHJv77HPkKUO7lXd5Vmo8W9aAPICyho959HijfZ8CyPFzwLflIYAzcDWIPoy1j48VcVSWj081PVtuBajf9AW+Ldn4ysvLWWsaBHxb2LGBM30dsJ6HoBzTKEflmIlPOSrH5j77HJcCtLp5b+ajOg5aPXz4MOfLsbuXFZzIMnq80Ts+DWpqauhjpKxswMdOeD5w7NevX/j4/Ni+b4IJEyawuwN/TEDznCfxfkDfwIEDz4QP0I/rq62tvRmwfHeDQ4cOsYh8OsAhUJWVlbF8yjGNclSOmfjOV45kwYIFJYAtKZa1W7duLDb/md8MrA4Ry7d69Wq2oi4HbLmda8Hm838PHjxod4hYPlwZQwBv9vAUeNiyZUvyPjaEhwKeQNwMO4L8FDt27IjlU45elKNyzMCnHJWjzxd73hxbUFyggzd/+C4GrOcMB8nWc1Lwbv9HkPLxMmI5cRVxzlzyPsJnkKkceYpZvnvuucfuu5n4+JegqKiIPk6DYFFnzJgR3Vb1+JRjAMrR2mvtU45WPuUYQEvIkfCpGE8mqh5su3LIZwewZMkSa28sH+sCLGRVVRVHcj4JcPn0AIbRNFn6ZgO0xTk4ZhXgyB3871dB9Hcz8bGCg+K9G+wCCwF8nHFu7VOOAShHa28sn3KM9CnHAFpKjmyIc3ac56MpALfqZ4GV19rFzmquZdUVeD5eunQpqwMPgGR9hONKy8rKvB9NnjyZ3dffA8n7ODO+ffv23o/mzZvH8gWMTgr2KccAlKNyvJBy5JD9gMT69+9vPU7P2nUMDASLmy6RcerUqfeA8NnDmfjIY2DDhg2+T3k1BYwyScDHKYk7d+70ftTY2FgBlkWup6sczShH5agcg1GOVr4k38MyaNAgrmdh5c3WdfbsWTa3rOsB2frOnDnDEYps2r0lPsA5+j8EVj7laIdyDPZm61KOTX3K0Y6WkiMrAMXFxa8CK2+2vvnz53MO+yvmZRmT9Y0ZM4aPWsPXtE/Qh5oPH33+Clj5lKMdyjHYm61POTb1JZEjZW8Dbdu25TqBVt5sfJyWUFpayhGR5hUEE/T9AHTo0KEz4Ou1cu7j61FwzbA/99fAyqcco1GOZm82PuXY3Kcco2kpOXJaG4fS9+zZk68ssfJm6uJUaE5Eho6XTvhaVgn4DgM25AoKCrigxy9BTn3kKgAfh1v+FFj5lGM4yjHcm6lLOQb7lGM4TufIMTPvAnV1dVx0kUMSOQzyiiuuiP5upj6Oz9m8eTPLSBdy7Ady4yOoSG0FHPOYmsrWH+TOV11dzfGjPJ8pH0/rH4CVTzkaUI5W3tg+5RjqU44GWkKOfKVX3rmaRuqcchp2/rRp06K/m4mPM+RYKnjYeZyaWx6wUmJCPgIP5zqkffjj9yA3Ps6QQ9m4ombe/4geK+vxKUcDytHKG8unHCN9SY7rsEK+nPiUoxs+5eiGTzm64VOObviUoxs+5eiGTzm64UvnKC5slKMbKEc3UI5uoBzdQDm6gXJ0A7Uf3fApRzd8ytENn3J0w6cc3fApRzd8ytENn3J0w6cc3fApRzd8ytENn3J0w6cc3fApRzd8ytENn3J0w6cc3fApRzd8ytENX8Y5ngJ/OQcXJRwM6uvrrb2xfZx7PXfu3M8Drr28ceNGu+9lcV6/AB4HM0FBQUFufcePH+dLZR4G14GSkpI6YO1TjmaUY7Q3tk85hvqUoxmnc2R+z4EyUFhYOAl8BHwHnDx5kiukhK+pH9fH/NYDrn7cunXrWwFfl7xw4cIzgMs/QnsyOR9ZCkpLS/kizVHgg2DRokX8JelflKzvCVBcXMw1XoaDarBu3Tq+oY2u0OIpxxCUo3K8UHKcByhEOUvvu+8+31aeWy4DZV5vNq6Pi7xzLasuYNasWb6tvKo+CjZt2mT4flzfcsAFrAYMGLAFBOzxGzB79uyEfOMAF7Lq2LFj+mVWPImpa+UpMGzYsEMg3KccfSjH9E9QjsqxOcox4Rx5LATH+s394EXQfCf+BC7LzrMf6o328TY/cuRINKRKWNEwr2DPas/KlSsNh4l1XtF4o+9jwPwCNK6nu2rVqgR8fJtkW3Az2L59u2GvioqKn4Bwn3L0ohxTKEfl6EM5pkg4xzsA6jc1gGkF7/QlwLV8zWsHW/t4CBxpDDDvxPcjdwPTp0837GHt46EKCwunAvNO3wUsX2NjY7a+9evX801kdwPzTr8AqARtNHcnK8eAQynHFMrx/z7H2wByfBkY9sANje/UuB2Y+yCtfZ8FKOcmY9OQb/W4DNBpvptZ+3ApTEf5thtvVEcBr5lpwNzFau2bOHEi3/D8EgjegX3Hl4Pq6uqQLl3l6EM5plGOcXzKMYVy9Pjsc3wS5Ofn/xYEbOWrWW655Rb+MnPb0eON9l0CSktLDdcNCjYb8B2Q7PHN0pd+YRfKtx8E7LF3797xgK09voYqSx/p06dPb7ANBGxFqe8E9O3ZsyfkMMrRg3JMoxzj+IhyJMqxqc8+R76ut6CggGNzfFs4IIfHgJOvSg4/jLWvHYAvoAmMRnjjlClTLgbsmUzAx3oEX38M3+vAt/UImDRp0kWAPa8J+AiuQPYobAC+LXwnc01NDTP8BLDyKUeiHIlyjOsjylE5NvfZ50gh2sk4gVM8n3LA6tOAQ2i6d+9eHzk3wNr3ToCDBuzMCgB+Cdv/4WNJY/neB1DP4XMAVns8Wx4D+CV3AVbnEvKh2sgcWZh/As8WDtmBjz32HH9k5VOOaZSjcszEd75yfB5ce+21HBM4FuwDBw4cuAbwIw7maFr6cG/0jl8GI0aM4AXCuy4nduE+9RDgeH38kjeS9X0DVFZW0sdJDrhAULwDywAbxRUVFdHXTCzf2rVr2RNN3xKQmuDA8vGxZK9evcxPeQN8yjGNclSOmfiUo3Js7os3LyDVw8hezZ6gpKSEP4PevwKrQ8Ty7dq1qxxwHgJbi3369OE1Q9/BgwftDhHLhzN3FWChkB+nItDHHHfs2JG8j080rwQsUFVVFYvJqs87QENDQyyfcvSiHJVjBj7lqBx9vozW6+DMMbpRXLrZ8/oZEMdr7WLt4lWQKiSn69E3Y8YMu+/H9bH29AxAdsyQly3VtbW1ufGxNoOLZEfBORgrvcOHD49Vj1OOPpSjcszEpxztvq8czXBOAqXjxo1j7wCnYnM63Zo1a6y91i7meD3gdTN27Fg+6+T6Uu3atePsveR9BI3+ChTvvYCnmTM/2rRp8wjIjY+LyUCAa6WWzx142SJOPlsNX1PG41OOASjH6O8qx+acrxxvAFyVw/PRBIAzzUeRVt5YPo7G5wh5z0fz589nsy5gVEQCPraNO3fu7P1o7ty5HNexFSTvuxT4fA8++CCrAY8CK59yDEA5KkflaEY5RvoyyvFv4GDzHs7Ro0fPNi/01NRr7WI9h404Vqg8Hx87dqwPmAiS9ZE/g3379nk/QluOPb3PguR9bBT7Km1Hjx7lnEDORbDyKccAlKNyVI7BKEcrX5LvYendu7dhicTm3gR07B1I/roxcPz48QEgN/WcYOjjZEUrn3K0QzkGexPQKUePL6kc2flYXFxs3d7J1tfQ0MDVAxOcFxDOrFmz2KNsmD2cvG/37t1sPwavxhjgU452KMdgb7Y+5djUpxztaAk5cpVXDoWEN3ytDo83Gx9f/NSjRw+2r3wzznLj2wlSw1f4aDDnvhdAUVERJ2S/Yl6oq6lPOUajHM3ebHzKsblPOUbTEnLkeJKOIDVsz7wavM+bqY8TrT8J8vLyuoME53mbeQDAx+GPAat5JO/jNDpcM3zeyaFBVj7lGI1yNHsz9SnHYJ9yjMbZHDmok3f9/fv3812++Sn69u3LWK28sXyEE8pOnDhxNUjpOJU9+nuZ+HixlJeXc+muD4OUj6NocuPjDAAUhtMRuGAWZSgtTmdf634O5RiAclSOF1KOvCuxwZi6KfLGwXlCR44cif5uJr4G8GlQVlbGy4eRdunSZTXIjY/zn9A4XQE4mIPnFfcq87tlsvWx1Y3GYiq//zJq1CjOarP2KccAlKO1N5ZPOUb6lGMALSXHbJAvJz7l6IZPObrhU45u+JSjGz7l6IZPObrhU45u+NI5igsb5egGytENlKMbKEc3UI5uoBzd4D+OIW7MCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKNDA0NgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxOCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNTgyOSAwMDAwMCBuIAowMDAwMDAwNjA3IDAwMDAwIG4gCjAwMDAwMDA2MjggMDAwMDAgbiAKMDAwMDAwMDY4OCAwMDAwMCBuIAowMDAwMDAwNzA5IDAwMDAwIG4gCjAwMDAwMDA3MzAgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQ0IDAwMDAwIG4gCjAwMDAwMDA1ODcgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNTY3IDAwMDAwIG4gCjAwMDAwMDA3NjIgMDAwMDAgbiAKMDAwMDAwNTgwOCAwMDAwMCBuIAowMDAwMDA1ODg5IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo2MDQ2CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:18.261454\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzQxLjY3NDgzODcwOTcgMTgwLjcyIF0KL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSIC9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTIgMCBSID4+CnN0cmVhbQp4nFWOSw7CMAxE9z7FnCDfKkmXQKWIZWHBAaJQiCioVKLXx61AhcWzPJbHHtnk1zXlQ9xidyS5qjSSRmE6KBRmgkZkOlKserKVFs5XwdYsb79SByW84Zla2wvRmQZ4YRas4TpvB69qD+2csAbPjBPukBv+MvKrwkx8PeI/2LD4HeYgH+v3cOoh9xrNAy219AYPKzF0CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQ4CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyMTggKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/b29vX19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Ofn5+Xl5eTk5OLi4uHh4eDg4N/f397e3tzc3Nvb29nZ2dfX19XV1dTU1NPT09LS0tHR0dDQ0M/Pz87Ozs3NzczMzMvLy8rKysnJycXFxcTExMPDw8LCwsHBwb+/v729vby8vLu7u7q6urm5ubi4uLe3t7a2trW1tbS0tLOzs7KysrCwsK6urq2traurq6qqqqmpqaioqKWlpaSkpKKioqGhoaCgoJ6enp2dnZycnJubm5qampmZmZiYmJeXl5aWlpWVlZSUlJOTk5KSkpGRkZCQkI+Pj46Ojo2NjYyMjIuLi4iIiIeHh4aGhoWFhYSEhIODg4KCgoGBgYCAgH9/f35+fn19fXx8fHt7e3p6enl5eXh4eHd3d3Z2dnV1dXR0dHJycnFxcW9vb25ubm1tbWxsbGtra2lpaWhoaGdnZ2ZmZmRkZGNjY2JiYmFhYWBgYF9fX11dXVxcXFxcXFtbW1dXV1ZWVlVVVVRUVFJSUlFRUVBQUE9PT01NTUxMTEtLS0pKSklJSUhISEZGRkVFRUJCQkFBQUBAQD8/Pz4+Pj09PTw8PDs7Ozo6Ojk5OTg4ODc3NzU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLiwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx4eHh0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExERERAQEA8PD1xyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNDU1IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCAyMzEgL0xlbmd0aCAxNCAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NTUgPj4Kc3RyZWFtCnic7Z3/d1d1HcfH0Aau4WaklC0YQV+FEqMvkgnLBjWBLCoS+2JfFhHNDGr2zYhJKym0rIysTE3CStnUQbMvZBQRkTVybfLX9Hyez3bOzs59f+779V73enzd5+MXDx7f5+G9j4/e+75f3rfh08IDDc/2v4D4v6COPlBHH6ijD9TRB+rog6mO58pCvkJ86ujDp44+fOrow6eOPnzq6MOnjj586ujDp44+fOrow6eOPnzq6MOnjj586ujDp44+fLPquHbt2svAT8AzwOJN8q1bt+5C8BiIHjML31XgVaAs3+vAdeAHwOpTxzDqmO9N8qlj0KeOYSrR8VbQ2NjI7TwKxkBUylTfB0BTU9P7wL9B9LhU36FDh7B5ja8FfwaF+44fP87/Jp4PbgFWnzoGUMcor9mnjnV96higKh1fBrCdPwKmcam+dgDfL0EpPpxTzQX3glJ8q1evpu9GkOJTxwDqGOU1+9Sxrk8dA1Sh4xmwDGC/cu76d2D1mnyj4BUAvveCU6BQ30nQ3NzM/XoxGAGF+u4E8+fPPx/sB2eB1aeOGahjtNfkU8dcX1JHyuaAefPm8S+PAKvX5Psr4GQO+7UBmI6RKT7MHQ/BdR7g9g2BQn2PgtrmzaXvCWD1qWMG6hjtNfnUMdenjhlUoeNfAOY6q3mCs2XLllI68uYmT242b95MX8r9OZPvMFi1atXUfuVuLtQ3AQ4cOMB7q/T9Hlh96piBOkZ7o/95oo65PnXMoAodp7NixQp67wdWb5Kvq6uLPtM13VTf+Pj4tYC+B0HhPtAJ6LsPWH3qGEAdo7xJPnUM+mbVcfHixZzW3QCs3iRfb29vo/Ve2Sx8XwLcr3yuswzf+wF9fN7S6lPHMOqY703yqWPQp45hKtNx48aN3K+fAFZvkm/Pnj303QTK8H0H0Mfbg2X4vgvo2wmsPnUMo4753iSfOgZ96himEh15zbOtra208w4+JLN06VL6eP5RuA9cD+h7ABTuw+7k/U5e0x0EVp86hlHHfK/Zp451feoYphIdjwPMWV8M/gasXrOPD8k0NDRcAkp5jw0/G87JXwieBIX7Tp48Sd9FgLd4rT51DKCOUV6zTx3r+pI7fgbg2LEBmMal+vhCALbzZlCKb9euXdyvO0Apvq1bt9JnmotP86ljAHWM8pp96ljXp44BqtDxt6C59mLZHSDFaxrza8AX2DC/Ms3lUn3DYNGiRZzP/RAU7nsYLFy48HngHpDiU8cM1DHaaxqjjrk+dcygKh2/DHjhceXKlb8DKV7TGB78eRKwfPnyp0HhPnZsaGj4GDCNS/XxxiN8XLIret3MGT51zEAdo72mMeqY61PHDKrS8UowD1gbTvOaxlwNWsCRI0dK8V0O2tvbHwel+N4CWltb/wBSfeqYgTpGe01j1DHXZ+74ELgAfBbg/+X8X/ulYB2weKN9PGBwTUIuwI4/fhgwKQ7NceOtvq2TFzr7+vqmht8FRkdHi/Hx3WdOVPv7+/lachfAVHLh2NiYyaeOM1BHdUzxqWPceHXMhsG4X7kmInYx373ms3pvAxZvtA/b179ycqI6Pj7Oy6ycukYvN2v1cYPawOR13A5AH1IW47sG8DyntjufeSXgWvPR02R1zEYd1THFp45x49Uxm9sAH8z/Amiswftm0eOtPj4AxEUeeeMRm8zfDM+roi9BWH1PAQpwosFXEV4KuI3btm0rxvcN8Btw+vTpKwBPedgxerw6ZqOOJm/0P6+OUb7k5x758Pr27du5X5nW6jX7/gN6enrouxsU7jtXu6zLuRy/N2U9Xpll+N18DnDqymWKo8epYy7qmO81+9Sxrk8dw1SiYyryFeJTRx8+dfThU0cfPnX04VNHHz519OFTRx8+dfThU0cfPnX04VNHHz519OFTRx8+dfThU0cfPnX04VNHHz519OGb6iie26ijD9TRB+roA3X0gTr6QB19oPmjD586+vCpow+fOvrwqaMPnzr68KmjD586+vCpow+fOvrwqaMPnzr68KmjD586+vCpow/frDp2dnZyGdi7oheZPZe+nVxkdtOmTVzC1/RNrVTfxMTEu8FrAD93WbjvXG2J4heA7wOrTx0DqGOU1+xSx7o+dQxQlY4fB42NjTcA0zcLU31cBBG+D4H/gsJ9vb29XFtyI+BSk4X7enp6zgdvAtcDq08dA6hjlNfsU8e6PnUMUJWObwW175PEn+NM85p97wTYrz8HpfiWLVvG7wTcAUrxbdiwgb+bQZDiU8cA6hjlNfvUsa4vqSMnVE1gzpw57wC/AFavyTcK5gNsJz9BPxb9ccREHw+I7e3tXPP9o+BeUKjvX6ClpYW/G37qyvSdZHUMo47RXpNPHXN96phBVTrym778OFlzczM/IfZjYPWafHeCuTW4b1O20+R7F5j0cfv4DehCffcANJzypexPdcxAHaO9Jp865vrUMYOqdDwBHgRr1qzhsfkRYPWafDyvejloa2vjdj4OCvNNAJzjtC9YsKAVcPu+BQrzkVsBfjpT3wz9B7D61HEG6mjymnzqmOub1XMd3d3dPF4VPr+aYv369fSZrkHOwsdnVuj7KSjD9wZA3/3A6lPHMOqY703yqWPQp45hKtMR20fvzcDqTfLt3buXvhtBGb4hUObv9DJA39eA1aeOYdQx35vkU8egTx3DVKbjvn37OG8trePAwECZv5uvAm7f20EZvmsBfbcAq08dw6hjvjfJp45BnzqGqURHvgzQ0dFB7xeB1Wv2kZaWFj47fx8o3Hf27Nmp69ZlbR99/J3eDqw+dQygjlFes4+oY9CX3JEvBcN5ISj8OQsyAuC7FPwJFO47ceIE9ynvQf4RFO47evQoffhPo+NJYPWpYwB1jPKafepY16eOAarS8c0A3m8C07hU33UAvqtB4e8FkB07dnC/3gRM41J9O3fu5DlVL0jxqWMAdYzymn3qWNenjgGq0PE4uKDGwyDFaxrzALgYNDU1fQ8U7nsMtLS08NXAn4HCfTyHw/kUfzffBik+dcxAHaO9pjHqmOtTxwyq0pHxeFDev3//0yDFaxrDRbO4kbt37zaNS/X1A2zeVcC0Vleqjy/mw8dzONPaWdN86piBOkZ7TWPUMdeX1PFysAhYrgPO8JrG8IH5F4Fjx46V4rsS4HD8BCjF90aA46P5XGOaTx0zUMdor2mMOub61DGDKnTk2lKcWG0Dk3/rnyD6fMDqGwfngfUAf3wKLAenTp2KG2/1PQo4Me7r6+M2ca0Ovj83PDxcjO9XgOdw3d3d/OMe8BFw+PBhk08dZ6CO6pjiU8e48eqYzXvAq8Hkx1BuA9zsu4HFG+3jDUCeV01+DOWDgOs+RS/DbPVxnecrwOS7cq8HvOQxMDBQjI8LWC4GyMklX7hMGH3Rn2JRx2zUUR1TfM9WRxwohi8C7LlkyRI66eZxy+KN9vHYwUWsWms3dujj4XlkZCRuvNXHh0YeAmfOnPk64HsI/N0cPHiwGN9XwCUAKbsA9yV9Q0NDJp86zkAd1THFp45x49WxPlzzsbOzsxl8HkSPS/VxEURMIRcAnhoU7sO09ZOA53Cm8am+wcHBlwA+vvIpYPWpYwB1jPKafepY16eOAarSMRX5CvGpow+fOvrwqaMPnzr68KmjD586+vCpow+fOvrwqaMPnzr68KmjD586+vCpow+fOvrwqaMPnzr68KmjD586+vBNdRTPbdTRB+roA3X0gTr6QB19oI4++B+g15x0CmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMjc5NQplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxOCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNDQ5NyAwMDAwMCBuIAowMDAwMDAwNjA3IDAwMDAwIG4gCjAwMDAwMDA2MjggMDAwMDAgbiAKMDAwMDAwMDY4OCAwMDAwMCBuIAowMDAwMDAwNzA5IDAwMDAwIG4gCjAwMDAwMDA3MzAgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQ0IDAwMDAwIG4gCjAwMDAwMDA1ODcgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNTY3IDAwMDAwIG4gCjAwMDAwMDA3NjIgMDAwMDAgbiAKMDAwMDAwNDQ3NiAwMDAwMCBuIAowMDAwMDA0NTU3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo0NzE0CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:18.340864\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUiAvTWVkaWFCb3ggWyAwIDAgMzQxLjY3NDgzODcwOTcgMTgwLjcyIF0KL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSIC9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTIgMCBSID4+CnN0cmVhbQp4nFWOSw7CMAxE9z7FnCDfKkmXQKWIZWHBAaJQiCioVKLXx61AhcWzPJbHHtnk1zXlQ9xidyS5qjSSRmE6KBRmgkZkOlKserKVFs5XwdYsb79SByW84Zla2wvRmQZ4YRas4TpvB69qD+2csAbPjBPukBv+MvKrwkx8PeI/2LD4HeYgH+v3cOoh9xrNAy219AYPKzF0CmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTQ4CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyNDggKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/b29vX19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Ofn5+bm5uXl5eTk5OPj4+Li4uHh4eDg4N/f397e3t3d3dzc3Nvb29ra2tnZ2djY2NfX19bW1tXV1dTU1NPT09LS0tHR0dDQ0M/Pz87Ozs3NzczMzMvLy8rKysnJycjIyMfHx8bGxsXFxcTExMPDw8LCwsHBwcDAwL+/v76+vr29vby8vLu7u7q6urm5ubi4uLe3t7a2trW1tbS0tLOzs7KysrGxsbCwsK6urq2traysrKurq6qqqqmpqaioqKenp6ampqWlpaSkpKOjo6KioqGhoaCgoJ+fn56enp2dnZycnJubm5qampmZmZeXl5aWlpWVlZSUlJOTk5KSkpGRkZCQkI+Pj46Ojo2NjYyMjIuLi4qKiomJiYiIiIeHh4aGhoWFhYSEhIODg4KCgoGBgYCAgH9/f35+fn19fXt7e3p6enl5eXh4eHd3d3Z2dnV1dXNzc3JycnFxcXBwcG9vb21tbWxsbGtra2pqamlpaWhoaGdnZ2ZmZmVlZWRkZGNjY2JiYmFhYWBgYF9fX15eXl1dXVxcXFxcXFtbW1paWllZWVhYWFdXV1ZWVlRUVFNTU1JSUlFRUVBQUE9PT05OTk1NTUxMTEtLS0pKSklJSUhISEdHR0ZGRkVFRURERENDQ0JCQkFBQUBAQD8/Pz4+Pj09PTw8PDs7Ozo6Ojg4ODc3NzY2NjU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLi0tLSwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx4eHh0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExISEhERERAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNDU1IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCAyMzEgL0xlbmd0aCAxNCAwIFIgL1N1YnR5cGUgL0ltYWdlCi9UeXBlIC9YT2JqZWN0IC9XaWR0aCA0NTUgPj4Kc3RyZWFtCnic7Z17nBZVAYb5dqlcSbSC2GLNwDAWEAgEFqRkwRBtISoLNhIEUjRAsSjICsGEgjBzKRIKuWbhZYEQS4yU2gQzijIwLOUiIeKF1mXNP3tfd+bH2e8yc2a++SDPvs8/wLdn5/nOvPNjzpw5l1bfEi7Q6nR/AZEIytENlKMbKEc3UI5uoBzdwM/xjVOFfAXxKUc3fMrRDZ9ydMOnHN3wKUc3fMrRDZ9ydMOnHN3wKUc3fMrRDZ9ydMOnHN3wKUc3fMrRDZ9ydMOnHN3wKUc3fMrRDV8SOR4Bo0FNTc2vgJU3H98rYOHChfPBg+AgKKjvRbBnz54RYCL4DyiojzQ2Nj4EVoETJ04EFFSO1ijH3N58fMox06ccw3E+xy1btnwVdAJFIJVK8Y8LQKg3lg8XyQxAQZPuTd/1oCC++8BUUA6KTvJ5UBAfg1sE2oDWrVv79Zs9e3bALynHUJSjcozjU47KMXKOr4PfAd7127ZtW9QcuvEVWm8GgV5r33/BRjAGlJWVFWVyFsAXCq6nte/n4BaAa7ME+JUyfDzNO3bsSMh3HPwS4Ho5ExieYo/27duzJRnsU45pKEflGMd3WnLcv39/NeDRPdnbwDlgCqirq7sdMMd7QQL1fPrpp2cBHtHwnQ3GjRu3FfD2zNOc87KJ5tu3b9/lwDiZVF8IRo4c+QTgoejbtm1bAr4XwHiQ5vsMwFl+AFwH8Cnv1ME+5WiiHJVjDJ9yVI5pPvsc2atZVVVlOHnHvxWwKWIU3ANYOP96vtG3b196/IYGfDcCowD7OtnneejQoeB6hrteBRUVFUb1mN9NwKgfLtW6y8C8efPy9eGgPBovTM/XDrDTuBF4hdh/PGDAgIdBsE85+ihH5RjHpxyNAsoxco40eEK62R6ora0N/73s3vCC3wRee4o3/8+CDRs2FM63AhiNDVSPDanC+ZYuXepfn2w3jhkzhv2rcX3K0Uc5xvGGF1SOUXx2ObJTdTiAtxT8CEQWNvcGF+Id6WaAOuLBbeSjoKA+dnJ2BajfuSA4QOP2FddHrrzySubYFvCpO/wXAn3KkSjHuN7gQsrRGuVo0FJyfB54D3KTQZYSPPPLly+fC/4JQr3BPj6CppqYALIXWgx6g9rA5paV7xjw6scBnNkLLQCXgsMgT9/9wHsungSyF1oCPgxwLgIOpRwNlKOPcoziU44GytHw2eW4HXjPyDNB2k+fASuBV2IfCPUG+9jx6B1tduawv3qAyr0dnAE4miZPH6cWeA/lXwdpP/0zGDFiBHsHOoDcfcfWPqN+3wNpPz0BVqxYwd6BzuDAgQMBh1KOBsqRKMeoPuVIlGOmzy5HfvVBAF72A/QEvcDw4cM/B4YA71ux3RF8KCsf3wUOAF6fdXtwEbjiiivYPX8+8EbsBM9BsPa9BiqaXj6yfnSNHTuWEwD4cvMS4NWPOSbg42tT76S9H3wQtGvXjqe1O7gYeL4yYOVTjkQ5Ksc4vtOVI2F/Z/MB8kWpk3ifrAUJ1JPgRI5N8zT/K/kiSMg3HRj1S2UlvCc0yvk0XUUZ8NP1wMqnHH2Uo3KM41OOyjHTZ5/j38G0adNSmXX1h+136dLlJZBQPTk5r0+fPmm+VoAT5fAvTkfgl0rIx27hGTNm+D7jevHqNxBw6Y6EfE899RRH/af5yDsAPmGThx2/Vj7l6KMclWMcn3JUjpm+aPPm8LjMO+9XAB+O16xZcyd4H+BX6dq1a/ghIvlwcL5lXAdwkXSZMGHCXrAcoLrsHUjYd/z48bsA67h69WpOZ+ckAG+aIDsKkvUdAD8BvwWbN2/muZwHOCSpqGlagrVPOZooR+UYw6cclWOaL9/1Hnma/XZOeXm5tTeurwZ4zYLwPvkEfDcArynSFxTc933gXTf9gLVPOQajHIO9cX3KMbsvdo4c6Lho0SIOrvCfe/r372/tjewjS5YsKTrJbaBwPjwI89bFxau8HAvrA3MApyR6p5NTva19yjEHytHKG9lHlGNOn3LMQUvIkesTckyJ0S3YB9TV1Vl7I/keBxwBiPt/yutj7dat219AYXx/BKWlpX7d6MOzHIc/Ju/jlEROg8DDt9nNOmjQoF3A2qccs6Acrb2RfMox1Kccs+B8jgcPHuT0a47G93x8V8YVWYI3J8n02vlwZbCzMXUS9jlwaWTU3u4QkXw4c/5g3NTJJVG42uLevXuT93G6BdeTMQJku5GvJZ977rlIPuVoohyVYwzfaclxJ+jdu7d/QlnB8ePH892OnbC5N7zgI6Bnz56+j3WdOHHiY6AwPm7N1aNHD8M3GOReMD9fH9oYuMl3M84n35EFrJgb6FOOPsoxjje8oHKM4lOOPi0lx2UgdXIgICcg19fXR3Ia3vCCXwJeHc8DPwO2balYPqZmNKg2bdp0InjnxTx9w4YNM3SpTp06Hcq9GnCoTzn6KMc43vCCyjGKTzn6tJQcOYMcvncCruz0uvWTeFZveMGPA/j4ApB7saQtZp+8z1vvkYu9fxc0NDTE0dn7+vXrx/w4aJSb2OzevTsfn3L0UY5xvOEFlWMUn3L0aSk5ciRnRUVF+EwuK294wf6gsrIS7Y1Np8RXBUpKSvYEL6+YnK9jx45cxeWnIAGfcvRRjnG84QWVYxRfvvMC4nrlS9anHN3wKUc3fMrRDZ9ydMOnHN3wKUc3fMrRDZ+fo3hroxzdQDm6gXJ0A+XoBsrRDZSjG+j50Q2fcnTDpxzd8ClHN3zK0Q2fcnTDpxzd8ClHN3zK0Q2fcnTDpxzd8ClHN3zK0Q2fcnTDpxzd8ClHN3zK0Q2fcnTDpxzd8CWRI9dhmg1mzZq1AQSvrJGQb+7cuQ+BWtAICup7Hqxdu5ZVHAFeAAX1HQS7d+++EVwPjh49GlBYOVqjHHN78/Upx+Y+5RiO8zlu3br1E+ADwFhin8v5h3pj+TZu3DgdGFvA0DcVFMT3C4DrZS63Zj4VPq7I/DXQHRi+wNOpHMNRjsoxju+U5vgK+CGYBkpKSooyof8BEOi19v0LcL8pLuPXpk2bLD6C22RwPa193AP+FlBaWso9ZtLq5f+JKzghH9dcHNVEe5DlXBYXFwds5aUcs6MclWMcn3JUjoYvWo579+69HKTJOgKusrtq1aovA27x2z/nXsmRfM8+++yngeHi9i9ngqFDh+4APBQ/Xr9+fXA97Xx4MMWBh6b5zgE4zfeDbwB+lPMyjeTDk+FRrqFv+LiEP+t3zTXXcP9HLu6PT78Ngn3K0UQ5KscYPuWoHNN89jk2gB49epgBdu7c+TaQVvA1kPsw1j7uDFpdXW362rZty+slrSDPxOHDh/P1vQrwyG3o0NQpnQSMQi8BrmC8evXqfH04EvfwLPY2fYaP+TG4tIJ33HFHwKL+yjEN5ZhWUDn+X+e4HBj/iXN7je3bt4f/XnZveEHc8tYbvnFgy5YthfNxwy7Dh1MceW/LSL5169b5Fww3Ra6qqqrN/RQc6lOOPsoxjje8oHKM4lOOPi0lx7OAtx/TH4D3aeRtp6x8PKq3eXBP8DDIXpC9rwmN6zgfePtb3QnCfyEPH9twgwcPZobsVF0KshfkEI+dO3eG+5Sjj3IM/91Mb3Ah5eihHCP5SEvIcSugFPX8FMhRqqamphxw7GOoN9jHzdA8X2+QvdBawJT/BPL0cWSM57sUZC/0Y/AR8FeQp48jN5t0qUtAlhINDQ1o1W0ZBrZt2xZwKOVooBxNlKOtTzkaKEfDZ5cjGxreMyuTehEYP2Wr5AsAT7Mc0nIXCPUG+zggx+s8Xp3ZJX0C4Jzy25wBgg9l5bsX8HDFxcXcGzntp7yq5syZwwJsBQUMlrH1GX0qWa569thPmjTJ75GYPn16wKGUo4FyJMoxqu905VgPBgIc+b2AWc4AV1111SBwNvC+FX8afCgrH2vCI+OIfF7lwPxqMHr06BEeno9DLhLwcTCnN7vh3aAHGDVq1GWANyh2JXu+XiABHwd0NFWv6GOgAgwZMoT1ooD/9Hx8qLXyKUeiHJVjHJ9yVI7Zffb9q+x0LGo+Ot77q4G1N9zHM2gcOXUS7xOObLkIJOR7BKQyaAUMLae0JeRbvHhxKrNSfsW8v3JaopVPOfooR+UYx6cclWOmzz5HditOmTIlLUfie9HceRwkVM/fgKlTp6adV7+e+GsbsC248zGKj+8E58+fn+U69ebPsQX0KEjIt3///ptB7vPJF75PACufcvRRjsoxjk85KsdMX7R5c6gq3z/eAG4H99xzD5KdciHgV6msrAyYiBC1nqS+vn4N4KQOria1bNkytn46AFSX0/XCDxHJd+TIEc7euBXcfffdu8FhwCnt8H0SJOvjpcNXu7PAtddey1hvApx0WNQ0csfapxxNlKNyjOE7bTlmwtd1fEnGHMvLy629cX0/AJzwDF0nUHAfX6bShxx5uRbc9x3g3R85H9DapxyDUY7B3rg+5ZjdpxyDaRE5zpw5k89W/nNPTU2NtTeyi4OAFixYwAE5no5Dggrna2xs5NCgEuA94l0HCudDA3EJMOr3IWDtU445UI5W3sgu5RjoU445aAk5PgjeBYxuQTYCnnzySWtvJB97bTlOx/D16tUreD5APj5OE+jQoYPZ7Rk6/yCuj9fn30BZWZnp69ev33arafvKMTfK0dobyaccQ33RckR+/F+bCy95Nw2+PBoAngFWh4jke/nll1cBo4IcWMmuXVtdNN+uXbvYuckHYs/HTk72IR87dix533HAVev9Na0A749cWjd4EnKmTzmaKEflGMOnHJVjms8+x1+D7t27pwyqq6sfA3bC5t7wglyqvmPHjt718iaTJ0/m4rmF8d0HunTp4tcNp5YbyvweFMZXW1vLi9L3oXpsK7KJFcenHH2UYxxveEHlGMWnHH1aSo7nAs85EfwDWN/8M73hBbn1oee7GPwbBC60nK+P2y16Pq4CcuTIkchrdUXy4Rz6GV4AVq5cGdll+JSjj3KM4w0vqByj+JSjT0vJkR3jcHLtXo7vjCV9I0I9GR587HOYAwru+yiA7z1gESi4b+DAgcyQo/85fjVPn3L0UY5xvOEFlWMUn32OC8HVV19dB+I6DW94QY6oqKysDN7lPUEfJyBUVFTYvfVLwFdTU8ONnrlSbgI+5eijHON4wwsqxyg+5ejTUnJMCPkK4lOObviUoxs+5eiGTzm64VOObviUoxs+5eiGz89RvLVRjm6gHN1AObqBcnQD5egGytEN/geV6NZ9CmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMzkxNAplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIyMDUxMjE0MTkxOCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjUuMikgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNTcwNiAwMDAwMCBuIAowMDAwMDAwNjA3IDAwMDAwIG4gCjAwMDAwMDA2MjggMDAwMDAgbiAKMDAwMDAwMDY4OCAwMDAwMCBuIAowMDAwMDAwNzA5IDAwMDAwIG4gCjAwMDAwMDA3MzAgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzQ0IDAwMDAwIG4gCjAwMDAwMDA1ODcgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNTY3IDAwMDAwIG4gCjAwMDAwMDA3NjIgMDAwMDAgbiAKMDAwMDAwNTY4NSAwMDAwMCBuIAowMDAwMDA1NzY2IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo1OTIzCiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:18.417020\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["pl.seed_everything(44)\n", "for _ in range(3):\n", " z_init = flow_dict[\"multiscale\"][\"model\"].prior.sample(sample_shape=[1, 8, 7, 7])\n", " z_init = z_init.expand(8, -1, -1, -1)\n", " samples = flow_dict[\"multiscale\"][\"model\"].sample(img_shape=z_init.shape, z_init=z_init)\n", " show_imgs(samples.cpu())"]}, {"cell_type": "markdown", "id": "01e26db8", "metadata": {"papermill": {"duration": 0.026565, "end_time": "2022-05-12T12:19:18.513844", "exception": false, "start_time": "2022-05-12T12:19:18.487279", "status": "completed"}, "tags": []}, "source": ["We see that the early split variables indeed have a smaller effect on the image.\n", "Still, small differences can be spot when we look carefully at the borders of the digits.\n", "For instance, the hole at the top of the 8 changes for different samples although all of them represent the same coarse structure.\n", "This shows that the flow indeed learns to separate the higher-level\n", "information in the final variables, while the early split ones contain\n", "local noise patterns."]}, {"cell_type": "markdown", "id": "4fc30ffb", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.022768, "end_time": "2022-05-12T12:19:18.562088", "exception": false, "start_time": "2022-05-12T12:19:18.539320", "status": "completed"}, "tags": []}, "source": ["### Visualizing Dequantization\n", "\n", "As a final part of this notebook, we will look at the effect of variational dequantization.\n", "We have motivated variational dequantization by the issue of sharp edges/boarders being difficult to model,\n", "and a flow would rather prefer smooth, prior-like distributions.\n", "To check how what noise distribution $q(u|x)$ the flows in the\n", "variational dequantization module have learned, we can plot a histogram\n", "of output values from the dequantization and variational dequantization\n", "module."]}, {"cell_type": "code", "execution_count": 33, "id": "d81503b3", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:18.615278Z", "iopub.status.busy": "2022-05-12T12:19:18.613123Z", "iopub.status.idle": "2022-05-12T12:19:18.657172Z", "shell.execute_reply": "2022-05-12T12:19:18.656456Z"}, "papermill": {"duration": 0.071507, "end_time": "2022-05-12T12:19:18.658957", "exception": false, "start_time": "2022-05-12T12:19:18.587450", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def visualize_dequant_distribution(model: ImageFlow, imgs: Tensor, title: str = None):\n", " \"\"\"\n", " Args:\n", " model: The flow of which we want to visualize the dequantization distribution\n", " imgs: Example training images of which we want to visualize the dequantization distribution\n", " \"\"\"\n", " imgs = imgs.to(device)\n", " ldj = torch.zeros(imgs.shape[0], dtype=torch.float32).to(device)\n", " with torch.no_grad():\n", " dequant_vals = []\n", " for _ in tqdm(range(8), leave=False):\n", " d, _ = model.flows[0](imgs, ldj, reverse=False)\n", " dequant_vals.append(d)\n", " dequant_vals = torch.cat(dequant_vals, dim=0)\n", " dequant_vals = dequant_vals.view(-1).cpu().numpy()\n", " sns.set()\n", " plt.figure(figsize=(10, 3))\n", " plt.hist(dequant_vals, bins=256, color=to_rgb(\"C0\") + (0.5,), edgecolor=\"C0\", density=True)\n", " if title is not None:\n", " plt.title(title)\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "sample_imgs, _ = next(iter(train_loader))"]}, {"cell_type": "code", "execution_count": 34, "id": "2df79715", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:18.711236Z", "iopub.status.busy": "2022-05-12T12:19:18.710706Z", "iopub.status.idle": "2022-05-12T12:19:19.802681Z", "shell.execute_reply": "2022-05-12T12:19:19.801929Z"}, "papermill": {"duration": 1.121703, "end_time": "2022-05-12T12:19:19.805548", "exception": false, "start_time": "2022-05-12T12:19:18.683845", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "a05e28af0506439f8843dceab189d0a8", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/8 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:19.238784\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["visualize_dequant_distribution(flow_dict[\"simple\"][\"model\"], sample_imgs, title=\"Dequantization\")"]}, {"cell_type": "code", "execution_count": 35, "id": "58bb60f6", "metadata": {"execution": {"iopub.execute_input": "2022-05-12T12:19:19.861772Z", "iopub.status.busy": "2022-05-12T12:19:19.861328Z", "iopub.status.idle": "2022-05-12T12:19:21.203466Z", "shell.execute_reply": "2022-05-12T12:19:21.202691Z"}, "papermill": {"duration": 1.371959, "end_time": "2022-05-12T12:19:21.206266", "exception": false, "start_time": "2022-05-12T12:19:19.834307", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "d60486171a554fbfb03474a308153d0d", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/8 [00:00\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2022-05-12T14:19:20.625605\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.5.2, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["visualize_dequant_distribution(flow_dict[\"vardeq\"][\"model\"], sample_imgs, title=\"Variational dequantization\")"]}, {"cell_type": "markdown", "id": "7c10c6d8", "metadata": {"papermill": {"duration": 0.029624, "end_time": "2022-05-12T12:19:21.267494", "exception": false, "start_time": "2022-05-12T12:19:21.237870", "status": "completed"}, "tags": []}, "source": ["The dequantization distribution in the first plot shows that the MNIST images have a strong bias towards 0 (black),\n", "and the distribution of them have a sharp border as mentioned before.\n", "The variational dequantization module has indeed learned a much smoother distribution with a Gaussian-like curve which can be modeled much better.\n", "For the other values, we would need to visualize the distribution $q(u|x)$ on a deeper level, depending on $x$.\n", "However, as all $u$'s interact and depend on each other, we would need\n", "to visualize a distribution in 784 dimensions, which is not that\n", "intuitive anymore."]}, {"cell_type": "markdown", "id": "620279fc", "metadata": {"papermill": {"duration": 0.029553, "end_time": "2022-05-12T12:19:21.325422", "exception": false, "start_time": "2022-05-12T12:19:21.295869", "status": "completed"}, "tags": []}, "source": ["## Conclusion\n", "\n", "In conclusion, we have seen how to implement our own normalizing flow, and what difficulties arise if we want to apply them on images.\n", "Dequantization is a crucial step in mapping the discrete images into continuous space to prevent underisable delta-peak solutions.\n", "While dequantization creates hypercubes with hard border, variational dequantization allows us to fit a flow much better on the data.\n", "This allows us to obtain a lower bits per dimension score, while not affecting the sampling speed.\n", "The most common flow element, the coupling layer, is simple to implement, and yet effective.\n", "Furthermore, multi-scale architectures help to capture the global image context while allowing us to efficiently scale up the flow.\n", "Normalizing flows are an interesting alternative to VAEs as they allow an exact likelihood estimate in continuous space,\n", "and we have the guarantee that every possible input $x$ has a corresponding latent vector $z$.\n", "However, even beyond continuous inputs and images, flows can be applied and allow us to exploit\n", "the data structure in latent space, as e.g. on graphs for the task of molecule generation [6].\n", "Recent advances in [Neural ODEs](https://arxiv.org/pdf/1806.07366.pdf) allow a flow with infinite number of layers,\n", "called Continuous Normalizing Flows, whose potential is yet to fully explore.\n", "Overall, normalizing flows are an exciting research area which will continue over the next couple of years."]}, {"cell_type": "markdown", "id": "27e38410", "metadata": {"papermill": {"duration": 0.029009, "end_time": "2022-05-12T12:19:21.385707", "exception": false, "start_time": "2022-05-12T12:19:21.356698", "status": "completed"}, "tags": []}, "source": ["## References\n", "\n", "[1] Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017).\n", "\u201cDensity estimation using Real NVP,\u201d In: 5th International Conference on Learning Representations, ICLR 2017.\n", "[Link](https://arxiv.org/abs/1605.08803)\n", "\n", "[2] Kingma, D. P., and Dhariwal, P. (2018).\n", "\u201cGlow: Generative Flow with Invertible 1x1 Convolutions,\u201d In: Advances in Neural Information Processing Systems, vol.\n", "31, pp.\n", "10215--10224.\n", "[Link](http://papers.nips.cc/paper/8224-glow-generative-flow-with-invertible-1x1-convolutions.pdf)\n", "\n", "[3] Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. (2019).\n", "\u201cFlow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design,\u201d\n", "in Proceedings of the 36th International Conference on Machine Learning, vol.\n", "97, pp.\n", "2722\u20132730.\n", "[Link](https://arxiv.org/abs/1902.00275)\n", "\n", "[4] Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. (2019).\n", "\u201cNeural Spline Flows,\u201d In: Advances in Neural Information Processing Systems, pp.\n", "7509\u20137520.\n", "[Link](http://papers.neurips.cc/paper/8969-neural-spline-flows.pdf)\n", "\n", "[5] Hoogeboom, E., Cohen, T. S., and Tomczak, J. M. (2020).\n", "\u201cLearning Discrete Distributions by Dequantization,\u201d arXiv preprint arXiv2001.11235v1.\n", "[Link](https://arxiv.org/abs/2001.11235)\n", "\n", "[6] Lippe, P., and Gavves, E. (2021).\n", "\u201cCategorical Normalizing Flows via Continuous Transformations,\u201d\n", "In: International Conference on Learning Representations, ICLR 2021.\n", "[Link](https://openreview.net/pdf?id=-GLNZeVDuik)"]}, {"cell_type": "markdown", "id": "48804717", "metadata": {"papermill": {"duration": 0.029953, "end_time": "2022-05-12T12:19:21.444776", "exception": false, "start_time": "2022-05-12T12:19:21.414823", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://www.pytorchlightning.ai/community)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "[![Pytorch Lightning](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAByMAAAE8CAYAAABjFR0gAAAACXBIWXMAAAsSAAALEgHS3X78AAAgAElEQVR4nOzdT2ydV3rn+eP3FQLYoVJsZFEgPU4pjYDCAO5QobxLai6dypKKlFQ2Mroj2inImwnEkrwZIl2kqgOtJJWcrGTUlKikoZpFqkr+s0zGZLuzk1xiV23kDiZyKkVNLYKRIkWFALqvB7/X5zBH9+Wfe8lz3r/fD0BI5qXFe+/73vc95zzneZ7nPvvsMwNsZ3F6Y9YYM2sfvmeMWb2wPnlvmx8HAAAAAAAAAAAANhGMRMHi9Ma4MWbBfn2h8APGvGuMWb6wPnmn8AgAAAAAAAAAAABgEYzEMxanN+YVaDTGfGmId+a6ApYX1icfFB4BAAAAAAAAAABA5xGMRG5xeuOIMeaKMaY34jvyUP/fhfXJ5cIjAAAAAAAAAAAA6DSCkR1nS7IqCHlqn+/Ep8aY+Qvrk6uFRwAAAAAAAAAAANBJBCM7bHF6Y3mHvpB7tWaDkve6/v4CAAAAAAAAAAB0HcHIDlqc3pg1xqwM2Rdyr95W70n6SQIAAAAAAAAAAHQXwcgOWZzeOGSDkKP2hdwr9ZNcuLA+udL19x4AAAAAAAAAAKCLCEZ2gO0LqZKsZyp6tes2KEk/SQAAAAAAAAAAgA4hGNlyi9MbCzYQGbIv5F69a4OS9JMEAAAAAAAAAADoAIKRLWX7Ql4xxkzX7BU+tM/rCv0kAQAAAAAAAAAA2o1gZMvYvpAK9h2v+Sv7VBmb9JMEAAAAAAAAAABoL4KRLWH7Qqok61LDXtGaLd16p/AIAAAAAAAAAAAAGo1gZAssTm/M276QX2rwq7lug5KUbgUAAAAAAAAAAGgJgpENZvtCKgjZa8lLemh7SS4XHgEAAAAAAAAAAEDjEIxsIFuSVX0hT7X0Jaqf5PyF9cnVwiMAAAAAAAAAAABoDIKRDbM4vbFse0N+oQMvd80GJe8VHgEAAAAAAAAAAEDtEYxsiMXpjRM2G7LJfSH36m2Vo6WfJAAAAAAAAAAAQLMQjKy5xemNIzYI2Za+kHv10AYkrzTz6QMAAAAAAAAAAHQPwciasn0hVZL1TNffiwHrKlNLP0kAAAAAAAAAAID6IxhZQ4vTGws2ENmFvpB79a4NStJPEgAAAAAAAAAAoKYIRtbI4vTGrDFmpaN9IffqvMrY0k8SAAAAAAAAAACgfghG1sDi9MYh2xfyeNffiz361PaTXGnkswcAAAAAAAAAAGgpgpEVsn0hVZJ1qbNvQlhrNihJP0kAAAAAAAAAAIAaIBhZkcXpjXmbDUlfyPCu236SlG4FAAAAAAAAAACoEMHIktm+kApCTnfqhZfvoe0ludy1Fw4AAAAAAAAAAFAXBCNLYvtCKjB2qhMvuD4+tVmSN7v+RgAAAAAAAAAAAJSNYGRkXl/IBUqyVmrNBiXvdPg9AAAAAAAAAAAAKBXByIhsX0hlQ36ptS+yed7WMaGfJAAAAAAAAAAAQHwEIyNYnN44YvtC9lr34trhoQ1IXun6GwEAAAAAAAAAABATwciAbEnWK/SFbAz1k5y/sD652vU3AgAAAAAAAAAAIAaCkYEsTm8s0xeysd61/STvdf2NAAAAAAAAAAAACIlg5D4tTm+csNmQ9IVsvvM6lvSTBAAAAAAAAAAACINg5B7RF7K1HtosyZWuvxEAAAAAAAAAAAD7RTByRLYvpEqynmnUE8eo1m1Qkn6SAAAAAAAAAAAAe0QwcgSL0xsLNhBJX8juuK5jTj9JAAAAAAAAAACA0RGMHMLi9MasMWaFvpCd9dCW5KWfJAAAAAAAAAAAwAgIRu5gcXrjkA1C0hcS8qnNkqSfJAAAAAAAAAAAwBAIRm6BvpDYxZrtJ3ln5x8DAAAAAAAAAADoNoKRAxanN+ZtSU76QmI3121QktKtAAAAAAAAAAAAWyAYadm+kApCThceBLb30PaSXN72JwAAAAAAAAAAADqq88FI2xdSQcjjhQeB4X1qsyRv8p4BAAAAAAAAAAB8rrPBSNsXcsEYs1R4ENg79ZOcv7A+eY/3EAAAAAAAAAAAdF0ng5H0hUQJ3jbGLNNPEgAAAAAAAAAAdFmngpG2L6R6+/UKDwLhPbQBySu8twAAAAAAAAAAoIs6EYy0fSEVhDxVeBCI71NbunWV9xoAAAAAAAAAAHRJq4ORXl/IBUqyogbe1blIP0kAAAAAAAAAANAVrQ1G2r6Qyob8UuFBoFrn1bOUfpIAAAAAAAAAAKDtWheMpC8kGuKhzZJc4YABAAAAAAAAAIC2ak0wkr6QaKh1G5SknyQAAAAAAAAAAGidVgQjF6c3lukLiYajnyQAAAAAAAAAAGidRgcjF6c3Tqj3Hn0h0RIP7flMP0kAAAAAAAAAANAKjQxGLk5vHLFBG/pCoo0+Vclh+kkCAAAAAAAAAICma1QwcnF6Y9wGIekLiS5Ys0FJ+kkCAAAAAAAAAIBGakwwkr6Q6LDrtp8kpVsBAAAAAAAAAECj1D4YSV9IIPfQ9pJc5u0AAAAAAAAAAABNUdtgJH0hgS19arMkb271IAAAAAAAAAAAQJ3ULhhJX0hgKGs2KHmHtwsAAAAAAAAAANRVrYKRi9Mb6gm5TF9IYGj0kwQAAAAAAAAAALVVi2Dk4vTGrDFmhb6QwJ6on+TyhfXJK7x9AAAAAAAAAACgTioNRi5ObxyyQUj6QgL7p36S8xfWJ1d5LwEAAAAAAAAAQB1UEoy0fSFVjvVM4UEA+7Vmg5L3eCcBAAAAAAAAAECVSg9G0hcSKM3btnwr/SQBAAAAAAAAAEAlSgtG0hcSqAT9JAEAAAAAAAAAQGWiByPpCwnUAv0kAQAAAAAAAABA6aIFI+kLCdTSu8aYBfpJAgAAAAAAAACAMkQJRtIXEqi988aYK/STBAAAAAAAAAAAMQUNRtIXEmiUhzZLcoXDBgAAAAAAAAAAYggSjKQvJNBo6zYoST9JAAAAAAAAAAAQ1L6CkfSFBFqFfpIAAAAAAAAAACCoPQcjF6c35tVzjr6QQOvQTxIAAAAAAAAAAAQxcjDS9oVUEHK68CCAtvhUWc/0kwQAAAAAAAAAAPsxdDDS9oVUEPJ44UEAbbVmg5L0kwQAAAAAAAAAACPbNRhp+0IuGGOWCg8C6IrrNihJP0kAAAAAAAAAADC0HYOR9IUE4Hlorwf0kwQAAAAAAAAAAEPZMhhJX0gAO6CfJAAAAAAAAAAAGMozwUj6QgIYgfpJLlxYn7zDmwYAAAAAAAAAALayGYxcnN44YYxZoSQrgBG9TpYkAAAAAAAAAADYSh6MtGVZP9zicQAYBgFJAAAAAAAAAABQ8Nz/8es/HTfGqMzilwqPAuVYpz9p4z00xhy6sD75oOtvBAAAAAAAAAAA+DeJMeYEgUhU6PqF9ckjxphPOQiN9gV7LQEAAAAAAAAAANiUEEBAhRSInF+c3jhEQLwVZrv+BgAAAAAAAAAAgGcpGDle+C4QXx6ItL/lCO93Kxzq+hsAAAAAAAAAAACepWBkr/BdIC4/EGnIqAMAAAAAAAAAAGinhOOKkg0GIg3BSAAAAAAAAAAAgHYiGIkybRWIlOnCdwAAAAAAAAAAANB4BCNRli0DkYvTG2RFAgAAAAAAAAAAtBTBSJRhy0CkRTASAAAAAAAAAACgpQhGIradApFypPAdAAAAAAAAAAAAtALBSMS0WyDSkBkJAAAAAAAAAADQXgQjEcuugcjF6Q1lRX6h8AAAAAAAAAAAAABagWAkYhgmI9JQohUAAAAAAAAAAKDdCEYitPUhA5GGEq0AAAAAAAAAAADtRjASIa2PGGAkGAkAAAAAAAAAANBiBCMRSh6IvLA++WCYf29xemPcGPOlwgMAAAAAAAAAAABoDYKRCGGkQKRFViQAAAAAAAAAAEDLEYzEfu0lEClHCt8BAAAAAAAAAABAqxCMxH7sNRBpyIwEAAAAAAAAAABoP4KR2Kv9BCKlV/gOAAAAAAAAAAAAWoVgJPZiX4HIxekNsiIBAAAAAAAAAAA6gGAkRrXfjEhDv0gAAAAAAAAAAIBuUDByjWONIYUIRBr6RQIAAAAAAAAAAHTDAY4zhhQqEGkIRgIAAAAAdnHIfsWyygEY2nKkf3fFGHOv8F10gSpmjUd6nXeMMSHWrgA0R8wxA9cUIBCCkRhGsEDk4vSGBpxfKDwAYC/0eTphA/y9hr6DO2XnuwWiB3bwd4/Fik6Yt19ongX7WQXQzWt3rAUgghXdpfNqKeKrf67wHWwn1nFY5fPdWVcizmFfZbMB0DkxxwxcU4BACEZiN58GzIg09IsEgjrhBlv379/Pv5pkbGzMTE1N7TQBLTyWZdnjJElu24HgKgPCVjq01bFHI8Ta3Q6g/uYjXrsJVnTcBx98YDY2NoK9CceOHTMTExOF72Nnn3zyibl06dKOPzPKMZibmyt8H93zzjvvBHvNr7zyipmZmSl8H0B3hBwzcE0BwiMYiZ08VLAjYCDSUKIVCO/NN980t2/fbtU7e/DgQQUq878fPnw4D1xqIDgxMTE2MTHRswuebtfbu8aYm/aL0hkt0cbzuq1Onz6dfwGArt2hEKyA8/777wcdE9gxZeH72Nnjx4+DHYejR48WvoduChmMFAIHQLeFHjNwTQHCIhiJ7Ty0GZGhy60RjASwq0ePHm0OIN2fbqI6OTmZL2C4r4mJiePGGH1dM8Zct+XcyJgEAKBkIRd/CFYAAAAAQHsQjMRWogQiF6c3VHrvS4UHAGAEKrmhL+14E2VQKntCX2NjY6eMMaf6/f6P0zS9ZAOTAAAAAAAAAICKJLzxGBArI9LQLxIILv9MNa1XZGiuf83s7Kw5f/68+fjjj02api8rUzLLsn+wvTUBAAAAAAAAABUgGAlfzECkoUQrENy4sZmC+JyyJdW7Tj2rFJRMkuQlY8wPbNlWNkQAAAAAAAAAQMkIRsKJHYg0BCMBlEU9q/ygpDGmZ4z5oTFmmYMAAAAAAAAAAOUhGAlTRiBycXpDGVzThQcAICIXlLx8+bJ5/PixftFSv9//EVmSAAAAAABUapy2KgDQHQQjUUZGpGHhH0CVbty4YY4dO2bW1tbyfpJZln1kjJnnoAAAAAAAUCrNxW8aY/4/Y8wCbz0AdAPByG4rKxBpKNEKoGqPHj0y586dy7MkkyQZM8ZcM8ascGAAAAAAAIhKGZArWZY9snPx47zdANAtBCO7q8xApCEYCaAulCX52muvubKtp+yOzHEOEAAAAAAAwahK2pV+v/+PxpgfaP795MmTsQ8++CCfkwMAuuUAx7uTyg5ESq/wHQCoyCeffJKXbb169aqZmpo63u/3P0rT9MvGmAccEwAAAAAA9uSQsiCzLDubJMlL+gfSNDUKQK6uruZfAIBuIhjZPaUHIhenN8iKBFA7Ktv65ptvuoDkywQkAQAAAADYk3lbijUvv5okiVlbW9sMQGr+DQDoNoKR3bNQckaksWUZAKB2BgOSxphVrlkAAAAAAOxKWZDzNgvyoH74/v375rvf/W4egNzY2OAdBABsIhjZLa9fWJ9cqeAVkxkJoLYGApLTaqpvd3UCAAAAAIBnjasXpHpAGpsFqTKs77//vrl9+3bhhwEAMAQjO6WqQKQhGAmg7gYCkppQ3TPGLHPgAAAAAADYNG5bnLzssiAVhKQMKwBgN8kuj6MdKgtELk5vqNzhFwoPAEDNaPK0vLxsHj9+rCe2xEYKAAAAAACesaBApPpBHjt2zNy4cYNAJABgKAQj26/KjEjDYj6AJvnkk0/ygKRkWfaeLT8DAAAAAEDnqT+kNvC6eTMAAMMiGNluVQci5UjhOwBQY2q0r1IztgF/1ddQAAAAAADqYFbzZM2ZyYYEAIyKnpHtVYdApCEzEkATXbp0yRw9elT9I48bY04YY25yIFGmqakpc/DgwUa855OTk4XvAQAAAGidPOHg9u3bHFnU0uHDh4M9Lea5QHgEI9upFoHIxemNQ8aYLxUeAFB76v0wMTGx7dNUkGS3QZ7+/+3+jY8//rjwvbt3727urnSTmyonOQpIXr16VWVo/iJJkl8xxjwo/BCiOXfunOvfWXs6d3W+hPTWW2+ZmZkZTjAAAFALGpfcunWLgwF0W97GZGNjo+vvA2rq7NmzHBqgxghGtk9dMiINWZFAcykYGSgQsjb4jX6//8szMzMvD35/u9+nPo4KSupL5WDKot+ncq0nT55UetqCMYamGCVSZiBy5xv0NtwrfAcAALRBYUwfCJv9AAAhxFwsYp4LBEIwsl3qFIg0BCOBVnh1hxexp8FemqaF71numjFuy78ou/rQ1NRUT4GpkydPmvv375t33nnHvP/++4X/OQb9LgVmn3/++bfSNF1hEFqK5YYFfj8rfCcsguAAAKBqzO0BAHW2GjkgCSCAhDexNeoWiDRMWIBWWN3hKzT3u27aAMy8vY78O2PM7xlj3lXZ16WlpTwYOTsb/xKjsrE3btxQAPUXCQoBAAAAAAAAwOgIRrZD7QKR9IsEENADG6A8YYz5VWPMdQUlL168aJaXl/P+lTGpVKt6F/b7/T9wPTIAAAAAAAAAAMMhGNl852uYEWnIigQQyT2bMflqlmU/mZubM1evXo0akBzIjlwo/AAAAAAAAAAAYFsEI5vt+oX1ybqWDSQYCSCm1SRJft0Ys6Z+krEDkh988EH+Z7/f/1rhQQAAAAAAAADAtghGNpcCkfM1fvYEIwHE9sBea667gGQsGxsbZm1tTdmRL9pysQAAAAAAAACAIRCMbKZaByLpFwmgZPMuQ/LcuXPRfvP777/v/kowEgAAAAAAAACGRDCyeeqeEWnIigRQgRPqIXny5Elz9OjRKL99dXXVPH782GRZ9vuFBwEAAAAAAAAAWyIY2SxNCEQagpEAKvAgSZI/1K9dXo7XSlcBySRJDnKdAwAAAAAAAIDhEIxsjqYEIg2L9AAqsqpr5cTEhDl9+nSUZ3D79m33V65zAAAAAAAAADAEgpHN0KRApKFfJIAK5WmRr732WpRnQDASAAAAAAAAAEZDMLL+mhaIBIAq3dN1c2xszBw7diz409jY2DD379/XX3uFBwEAAAAAAAAABQQj641AJACM7or+j9nZOMmLn3zyifvrkcKDAAAAAAAAAIBnEIysryYHIh8WvgMA5bmTZdlPer2eOXjwYPBfevfuXffX8cKDAAAAAIA2GbdtOga/2JwKAMAIDvBm1VLTMyJvGmNOFb4LACVJkuT7xpgzyo58//33g/5SLzNSE9DVwg8AKItLfx5Mg/b/e/AzurrN94FQZu2ipVugHN9msfKOMebBwN85L+M5Yr8OuWtEv9//5TRNXx74jev2WDywx8V93WvRewEgnnHvejPuXW9+LU3TF7f5re66c89+rQ7cI1AuF3A8kmXZTJIkL+3227Mse5wkye2B+8adwg8CALC73eaS/pyxceMFgpH1824LSrMuE4wEUDFtijgzNTUV/Fk8evSo8D0AUbkBeL44tMuC3qDB/q5L7i/9fv+naZresgP41ZYEgma9YOxWE5f9WNjHwpoLArnnc8h+bWdwUvXAXtfrFhDyz8tXRjgvzRbnZs47L1e9BWmMTufXCfu15Xudpqn5+OOPN/9b/aanpqamvR857v6SZdkju9Fp1Z6LBAnQJaHvJ757DQ/2H/LuA7+z3X3g5z//ub+hcdPAdcddq/KxSr/f/3Gapn/jXXdGeU473WP3o+nHaztagzuRZdlXkiQZcz+TJMnmfeLWrVuF//Xw4cN5JZ6JiYmxiYmJnn+/8e4bN0c8fjGEPCdinFtHShiHx+nhEs7KHj5bux3X7TZltnWzQ8xrX9Pes+XCd3Z3ZIfqX1td+1d3eAzDc+MIt5Fpy3nLgKXCd4xZs8fhjre+sVezEa+Z9whG1su6HQQ12oX1yXuL0xtvKxDQ9QMKoDL5jVcTxNAGMiMHaSDx94Xv7oNdmP47+y/cswGBNkwe9F4tZ1n2+0mShK+na8yvMihutG0DCW5BTyWTtTlAf/c3Cbj/1uKQvyFhcnJSi0X5n/qamZnRguGLNuDgBvTveot+TTx/ZreZnISw3eR0K7P22OnP6S0e381Wk7DxPU6sQzthFyyfuXa5wNbGxkb+5Z+Xg+eoHD16NP/Tnaf6U/esgfPSZVt8jyDYUHSOzPf7/T/yMx7v379vbt++vXnd0H/rGO3EXS90nHRcjh49enBsbOyU3XB5zV4rrpDNio7Q4tiHkV7q+Zpc20eRj1EGrzXuPqDrjK43g/eC3eh64+4Jr7zyiu4H+rf1dcYGtr5jrzu7jU8WIq7FNPF4bUfHcSHLsjfc/fxnP/tZfr9wX7vdKwZ594xn7hvevXylovvGfMTxYQhf2Gbs1yUxEjq23ZTpZfOueoGLpo8xY57nrzZszBf6fdjq8zn4O/xgGBsqd6Zx1bydTz6Tfa95ir78DTC6Hzn+GoebP9qNTW5DjH8tWbPzx1GPx4mI44g1gpH1oUDk7IX1yVYsMFxYn1xYnN4YJ0MSQFXUN3JmZmbXsjqj2mVB4Z5doDyuxYfHjx8XfmBUU1NTL46Njbld1r1+v380TdMvN3yyoLJHH2nn8ZMnT7bcKb4XWjjWlzcQRrNsG0hYXV3dDCYMuzCkz6o/cPf/7ijgYBeM8q+JiYnjNgj0LTs2u9LEANDly5fzxdCtXvOoTp8+nX8NobCoZ2wgbqtF2Z2emwvUGbup5OzZs4WfKdkR+9q+6jImdO1yi5WaLI56HfNfv85vnyaYbjFzdnZ2zFvM/POKFzPr6pBfmUUBAR2PDz74IH9vR11MFhdU9o+TjsuxY8d0TDavFRprJEnyDXtMgFZToG2r7LC90P13bm6uaW/XCRvoyxdl3bXGjVF2uq8Nw/3/+vfeeeedfJHR3gf0pcDWGbs4uGavedvdB/IsVv0boTT0eG3nmXuG7ud6z2/cuLHvOYk7D/RviT12z9zLbcbrpSruG7ov7uWeuBWNz0NQ4D0mfQ5mZmai/o4A8s/s2tra5rkzDLcpcyuDmzIHN79tlc3rBS5WmrzWEPI8txtDCt9vAp0bly5dGvqZ7rRZz507jn9+6T3aKhhmA95/42WHd31D5bjdyHTOrXW47HuNrUYZRwzOHR23IUbHxq5tbH7G7Zzl+/bzvVtgMr8mvfnmm4UH9uPq1av5/00wsh5aFYh0VG52cTq/kBGQBFC6JEn+H2NM8GDkEG66YOTyctiNw/r35ubmXrYDiBOFH2iGQy4QqclhyIUS+/4YG0BCcxyxi3vBAgnDcgEH11vWZSPoPLJl0655AaDlpgS5QwUizbOLTdv1yd1yUc8tzu6ygWNLoZ57APP+hNF/bdtNAkNwAVz/vFQQ7NixY35gkiDYwLmnDUBaxAu5COVzizr60uLya6+9ZuymJ10nvpkkyR9u8xkBWkGLZaHGbVoka1Bwa95+xvN5he6L3/3ud6OPUXT/dPcbLfzquqPNQXZx8UMbONi2hHrIMXbDjtd2XIWFPNvDHUfda2O14fDHC/Y+7jJer/X7/T9N0/RPyryP67XWaIyFf5NXHwk5fh/clGm2CGC4DQ9u04MXuNCmzOtN3fwW+jxvajBS4+JQ78NWQW93Pvn3GjeX1p92E4bb6HutyefUPo3bTa1ntWHXVVHQear3MOT9ZzCgqY1E+mzrM97r9V6y978zdlPM/7nbxoNY9wuCkYiKgCSACilo0NONt+RJ10qWZX+mHcwa4IccXGgBVAO7qamp4zsEBupuOUYg0u3YtmVtq+7LguHM2kWhfLeeFoU0KI8VSBiGCwIpoLFVAMhOohoTlAxhh2NRWNTTZzr0pKoimwvP/oTRBQfLtlMQrIrFzBrY8twr8/i4BWbde7QRxh6PD211hPmdJvYAGkMluf9M9wJlL7ispSqCObqvuvuQ5jbKVpiZmdH46YfGmLftNZHrzvZ0LP9CC8FaoNc9Y5QMtBDc8dP4Uvfxubm5F9nMgir5Gx40xnTzac1/JiYm3Nxnt0xsYJObSzsDVUUqzw6vwLILQrpNrbr/7DC/Dkq/R/c6fblNTd6mmG95Gw+u+Bub+v3+r6k1TixJyw5yU023+UOogKQdIANAmSoLFqini0pV6GYfkiYMXqmNJvZrURbLKbdwHJLbrZ2m6cVqXyKGcMhOaLVw39P5cP78+XxgXObgfDcuAKTnpednMwRP2b6wKyP2UGwblVr+HwoGaVHPHb+Y2QUlme33+z/S4qAWnxUY14KvMlGqCkQO0iRWz+d3f/d388B9mqZuMfMfGpwxP4oT9rUWzr0q6Hql46HzxC6+5KVbO3IsgLZy45QfuHuBrrnnzp2rRVaZnoOuO97Y5Iy9LrqJR2+rLJaOGrdVa3QsDyoTUveMsgORPlc9R+eUAtw24/ZD+zy7PLZExTSm0VzMzX20Gc9uGv3QXhOPcIwwCn8+/dZbb+XnlK04c83OucIumNXHrL0vL+ne4z5XuvZXtdbhNjW5eaSek20rdcpubFp1x0Pzy5jjCIKR9XF8cXqjzQFJlQ95vfAAALRTfj0/efJk8Bfn+ubZicGhwg/Umzan5CWRQtMOY4u+XfU1bnfd/f1gELIugZ6tuIH7YFDSBhwWtvhfWsULLrrrjT7HP3QLtHU/fkMat9eODzVB1kRZ1xRNGOtazkwT2S0WM39gJ5JNuzcMY8sF5bqcezpPdM5oYm/7iv6AkuFAIy27cYruBdpoUOXi4U50/dN1R9dD26v5Q3fdCdG33uctSjZp4XjWbpw6rrGbjqUWxeuyaUrnlALcel52bHncPl8CPqicC1p4m61cJnbXN2Rij9yGSp1TXlCybRsx3HrHh5qbaY7mAn912rDrNh4oeULrG95n/MOnT5/+98L/EBjByHo51fKA5AoBSQAdoRIHa65xdGheMK9R2ZH9fv9rxk5uQtKCtDJRbYkJSlTV0+x2mXRN4hb+NHh/8uSJTrpv2V2drV048hYgD9kFiGvuGGqBtgUlWV2mXZ61rV27mqg9p3AAACAASURBVCg3JavEX8x0E8ksy37UskC5O0a1XFD26dqg52cDAWfIdAEa45BdgFvS5/fy5cv5vaDu/fVc1ZSB606U39MwK24xWBunNHar67F0m1n0PO3Goh+6DZzojNpuInPn58CGzK5U40AELrt/YCPGP7QgS/JIv9//yK13aE6pOVodNzP53PqGCxIfOHDgN83OrVr2jWBk/RCQBIB2yK/lCriEpl1lGuBkWfbVBi1ynlC5B020Qy9oeO8xWSj1dMVfEGp6Jp3OXwUcNGj3dnX+sANZktoteUrXnjqVLd2nK36mnY6prq9NtEVm3rdaFAg74x+jugcH9Px0nXNlW+3CBAFJoL60YepHWoBzmfFVlvHci4HrTu0XP0vwJbfBqAkbp/T89DwV8LFB5WsNbcmBvcmDkXUe32yRif0DsiSxH27uMpDd39Tr3nyWZR9pXUDZkLofN21O6QeJNRYiGNk9BCQBYP/ynVUVZres9Pv9n6qXoZrBh+SazdsF56bsSsx3+IYOYCj7dGZmRlmXP/abbqMWDtmswXx3oCt11oJMupzrFafsCbtw1Kbgz5ZcILIFvajG7fVi89ysa6bdqFygfCAQ1ujMXRccaNIx0vN02apamCAgCdTWvN0wNabrp+5xTQ3k6brjMuy6HIz02wA0bTHYlca048ol2k+gTvxMbJcl2YZxJqqzRXb/UgPn0wqgXtM4QusCyoZs8pzSBSVjtFZyCEbWV1cCkg8LDwJAQKEHAl7Z1XuFBwekafptfUcBydC0YGI+L316rgHnyyFXXi/0jk/XKzJN00uFB1GlPMvA3x1Y92ymvVL2hBega0XwZ5BeW4sCkUdsyeBpvZY2npt6XZrU67Onz6B26jbxnNQ9w5VKbOJ5NxiQtIsrAOrjmdLjbmzddNr4pYBkF7WhAofuGV5A8hQlW1E3LqOt6eNM1Ic7pxpYVWTFlXfXmL9pVRV2EjOgSjCy3roQkJwlIAkgkp7dsRfLrsFIVzbUBcxC0o5nr0Rk3Qf/+SQ69O6qgwcP5oHeLMsesXO4Vp7JMmj67sBhuOCPFsHaOCnXtbRFgciP/B5SbT039br02bOlj8aaeE5qMbnpk3o/IGlLHXOvAuphpYWlxzd1NTOyLa97ICB5jf58qBs3zvTaA9DrFPui63fDqoo8M45o68brGAhG1l/bA5J3CEgCiCBfcI2xcH748GH312GCkQ+MMdfHxsai9I70Fk5q3auu3+9/zUQo0ere0yRJLhceRFXm7aJJq7IMhuH6/bhJuQ3+NL0Rf06vq0WByDEFIvWaukClj/RZbGpAsg1cQNKVNGOxDqjcMwuILbi/oYV0XirYI1mW/aXrKwjUieY9Gmda1xjjYD/cmNlt8K1xQDIfR+g67WV0YkgEI5uBgCQAjCYPAMTYnTQxMeH+Okww0rjsyJMnTxYe2C8F97wSPnXdNTafpumLGlCGzkDy3lMyTeph3pU706C8bVkGw3KTcrtL+MM2TMpbkD24GYjUselKINLRZ5GAZLVcBoH5fFH5z1lUBipDIHIPRmlTgXA0l3Wb3J4+ffpfeWtbK1+7aOr1yI0zrWtt2YyJargNvq4McA0DkpuBSAVOu9yjea8IRjYHAUkAGF4+AL5161bwt8zLjLxTeHBr+rm1qakpfyIfjBfwqWvAI39eoQNTs7OzLjD8LgsjtbAZiGRx79lJuQ08EPypzqEsy/6bFvLUf7CrQXK9br1+G5B8ryF9WFpF10WvnBmbaIDyzROI3DfG3CXTfUOtOQ4cOPCbda+Gg/1p8ua/gbnP+8x9sF+uMo9tS3SlJm/oghtHvPXWW61vRRMLwchmISAJALtTtsFxlUOLscigoGK/3/+pLcE6rPzaHaNUq+vDmGXZ2cKD1dOx6Ok4hM5S9fpw1mVg2mUEIrdANloYk5OT7t8Z5ZrrjPf7/feTJDmo7Oym9x/cL71+vQ/qmWmMudnsV9NMWlT2+kdSygwozwnGKmgqV9Ehy7L/wmYi1NXA3IeNb9iXgb7rqgRWdWkbjSO+5cYRZETuHcHI5iEgCQA7yxf3XJAuJAUi1f8xTdNRUy5XFMCcm5vzF9aD0CBIJSzs4nLdSqLku3dDHwu9hzMzM5qQ/8QYs1r4AZTpiOsRyeJekZ+N9tlnn60xKR+dVxp72Gx034p203apR+RuvP6fvRpM6jvJnYv9fv9PuSYApThke+61pQcyOkbzPS+znns3aktzH2/jG1UgsC+uZKttTbRU4XrXETeOUNsFxhGj89ZB7xGMbCYFJO8sTm+0cvJKQBLAPowrQ1CDlRil+F555RX315EDYGmaXtSfCkiGVtNSrToWb+hYrK6GjRe6XpFJknyj8CDKlJe/1O/TLlgG5Vtz2WjPPffcL9meFyiHFuuO67y8dOkSb7lHZYVqMKnvLJ2TuiaonzIl94D4bIb8mII5ocekQFm0udPeu8+wkQV15m36OM44B/ulc8nLDv+LCq5/qrTzl67lR+iKX13hbTAmGNlg01oMJyAJAM9YUDk+Lf7HqN/u9Xzcy0qGsiP/xSsvGowWVlSW1pavqMt94YSOhZ5byGNx8ODBvNxtlmWPKDNYLVf+UosjXe3DN6ya9ryoPX3WvWvmKGVaNYZcop/H1pRhUfGkvvMUFDGfX0e/1vX3Aojsiu696rnnPndAE2ks45WbJ8DTIv1+/9dsoLk1BkoLH+r6Mcb+aE1Jaw4247bs7PBljSNUjazrLT9COdCOl9FZLiA5e2F9ci99dGpNAUm9Nrvo/4WuH2wAuzriFp9jlGg1Nhipcqtpmu6lXOCDNE3/amxs7JQW2EMHb/TvqUymnZxWXr6n3++fS9M0+MLP7OxsXirXGPOdPfaQQxjLbnGvTllnKqWsgLWjIH1d+jmo54U+p2NjY2dsIJ30jG3oWqvPusuCNsa8O0KZVmVl/0WSJPlCSB2O/+B56VOQuopgqSb1mlT3er2X6nLfiEVlgbzduPn7XXUmt85LZUfOzc29aKsaUMoMCE9rCWc0N1BZtaro+q/7gNvUqP8+fPhw/nddj+7evZv/XdclfdGHqho6Rjouul/oz8H79q1bt/LMRB2vqjJjNMfVfE+VgJIkGfm+rTmot7l2aK7qUMzxiu7V3jwvGP/+X1eqlNC2CjN6PVoHOH36tCstTJ9s7IvOJ10jJiYmypxLb44jqmz54cYRujfpGtn0cQTByOYjIAkAtnSCgl8aJMSYKHmTo78uPDg8jWBOaRIZOhipRU39u8qySNO06kXlWReoCj0QsgFXQ3ZZpTYD/1Ut7mkArs+kG5Srh+hO9Fw1MNdCkv6sokybrkt6v65evarP6X+1WZKtDqhrYUmLXu44ub67I3p7xEDZsnbN6ppYxXHWa1RJb/3pvnaj81MLm3q+obPJd6L7pQ2QL9lg2L0dfrwR3GKmO+92WoT0rwt6/8teXNbualu6fYFgJBCevzGl7E0fuv64wM9u94Fer/fMf7sWB7omUXkiLh0fHadhgmCDY03Nc3R8yrxv6/fYjSyKlJ4YtUrMftqFaA4Wqz+8jsHS0lLh+x3R2uoUCh7p2E5MTJyy4xw2YpajleeUrn/aBH3x4kXz9OnTPz1w4MBvFX4osKaPI9xX3RCMbIeuBCRX7GsFgEErWtiPufisSaq1n9KgWuhdm5iY6GlQEXLhcyDLYuTJaWD5zsfQCyh6z+zC8lobFs2bKnbgfyduwWhwwG0zlv/OBvf8DLr8g/vCCy8cnZmZGfMXkpQV5haRyqLPvHa1nzx58kUbYGtdmS1XSlkLXttMnNYK3ylyx/HmCBmRxt+9WmbGrgJgyuK0u3UHH17b4rx0tFhwROdnr9cb03mtxThdy3Vuxg6O6fOrhaKzZ88au8HjROGHGkLnnI7BFuecO9/8D7o2VIy/8MILMzMzMwfddUHnjQKE+oyWcW1zu5enpqambQkz7mtAOPnGFN3ry7zP61qkgM3AvWDN3gPueJ9z96SO2HvBuP277glfmZubG9N9VJuYdD/QdYmMyXA0p1DFCnf9z7JMNTL/xh4Xd5z8a7I7PrpWz2ZZ9tszMzMv6f/XMSrz3qHz2QYVR5nv7TcQNKuNSy64FPJ1agylsY+OQZIk/3mb8VKbHWnzi9M5YwPNy/QpL01rzyld/7QRZGZm5jdLqCxSp3HE+sD9yXjX9EP2a8txhMtsr9M4gmBke3QpQ5KAJACfBiDHtaAXa/FZi+u6kduAx36DfFrw7akPWuiFZg0y7OR0vsJgpAZCp1QeM3Qw0usdR1ZkdTZ7JpQ1KNfnT0EGHX+3az3Lsp8kSfJ9V6JFwdGdaEejHaBrLHGi3+//Tq/Xe1HBH52rmiiXlX1QUYmZ6LSYpInTwK77de813omdCdrv9/+8zED54GKm9a59vavDLqjZ81OTx/l+v/8H2lSi91FBSd3XYr4WLaDq8zUxMXHcfj4adT5uMWFf9xZdd3z/7ft+yL5uLbTmlQt0rdH7UkZ/OR1jGww+wb0NCOZQv99/6+c//3lpG1O2uBat2WvRzV3uff51Kh+722uTrkvzL7zwwldPnjw5puu0FhJ1XaIP8t5pTKkxgrepLT9OSZLstqD9wLs/rvj37eeff/5rp0+f/kWdA/q3Y28k0vhbi8s6N5IkGbb05WBwdeRfq3G0Wg1oDBtyzOxK8idJ8scNqRLwWeE7AWgsqUoNo9AcRl+DXNnGOpRsdK1ktBmbjVdb0rX+w60e2K+9nFMK9G1F55POqyoqiQxSlSF9ZVn2zSGu3Xulth/nnjx50oRxhH99b8Q4gmBku7Q9IPmAgCQAz7i92fY0yNaCcKybqgYG5vN+Dn9VeHB0NxVIUZ8uLd6HnBxoYKgJiV1UrmqwHyUrUu+VFg5sEKrKrM8uq2RQrh3nXums61qwT5JkxyDDNh7Ya8ZNG7zMB+gqHaQduxr8l7GIVEWJmdgUvNH7p+OUZdmjJEkuV1D2c96Vh44dKNdips5LF3i116VvDDFZ3InO6YU0TZUtO68J9tzc3Eta9NN5GfM1NXHXuo6BPkdeIPi6ff6jnnP37Lm6YjOVF1544YVz6nHk3vuYfZx0XG0wcp5gJBCMNk79ojYVxF6E1/hU14kA16JB+aaWJEny65J6BJ48efKgC3jVsexa3SlzXvcNLfTa+/Yf7nMDjrtv63hf0XhSC+S6p8bezGKzI8dK3kSkMc6ZLSpA7Ivre1ZxVZ+RuDLvIWjsbKs6ZMaY/2mM+X+H/We/+MUvHp2YmNiyvrBfRUbP1VVnqSIwOTDOpHfkFrYLLO/FwDn1o2HnJlmW/XtlfBce2KZMta5DsXvJbkXnss2OfCniNVDrDWOMI+IgGNk+LiB54sL6ZOt2nBCQBGDN2p5rL8YORBpvx2aohUK7UP8t/buhgzqaZHgLm6X3jrQ9K/PnEZJb9LfvHapR9aB8IXBmnctec4tIx7WIpKzP2Jl1JZeYicbPMOj3+/9ijLmYJMmVKnph2t2xeZA3JmVD6rptA695EDLCrtwVu+liQWXR9JouX76cZ+vFMLBrvfbZkf6Ccr/f/3Gapv8pUFm3B7Ykk87hlampqeNaQIvVI8vYEuteqdbxtveRBUqQV+hQsCD0WHSQ27Cg+8HTp0//9sCBA/97hBKT/nVp855QRuZ8m+i+oeu53dx23S7OhrrePnBVabIs+0ttZnHj2Fi0GG/nRo2raLCDxtz/dN/W2CAEjSs1/zDGfDTqhjCb/bQVV1JYf57QGEOfAa09aP6h31dmdpvGmdrEZ7N5Q8/nWkHvUahNDN459T9HKdm6w/m0bZlqrTvpfqTnXmag22VH2vWB0NfA0sYR/sbrksYRy8pwd+OImPepnWx7pqHRNJlVWdNW1om2WZ+ztgwTgG45ZBftP1QgUoMDZeTEnIjbpuvGBkNCbfJYUV8M/dtazA/JZSQqKBj0Hx6OMpNe1MAm9DHRcbbBjkYGbVqgtEG5JlAKumiCo0CDMeY37CJPrInrPVsi8VWbtZz//i16zwVlJ1B5EC3qL4pE1y69BhuIVEDoZRvYrWKBYV49PXTtiZnFpmu2XrNdzHw7SZJfj3hNemDfT52XjzXZjzlh9BZAat3H1C0ou/tymqb/IdKkXdeEr+tY6/fFvB54C4L0UgL2L79Q6j4ec36gcakW8+z94LytchCz1527J/yGxioKROl+FHoe0UZ6j7xA5PmIY8qbSZJ8+bPPPvtnHZ9QwaqtcN9opZABFdd3fdkGkf6dMeZ1XTs0v3IZvAqal0VrFNrU2uT+5A30fwV6yq5M9Yqdc/2KMeZXdT1VRRxd79577708qFbWPclVBFO7JhssDSmfC8UeR+geoYxhe2/6eknjiAV/HKHXWMU4gmBke33BZkgSkATQBrN28PP3rh+hsiHLKBXpTSRDLjg/SJLkexp4aFd1SBowaUFeQcEKyqDkvy909o4CAHqvbJlcdlJWo5TFvYFgz/lIgYbtrNrg0nUFOsoIQGh3soJoTStZ5AKR9v1RQOjLVfaAcQHdmGXRdG5qwqjAoDHm9yJk6m5H5+WX9Xs1aXRlw0PTIpE3qT9Uwusa2TYLyjFpB/HrLiAZa7LuLSq3ct4GlEjl5L8ae+OUNoZog4i9H7xaciWSO3assqZ7MAHJ3blqBl7pu5juPPfccz2dG5pDhp7nOcpA0nmeZdnRwoNA0QNbdUNBpFe1iVBBSc3rYo0rB3ktXAhGtsM9m233SzbQ/UiZt2Vs6HW8cyrkfEDjiDfKGEfoHuGNI8ps1VD5OIJgZLsRkATQVEdc6cJ+v/+Ptql3HoQ8f/58Pmguo7SIlxW5FqH8Qz4ZjrFrNtLAbDdHXP/O0JlJocvkYmTjZWRFDgR7Xq+izLBXaivPiNKEKtZCkvGyI5sWjNQEygUiI2etDuOEAroqrxurPJB/biowWEFfozsuIKnnEeuc9D7ftcyOLHlB2dFGpLf1e2NtgPLumWS4APujjI2xmD2slBGpjSHe/aCKEpluHeQ6Acmd6f7tVdooa6yle3Ye4dF4Kdax0b3DZpqFzgpCu63azZ75XEfjyjJKNep8tQH0rxQeRNO5QPfbWjsrK8itDfhWyPGz5pUHY44jtP7X9XEEwcj2cwHJVu4+UUDywvqkFsDfLjwIIITVIb+W9/B1c/Df1o4qY8xnxpgfGmOuKQCiDD8tMr/11lv5oMYLtEWlm7FKTVgxJq/aTbamAZvKUobkla3olZjhki9ehw5W6b2xQY+1EjPk8KzNjNdYg/Itgj1Vl+O9YgOifuAtOH1WbSCi15SsKC3EutKsNQlabZbSiUHXoIFzs6rrkBY31Rsx2uKm7q92oeiNwoMVq2hB2VlwWQQxAsEuiN7v93+58CCAofX7/T8yEcaijj7/LiOy4vuBM+8vJOJZ/lzO9hYu06rbyBKrXOutW7fcX8mqx15orvMbrqxwGQFJ9cy3AXQ2X7WPKwH6uts8GTsgqfGzV9UliH6/n980Yo0j9J64jMi6jSPKqDrnEIzsBgUkf7A4vdGoXfejuLA+uWBTmwGE1Rvya2kPX8cH/+2f/exnYypbqOCjSqKpFOsrr7ySTyQ1eC2TBgmuL1jE8oP5qF+L+6GVnOGymTkXOljsDWLpFVmRLMvOmmd3Hwalwa8mLObzxvl1GJQ7K36Jxlh9VeqejebTwp6ujerfmqbpsRqUTdZmi54mojGy5fV63cQsSZI/rsG5edMtbsZYNNJmA7tQdLBu2bpuMbeCBWX3e3X8/U1KQWnsY/uuAtibI/oM6bMUI0teYwB33bVZb3UZq8xrs4TGUrGuT02lyipeNn0Vx2vZlS6M3JuvlqXV0QgqK/wlXUPKCEhSlr4TVvxqLrEDkt4aYYgA9yGNI7RRONY4wt2n6ziO0KbLmL2OfQcK30GbXVuc3lDgrnULuovTG+OU7wPCGeUmpMXavWQNafE4Vkm9/VImjCaOmkAmSRJzVK5s0J/0er2XNDgJ+X4oKKjj+MILL7yRJEnsIEeUzCSdW5oY9fv9n6ZpSjCyGrMxS2C6HnDW6zXMftV5d2RsbOzMxYsXo2wc0GdVE5MXXnjhq0mS1HrjmJ6nXdi7WGWPSE9e+SPW7lXv9b5dow0RytL7Sq/Xe1nnjj4/ITeB6Dqu6659b2vxmr2S6VUtKBub5fLuxMTEcWVHlb1BCsCu8vtnrAoqWqT3+tXW6gKgvs1Zlv3DyZMnD+raVEYriybwFsGrKPsvD5IkuawNu5pXhs468Y4zwUjsxwNdQz777LNP5+bmfknnVazr6N27d91fZ1m/bTXXXuKjpaWlMR330G18nIFWB/u9N+fzylgbsL1xxNfrOo44ffr0QV0DYo8jCEZ2T+sCkjYQqQ/ydOFBANEpk6JNk14FR7ydz38YO/MnSZJv6NocepLqMlzm5uZchku0636/3/9amqbBB26uV2Sapt8uPIiy5It7sRbevUF5nYI9gxRsPzI1NdVTgN8LngZjP6tjdhJUdj/CobjNAXaTRi0WEFSST9eeGOenNqXY1/uTyJtSRmazA29OTEx8SbuOXWZxYMdt1nvV2a9+3+Cqj4PO++Na4A59zmmhRjuSAy2mAJ2TZdnvJ0kS5X7gl4lO07RW9wPrgZ2z/EDjqjJ6ddWdNo14m1iq3Dyl+8aSjkmZJfCAET147rnnVKnqh9qIp7WdGJtQvcARAfT2u2OrylzTtU8bemO0e/EC3PvOto05r3TjCNt6qI6B+FLHEZRp7aZrbSnZSiASQGi6+XqT1zKCAje1uK8bfugeYF6mYsxr/rz6eioQGXrSQonW6vX7/d8xkTINtFBUs96DO5lXuRlNpGKU2vImPbXt8e0+j3aXf+UBqtildFQm3PzbhpE6vF7fHTvp1s7ad+3ENsZX5T199Hmz1RfWa5CN66oZBL9fx+rHC3TEIVVxUInWGJ+lqstED+mmzd6OUsWhaWo0h9D44bo23oXuP+6d65S8RAgaW349VisAxwYkWb/thhV3X/I2FgblBbjH9/nvRptXujYnVp3XPDbHEQQjEcsVG8hrLAKRAELT5L2C4Ih2IX1HA//QN30NpuwArRdxB2KUslgDZfnqUA6yi9R/6UWVaI3B9Uxwvdhq7l6SJJf0OY3RS0HBSPVcVWZH4cGasKU7TY02B0QrpaOsSJcFU+PNEA/sztoTNmgY46vyLF1tWrBqcRySJPm+secIgNrILxSxshlqUCZ6WPncpayeT3Vm53M/rUmmef4cvPtZEAEX4QHniusdF2uco/kOOkUbeh/pvhSrd6633rUf0cYRA5n6nR9HGIKRnfaFyJkyURGIBBCaFhvOnj2rYMBju/O5zEyYvFRDjB1jXi+1GMFVBTh7GgCGLtVLVmQt5MGeyIt77zaoJOGysqIUlIsxmdJnKEmSg3Xc4V6z7DQnnzTeunWr8MB+uetPmqbUVKuYl0lSl+tE/jxCZ7gM9LwBMJr8c+OVawvGW5CrY3nWQfdcFl6XS7W6IEqapn9deLAa+X3j8OHDNXk6wPbcJlFXISS0gb6RaD/XOzdagCtQgDs/H2O0n2riOCJ2diTByG6rbSmwnRCIBBCaFhVdllaSJMcq2LF0z5VECL1r1su4eiPCztl8QOUFPIPQ8fCykuidVZ0yBuV1L8/6DFuyM1p2pFW7ybm3O7o2/SyzLPuKrm1eECcI1xvTZlSwGaJiXuC/LjuJ8+fxyiuvFB7YD8q0AnuXZdlvmwjjFY1H7captQZV6cjH5rFK4jVBDccs+bljzyWg7vKS9JqLx9h8yXink64oO1LrXKHbHAzYcyWwfr+f3zhCjyN0P/I2YDOOsAhGdlvjassTiAQQmhYa3nnnHaNdxMaY1ysMfuXZkaH7vGjAryCHzbgKuQllPMuyryoYELpEq3sPyEqqVpZlR+/fvx+8b4I3KG/S4p6zEmsyVeedwt4CWl0CQkeSJBmLESh3G0LSNP2rwoMonbcxpS5YVAZqxvWLDK2hVTp0jVrX/CZWSby68zaL1KYcnu5joTPqgVhcJluXNzUgqM22RKE33ptnq+TsORjp+kWGxjhiawQj0RgEIgGEpsHQQCCyykFCtF2IN27cyP/s9/vnCg/u3byCAaEDkQrw6Lgo4ENWUqUO6fhGHpRfKTzYALEmU3qvFdzv9/thU64CqOHCXr6hLkZJvrr1KES+QPBPNXsb1iIGI+n9BYwmWolW735Qm6oAQ8rvXzEWfZvAzutMnTa81fA+Buwkv4bQHxsB1fm+lD+pGOsedawuNKSox4tgZLfVvXHqoBUCkQBCUUDk4sWLdQlE5mKVgNTASl/a8RUqKz7LsrMmQolWHRcdEwV8Cg+iTFGDPerN2sBBuRNtcG4/py/WLSBRw4W9fOdrjMzIXq/nSrQ2bZyMBvMWQBpXuQaoWH4/UCWHkLQx0KviUGYf+RDy8VVXg5E2A3Gt8ACAYT1wmVERy2qy+apb7mizeU0D3Pm5GLoalFfq/d3Cg/UXdRxBMLLbGlMabXF6Qwt/xwsPAMAeqD/k0tKSC4jUIhBp3YxVAtILGobo0XdCJbHW1taCD9q8cjCNzJprkXxBPPQOQQ3KbbD5bwoPNke0yZQX/K1VQMIu7K0XHqhOlB2s7pimafrXhQeBiOihBOxZHowMvXnKu8c3sXf5PW2qUbUVANij/NoXer7jbSRk81XHJEnyf2sdoIYlq/NzMfQm18OHD7u/NnGDa9RxBMHIbmvEwNoGIk8VHgCAEWmXs0qWKuClYEKSJF+uWSk+1dO/rEGaV8oyCJVTVQlI9XkMsBNx3njlX0NRELaBDb7bKl/cC71A7pX7bGpWZC7WZMp7v/fc8yKi2mSG9Pv9XzYRzk9v0tjExefW6Wq/MwAjiZJd45VibuT9IE3TvIkW11Gg3rys7roF5/JrH71OEVAelPPmW63GOGJ7BCO7rfYfCAKRAEJ57bXX8uCZK92TJMmv1HSXUh4cjdEw8VfO0wAAIABJREFUXgFJ9QFUZmPhweEpSHJcE6fQu8ea3kuwZaKUwfQmtE0vgZmPoUJPprz3u47ByNpQyekYfT1adH62QtMn8QBKETujoamb4/L7WMT+tgAC8KoM1a1sab4JkQ0NCCgfz9fwvhSl4o63CZtxxACCkd316YX1yVp/IAhEAghBi8vvvPOOOXv2rOt7dt4OOOra/0XX5uu66Yeu0e5Ktfb7/XOFB4eXl3nVexqSJjrq1ZZl2U9YeG4vb0Lb9GAPi3wVU6Z3aC06PwEA++C1S2jqImL+vLuSgQIguHw+TjASodX1nIrYMoFxxACCkd1V64VeApEA9kuDnOXl5Twb0tY6XzPG/IYxZrkBb26eHalszpC08/Ljjz/Os4r2WApmPMuyNxQEWF0NextxmaBJknyj8CBKl2XZUa9sUDD6LKr/QAuOaD4493Y8BuG953G6xbdDtKxRe37+uPAAUBJXghgA9ikfp9iNmJ3hLXLXddMpAHQVAe5miTaOIBjZXbXd8U0gEsB+KBNSQcj33nvPzM3NucDH63ZxvynZLqtaENfCeOg+DSrVai0UHtzdiSRJDurfCLlzTLvPVaI1y7LHTe8l2BYq5xsjGGk+L7H5d4VvNk+UHY5eqSZsLw9G3rp1a9sf2I80Tf+J9x5V8DYLAaiYxt/qL89xaBavYgUVDgAAlWnRJuzgCEZ2Vy0zIwlEAtgLF8xS6VBlQvpByDRN/xeXadgkaZpeMhGyIxVIVGZjlmVfHbU3RZZl3zReuddQVI5WO66SJPkeO5mBevFK1QEA0BlPnz79R442AAAYldbc+v3+P/PGFRGM7KaHF9Yna7dTjEAkgFG4AOSlS5fMhx9+aJaWlvwSe40NQnpWtCNbgdXQwQAFJJX5pkzHwoPbm02S5KW1tbXg2VunT592f21CCV2gU7zsbHq5AgA64ZNPPjG/8Au/8L9ytAEAwKgYR2zvwLaPoM1qt5hEIBLAbrQgrubJ+vPo0aPPlC9VFmSapn+lAF6apq0py5MkyWVjzJL6KSrrMxRlNurfVKZjkiTDBmzzsq5emdcgdCxtSaW1Bjf3BgAAAOpipOonAAAAnmjjCIKR3VSrYCSBSKC7lPE32BNRwSljG1vrS4Eqr/+Hb91ez1oVgByg6+OSK0EbijIb1ZtqZmbmJdtLc7f7gnq0HVcPwdXVsLcQvTbrSuFBAAAAoDrjDW0hcMTYzAwAAFCZRo8jbt++XXhgvwhGdlNtgpEEIoF6OnfuXJ6FuB319xsMIkbmsubu2WtYV8oF6vVen5iYOKWgXcisRP1bKmtrjJkf4v3MsyJD94pUsNn190zT9GbhB1Apfc5DU++E559//tfSNOXgYq/yydxO9ygAQLdoTBmyjcCtW7fcOPlIQ+cd2khoHj16VHgAAKribZCY5SCgJvK5JeOIgkOF7wRCMLJ7atMvkkAkUF9a5LU3zp2s7fDYKAZvzHe8nUP0KPs8OzJKMFJB57GxsVM22Ljdbq3xLMveePLkSfASrQpESpqmFwsPomprU1NTvdDPQRPQmZmZFwsPNE8+gb57927QJ65JkLXd5xGf3yOC99I19vyMcd4DAKLRXKGnKiohFxFVDcRq9CJijIwGANgrNkighjS3PM44ooDMSARTiw8AgUigMZ7jUFVu9enTp387MzPzm8pGDVlu6caNG+b06dPGZkduVyb1RJIkBxWIDD15eO2119xfh+1biYbTAN9udBimPHCdRck48EpSt7X0dK0pc9c6RA9bAGiO0BtUvM1GszuMkeusR4lWAACGwzjiGSotOx1rHJEUvoO2q3xxi0AkAIzmwIED3zbPBu+C+OCDD/J/Jsuys9v9e1mWfdNEKNGqTE9bBvQ6WWC1tFmuJCRvQHuk4e9PlJ2CMbL92kilnYfI3h+ZyulYTT8/AaAr8o1NodtHaLyiDSpZln2l8GD9nTD0iwQAYBiMI4ryKlCxqisQjOyeSrMQCEQCwJ6saPFdZU1DBoeUpba2tmaSJHlpm74Ns3rs448/DlqyQk6ePOn+2sTd5l2Qb17yMvWC8II9W51vjZFl2e+bCAN0bxJEZuQO0jT9u+0f3bsWBcsBoCui9RFeXV3VGHnMBfca5IR7/gAAYEeMI4ry50swEkFcWJ+sbERKIBIA9i5N0zw70vVZDMXrAzm/xT+pXpJ5OdeQFHCxQZc1gi61lZeoDD0ob/gOQeeQC9KHRs/IoeXXjaNHjwb9R70JV9MmjK3kHV/uEwC2k18fQmc0mGeDeY26J2jDlMZaBCMBANhVlE3YprnjiPEsy74acxxBMLJb1qp6tQQiAWDf8gzC0KVaNcCwzbVP2drwjnqmHddjoQch9IpshDwYGWtxz+4Q3CoA3gTzJlLGgReMZAVxZ1GC5eoBarMjp11fUNQCwXkAO1nTImLoUue6z9tewoNj5DqbV693ApEAAAxtTesejCNyJ7RWE3McQTCyWyoZkRKIBIAgtBh7XX0W1W8xJC87csH7Z/O/v/POO0F/lwZ4yu7MsuwRwchai9I7weyejVt7/X7/awOvIxj1Qez3+z9u4vtSsmiZMK6XLtmRANAYUbLlzbPVQRYKD9aQ6/UeuqoJAAAtxjjCcuOI0OuAPoKR3VJ6MJJAJAAElWdHev0Wg3CL7y7IYkszvBGjNIN77kmSXC48iFpRUCxGsEelMG02bq+B2WfzaZq+qM+MsuhCcpOfNE3jNGdol2jBcnfNy7LsbOFBAEAd5RfuGIuI3/3ud115+XMNyGqYd2XkvR7IAABgZ4wjPpePI7TWsbGxUXgwFIKRHVJ2v0gCkQAQ3B1XQiLkQEkDDQ04FGSx2WoqzXBQmV+hAy5eVidZkTWXpunf6BnOzs4Gf6LeTrvlwoM15nYKxsiK9EqOUlttOFHK6eh6qIVcTcR0+hd+AABQN/l9M8Z4ReNgZTXY8vJXCj9QH+P9fv9P9WwuXrzICYoY8pLpoUvkA0ANRAtGNmkckWXZn5nIWZGGYGSnrJf5YglEAkA0+QAmdKlWN+BQsMUFXLSLKyQtEtnG4NddzzfUWrRBuYJ5Xq/SpgR8FlzGgbI7Q/MWUQlGDifa4vPVq1fdXxsVLAeAjlKQ5F2NMb3ey8FoPNyAMcuyNhXquZIViUjyMoahN4F5jhS+AwDleOA2unZ4HHFFCQl6rjGzIg3ByE4pbWGLQCQARHUzy7KfqO9iyIGSnw2kr7W1teCDkNdee839lazIZogW7JFLly7lf/b7/T8vPFg/4y5Iv7wcPj6lhR31i9Rnm0D90G6aSOengs26HtpSwk0Jls8G/mJREECT5GMWjY9DU1aDu/dnWfZeDcusqarJGS10xs5mAEK7deuW+xfrXr4QQLtFm1s2ZBxxqqxxBMHI7iglGEkgEgDic/0WQ/eO9LKB/EbbQShwqmCL+hCS+dUYUTMN1JtPAZ80TV9uQAbainYKanAeY6egm/QkSfL9woPYzp0syx7FyNw13vUwy7K/KDxYH8t6D4wxnxljPgz8Nc+ZB6BB8o1uoSuHONqkomwBjQX6/f5HNVpIPJJlWb6pSwudodsrACUiGAmgSnkwMsamJrP1OKIuNscR586dK2UccaDwHbRV9IVfApEAUJqVLMv+y7Fjx8YUHAk1YHDZQAo+hS5Defr06fzPNE0vFR5EnWlQflyD8hi75LRwpsD32NjYkh2r1DFQvaD3QGXPYu0U9HZgkjU8AgVvx8bGTmnxOXQfT10DlSHe6/VessHyugXM83H3kydP8pJ8CpKHCJTrvbTltDkXATSJ20B1XJtUYpRTV0UH9cubmZl52bZNqHrThhYQP1IfqvPnz0d5zUBstnShsRUZbvKGA6jIPVuqtadyrTFKnmscoTHK1NTUy3auVatxRFll3glGdsP6hfXJBzFfKYFIACjVgyRJvqdFeAUxQi7Ch17QN7YEpZ6nMniSJGGS2Sw6XtcUoIgRiFPwRAHJixcv6vx4P0mSL7ueNDVxwhjzrcePH0cpz2ps1nCv18tLtCZJUqfX3gT5+DP0ddDRMde/W8Ngef66dV5qo0fIiaPKadtrNecigKbJN1BpzBIrMKesAWXOT01NubWPBRsILdvmAuIHH3wQ5R4IlOHu3bvut1AeHkDVNMfqaT4Ua+7/5ptvDo4jqgpIbo4jlLFZ5jiCMq3dEHUxYXF6Y4FAJACULh8duYzDUDQICT0Q0aLQ2NiYspi+U9GCDfZOx+u6MqVi9Y5UuVYFOjUQrlkPBQ3Q/9LYoFSsnYKuFIwrv4yRrCqIq2BujFLCW/T3OFT4oXId6ff7P4oViPSu1ZQLBtBEK/1+/6eh+6r7dF/QQqK99p6qqGTrCT8QGWvBFCiDPksa02RZ9tu84QAqpgpkjzSO0Ib6GLYYR/yo6nGEMjbLRDCyG6Lt4l6c3lAE/1uFBwAAseVlJBQkitUzLRSvt+UVzopGitqHSRSM1EA4SZKXsiz7HzXYHT3rlyxRwDQW7by0KIu5By6IG6u/h46919/j/QqD5fM6J9VjVeVj9XkMHSCnXDCApkvT9NsmQl91n1tIdH2vsyz7B1tJIbZxO5b+gRufEIhEGyiTWeMsDUU4oACq5OaWZYwjNJfzxhFlXP9qMY4gGNkNUVbQbCDyWuEBAEBZ8pGDF8yoHQVKbf+xd20AFc0TNfvM0UDYC0h+VOGChMY3H7oBesySJS4TTdmnZA3vmTJh/kXXwVg7WLVbVOemJos2C6bMDMlxGxy89uTJk7HLly/nZQJD9Qp2/HLBNe3d2nozMzOm3+//uOvvA7BPV5TVoPtrrHuCsQuJyk53m1W0sGfLxMa6P8zb68MZ9djTIialWVE2ZR6rl1po3rlMdB1A1a7EnlsaO47Q7/DGER+WNY5QNnqV4wgFI3uF76JNPr2wPhl88ZdAJADUQilBov0g66sdkiT5holQFniQApJ2QD5mB+RlLkpsBn00QI8diDTPvp8svuzdgzRN/0pB3Zg7WF2w3O5eLSt794gNfp7S4rPOlxs3bhR+KASvXPB3Snhd2Eaapv+09SMAhqS+6pd1T9DGjdi0WUULerpGq1+lMebv7Vgi1GLivM2YuJam6YsaI2lsHasnJrCTNE3/zm6iC0pVKGy1h15JWcYAsB3NLS/Gnls6ZY8jNJ+N2Vt7GGRGtl/wnc0EIgGgPsoKEu3FQKbNTU6bRoveh8nRgPytt97K+8cYY5ZsH4XYgZ8TdqfgZtAndiBSi4k2a/g6WcP7tpklHnMHqxcs1y/5YeQg8oJfllWvLVbfUvPsxhHKaQNouiuu51MZm/W0oKeFPZWct2OXU3Yx8Y6u5SMuKI7bYEzet0rrLqoYoZKwuk5rjBQ6Mx4YQV7FI8bnyvUss/3aq27XAKDb8nGE1gQaPo74Z38coaCn5rNVjyMOFL6DtrkT8vUQiASA2rmZZdmfzc7OHtQifJ0WKNxOMld3H82WpumfaAygAWzs4Ld2SGvRTb9rZmbmZRv4uW6DPyEDd7P23+ylaZoHmjQJiP050meVrMigdE5cHxsbO6Xrjo5hLFosU1BQGTdjY2NLWZa9YTeFhMr+PmIDgr0nT57kryVWNqRDuWB0WFdKEi+EXheouQf2uvytMsYsjq7XGkeo/67uRVNTU9N6DvrSomaSJB/b47DVdVa9qv+9FgzdN3QP0HhI94CYm1GAEej8Pa5SrRsbG0HfNy3GqyrJ0tLSWL/f/+923sEGqZbxyvyyERN1tjmO0JyvjEoLZvdxxOMkSW63YRxBMLL9gk2wCEQCQC1poPSdsbGxM1pQjr1oPQo9HztookRrO6w8ffr0azMzM7+pAbIGtjFpkUMLiDqP9OfExMQpu0twzQZ+bm4zEN/NuM2EPKesM/2sdgpevXq1tHIlej02+PM2k/FglEn41ddee21M5WdCL5L5lDWrc0UT016vpwnftSzLvmk3Xqzs8byctf1KdY7n5+TFixdLmTgSGEeHdaVlzXjhO+2nnk9/pA1NZYxZHG1m0j1CX8qm0O9W//SjR48eHBsb6+10ziVJkl/77969m99jynrOwAjyTQ0KKMU4P11VknPnzv3i2NjYt/r9/ltpmn7bjvm7tKGitbwKJsx/UHf5OKLX69VlHDHWlnEEwch2e3hhfTLIDZtAJADUmnaNntHuqboEI71Mm++RadMeBw4c0C7lDxWE0QC3jExcNxj3gpJuEH7NBiZX7de9bSa2h+zXrP3KB/DKhNRgXZ+ZMgfqmkzos2qzJAj+hKONGZeUrVjGDlYFO/U7dDxV8mZmZuYlt3PVGLPuLZw9GNjBOu6VH9P5eCTLst+2pV/zfiHaFRu7TLCjz1UDygV3MZCCiOpY2j4Gvc6uvNatpGn6n1RZQdmRutaVXT1E9wmNMdzYXIvwXlbQM7TxhPKraIB8ffGVV16JVoXCbfjStWtubu5FtWywX8aO+80OSRduvLXd4wAwtDRN/1hrH3UZRxi7lrAVzSFjbsYNiWBkuwW5AROIBIDay0sUKnOszF1bO/GafRNsaRedXG9PTEyc0SKB6+9SBheU1EKeJgM6173A5NKwT0EDdX1G9G+VXa5EC5GazJjPdy7+IYH64JZVNlXZimVdC92CmXau6rqnCaItqTNd+OFtqISO/h2dk2Vev+teLtj7fB6h7zCAPVBg4m1VD9G9t6wya9vRImZZFRiASO5lWfYTbcCK2R5EC+r6zCrg6bKCNP63436zU2aQ+bzvpDb8/S5BSQD7tFqncYSxc8+mIxjZbvvOiiQQCQCNodKAp9Rnr+pgpJsw2t2rlGBpn+V+v/8HJ0+efLGK8h8KUCgIqi8FgHSu6evw4cN+6Z9NWihRqRL9f/qqcsegzezUX98luBKHDfKWvoNV55ULzrvsF7dzVRkEvlu3buX/5c7Hqnp4KHha56xIsoQABLDQ7/e/ojJrGiPXqZ0B0ERJknxfFXkUJIxdxWGrrCCNr7Ya7xs7B9XcYG5u7mCWZe8lSfJLhR8CgNEsM44Ii2Bku+1rdZBAJAA0ymq/3/+xeuNoklZlg2oFACx6RbbTgzRN/6ML+GhQXlWAT79XX03oq6TPhYI//X7/p2mazhd+AKFUvoPVZb+4nauxSpnthxbrFBy3fX0XavcEASAQlWvNsuyjs2fPjmkzSJVjZKAFNL87o3FtWSXlfTt9fv2MIQUkbTl8siMB7McDxhFhJW16MXjWhfXJPd90CUQCQPOkaZqn5Sg4VBW7GzUPuBCMbDWNMc6rL+jFixe33aGMz2mDgAuKpWk6R3nW6LSD9ce9Xq/S62GdeeWC/zPnI4CWu5Mkifo+5ZtDtuvbCGAod+wG2HzeV0feJslDHFIAATCOCIhgZHut7fWVEYgEgMZaUY8MBQOrCg7pd5vPAy7fLjyItlE0410/0IYifRY1aVHg1hjzeogy+tiV28H6+OzZs0wYBygjUouIT58+/VtjzJXCDwBA+6zYrPl8MwabqIC9cxtgvb7TteJlSBKMBBAK44hACEa2156yIglEAkCzJUly2dheYFXwSrSywN0N89odrSC0y7TCv9Ek5erVqy4QeZ1s4VKxg3ULeh9cedYDBw78x+JPoApedglZqkA8Kkl9XddB3ZtZSAT2bEVVcDT+Z3wFoEMYRwRAMLK9Rt51TyASAFohDwJWUZpQgciJiQljgy4sqHaDMtC+7DJy67pDugouEGkXafSZoE9k+djB6nFZuubzjSsK1N4r/BCGkc+zjh49GuzNsvdOQ+Y0EN08C4mju3//vvt/jjToaSOiNE3/RP/6W2+9xdsMoEsYR+yBN444RDCyvUbKjCQQCQCtoSDgdS2+e1mKpfB+H9lf3fIgSZL/TZlWCkZSsrUQiHyXQGSl2MFazNJ9m+v0vrDZBmi2ZxYSyezandeDb7zGTxPlWlG5d5V9pz83gI5hHDEiv5cvwch2Wr+wPjn0JJlAJAC0Tp4dWWapVg3ANBlVyc69lgpHo6kkpjIkH+u863LJVpVbJCOydjq/g/XSpUt+cHyh8AMA0C2b9wVljIfMdAa6QuXe6c8NoKMYR+wRwch2GnoRmEAkALSSyrytaWBU1qDI7YhN0/RS4UF0hQtI/kQlWxX86FrQR5+5GzduDAYiyaKqh87uYNXmAG+zCMFxAPjcvCvlrftCW7K7NPby+tACMd3z+3Nz3gHomNaOI2Ku4xCMbKeheo0QiASAVsuzI8so1aqByuzsrFHfQEr/dZ4Ckr+uoEev1+tU0EeTDwUivTKYBH3qp1M7WHVt1jmpzQH6TKq/K8FxAHiGMsVfd9ldTQ+o6L7mrvtASVZci5CLFy/SPw1A1xTGEU2+Dmoc8f7770ddwyEY2U67ZkYSiASA1rvpMtRiL6oo4KkJaJIk3yk8iC56YIMem0GfsvuXlkmTDWWBavKhSYgmI5TBrDWNgc+3bQfrIL9cMIFIANjRiio76FqpLHIF85p2b9BYRD27dd2fmJgoPA5EtrnZy6sQAgBd8cw4QsG8Jo8jNE8+fPhw4WdCIRjZPp9eWJ+8t9OrIhAJAN2QJMllU0LvSO/fv1J4EF31wC5MfP3555//l6WlpVaWbVVGsCYbygLV5EOTELKDG2HZ38HatnNT56VfLphAJADs6k6apv/BbVbRvUHX0SZk0GvDl56rxuO2SgnlMlGFPCCpYLg2ImosAgAdUhhHNKUSj8YRWtPw1w1ttacoCEa2z44lWglEAkCn5EGRmJNB/dt2B/a76htS+AF03ZU0TX/LlW1t4i7BrWiRTwEslaNyZVltwGfHcRhqZXMHqzs3m57B63a0euflefqWAsBItFnlV13vdWUI1HUxUc9Jz00bvuxY/O0kSX7FEIxEdfKNiK5kq3pWU7YVQMdoHPEbGkcoS7Ip4wiv1cyr+guZkRjFtiVaCUQCQOdoAXpNCxSxFiW8xXuyIrGdzV2CypLULkEFfprYr08LKqdPnzbvvffeZjakHbAvEPBppGd2sGoipglZE8uLuWxILzPmVTsZBgCMRpvrtJPvVVdyTYuJusbWYdOKWzzUc9Jze/r06d8yFkGNaE74qmsX0paNiAAwgjt2HPF7uhb6Qck6jiO0ZuiNI/K4UsyNJAQj22fLYCSBSADorPy+EKN/jAKcCshogLXd/QfwLKdp+rKyaHU+1nmX4CAXhNSCiv60wZ6v20AW537z5TtYtaDr+oVpN38TMkvcZFIZCPY6f91mxnBeRkbmEdB6q/Y+/3suU1KbVlZXV/N7RJllKDUOUUBH4xC3eGg3RP3egQMHfotrPmpmNUmSX1eWjStX6MbQ3DsBdMhNOy973c0z3ThC1WzK3AC7wzjiVRs4LW0ccaDwHTTZwwvrk4XyYAQigdbKd77qpvLo0SOOMkrnasq73pTAEJRtcMIOeJdnZmZ6Ggzfv38/D6hoYF6n65kmCBq0a2e3qMegMeZSkiRXyD5onTt2QXc+y7Jvzs3NvaTj/sEHH+STttu3b9fq9SoI+eabb7rdrPlkMk3TP27TgrRXHqhun7Wbxpglt0EBQOvdtF9HlDXwwgsvfHVubm5M94jHjx/nYxfdI/S1sbER7L1Q0EbXegU9tfnPs2Y3eBGARJ09sFk2GjMvT0xMnNJ9U19ra2v550WfnZCfGQCoqZUDBw6s2DWQeY0jTp48Oab1NK2DuDFEV8YRBCPbhUAk0C26cRzXgF69y4AyKQiuEhMKziRJssKbjxGt2sG4vhYmJiaOa5egvhT80eKEvqqgQbsG7JocuIxiZUIq6E4QshNW7DXtmaDkJ598snluVrVwts25+ZMkSb6RpmnrrsNe1lHdFtzv/Ou//uvPJiYmvqj7IAHJ+GL2rWkzMpCC03rLfJIk83Zj1Qk/MClaVNT94u7du/mf2mA1zGYWbX7S2FqLhjrf9d9+VRN7rf+O7QdPj3b4n++6j0vv2V6Sy3Zs9Uav13tJC+PKmPQ/M/qs6DOj/waAFtKcRpnjC24c8cUvfvEroccRLghZ13EEwch2eSYYuTi9cYRAJNBquoksnzx58gu73aB0Q9tu8VQ3qsESnrZ5Mdph1p0DIWmR2J4n3+t6cOb/b+8OYuO6z8SA//2egMCKhAjoIRDdrBWgUC5pydC6tVvS2T0astLsxQK2ktvAuuzWiuUTDyuqB56sgEZPNnYbCwXknrqyrWsRCW1vllaD7iWCgUgbhEYOxcorI7l4xsU3+j9mxCEpinqPfDPv9wPGkvmGFN/jzPCb//f/vi+eP7EpoO6v2RE38u1YJCX7/f6fvfLKKy9U1QbVrum6dwmOqoL26rahXcrNkeTUxIhqU57ZelIybsePH1+IRbO4VYnJTz/9tPEFs3g8njhxYlidu8ljc7Uoimtjn7TP4jn1NG2HNsYvVUVyNQvtwIEDYxsu99s3vvGN/55SejOSkZu9Nm0Xd42q2lRLtm0vnnfsv6qqiaFruf3a2ZHNVYvf/va3Xzp69OihDRUI6+J3RsQ3Kb/WPeE9183878TC5U5fB4+MfaTl4ndpVzR0rq37HbmFWPxeLopiOVcZV8+ZP9nsORPPkyrGGr1u1eJ8kLhsXrxG1TlSYy/inUl8H1Qlj+rQtZgy3idlk9Qt4EFez433m2k0jjh69OhCrAVtFUfcvn17/e8NxRGNi2Tk/ZTSi235hngm65ntnIjUtgOm24O8m+YXFgbYQixILOx0UfRpVC1a8y7XTmsiGdlBEcOcL8tyfZfg888//6cLCwsvVIF4tSgRCxLVY3p0QWInNu4YjP/fJGlyc6Ql26RVH9zL339TX7uLPsi3SJif6vf7//H48ePfr5Ijo4/L+LN6XO5G9biM15R4Y73xDWZuxfo3bX5s1rnQGhXJBw4c+IuxA+2w3O/3/2R+fv77kv+N8n62Hs/6evHA75ZtVRurUl5UPJaTLXM5Fp+rPjk2tlR/7/f7v0kpfZb/90FOKt3Lf+520XD4b0VFRZ0idmpAkwujbXuEY2AnAAAWmUlEQVRcdelcd6J6jK9u8pwZPm8OHjz40vz8/DAIqlrSP0mV7H/C4vwzi++nK0n0iEUnKM5p8n1Qo5uvY9NhVRk37Zp4/uSqv7GPT5AbG2LeuZHXxfU4ot/v/7N471Hdqck4ou7ROSNrLg8O5G9SMnI6DB9oI4nIb3X9gkAHxHP9u7lyYztz2+yUvdP1yrYpFomd2lvIjVSP3dQqajjwm3oNE4FlWabRndPPP//8ifn5+Re2WpAY3SU4KhI7T6g07Y28Abgx4a+HVeKM+sVr3WpZlqtVYjIelwcPHvzh/Pz84c0el1s9JkfFgtkmCfGh/Abz02pHa1mWbX693fiGd6dJpMcW6kdcy5WpbX0+PijL8o9z/LVZfLVd3DXq3obfo5Jvj9u4OMP+uFN12mBHquf1tpXrOc5pRIOLiHU+H8+PfWR6delcd2PsOTOSWBiNEzbGDMfybWg02d+gppJdbaxwvTT2kfo08bt9Et8H3WvwOrcxfmrk+dOmqr+aVMnEx+KIjXFDQ3HE8P1Lg5ua7hzIJ7cXL9g0745EJPvIQsH+uac6jc30+/2fRIAS7QTrFG3pMkkPr31NW985nYPtIxuqDYaLoxt3CY6KuabRCTJ/6N6Gm58fuzFMTG6xo3+4MNbv9/9FJM93+LWrN+bV5qB4XN5refJxoy4utD7IjwOAthjGRVpXMkUebIjXt03yZ9X7hSbiqC5tzLDG1LyureXZ2NR+ww0fdY95GnUgv5C/OXaESfNFXvyQiGS/TNtOFJh0Z8uyfCESkXW2aI0WgtFCJCp2yrKUjGSvPdisUma7XYET3rKFyTC2o3+7xyQANGS4iFj3eIaRGWRd74jCZNiYwARgZxqJI0Zmot4pVnozNxosM2fvPJCIZB/dX+nN7GSHGrBHvvrqq5+kBlq0VrMMyrL867GDAADAvhgMBj+sZhnXaWQGn2QkAEypfr//p3Fmt27dqvUER9q0DmdGptxWRyJrspn7yX4yAwHaZfHAgQP/OuaV1R1EjLRo1ZoOAIA2GauGGgwGLxVFcehJ32O/3//7siz/39iBR1XvbZ5hW5kriuJw3bF/OHHiRPVXyUgAptXcdutcT4gntiv0q9rxtv136Fx0V7t5c7tT2Z2RDgvDmZFppTcTswYXJSSBXXhdVSS0ztnUQFVkJCKPHj0af70yAQsyAAB0yGYLhZ999lmKasEn2Wr2dEppYTAY/OeiKP79DufV7ZfhBuEbN8bysc/s+PHjwxncRVFIRgIwrWL82MJW5/bb3/5201mK0T3g+PHjW35ejiN+XBTFX+bNTW01XEdsMI54WBTFemXkaEIyLsrs2GcBPO5+vFDlVs9Ae8T84DMRJDWRjMzMigQAoFWKooiywIWTJ0/WMu8oYt8LFy7EQmP0F/vb2Ijb0jj4SCx0/u53v6s9/o958blNa/0llwDQIrlLwvdHOgI8s9OnT6e33norfpH+PFdHtnEdPeKI/xBxRN3JyJE44nb8pxg9GAnJld5MlKT+NKX0xdhnAzx6bbi00ps5JhEJrdRIVWTsZJqfnx8GZy0NngAA6LZhjBpxax0ino6E5Icffjj8aoPB4L/kNm5tcz4qQq9evVr7tzVyLcX/AEy1siyHG29eeuml2k4zfjdfunRp+PfBYPBJLiBom4gjDsf3+vDhw1q/tZFrOYwjirF7PEpKruYL8+7YQaDLojVjJCGXPQqgnQaDwVvRiqpaNKlL7OZKj4Kzy370AAC0ULRYq3URMRblLl++nH72s59F5eWhwWDwcVQQjN1x/xzr9/tvNxH/p8ev5Z2xgwAwXW6kmuOIlDc3vf/++8M4ot/vf9K2OCKldHEP4oitk5HpUULywUpvJnrOf/cJQziB6RevAT9Y6c1EW1Zz4qC9zsZupmirUOdupsOHD6fFxcVhj3ctWgEAaKlGFhFTrmy4fv16LCR+p9/v/6+2LCTGomZZlt+MRc66qxlCvAfIVEYCMO2Gv+tGfvfVJn5P3759Ozb4x4zq1bZcx5wcTU3FETtORlZWejP3Vnoz8RN4Oc+IA7ojnvM/iteAaOPs5w7t1u/3L6QcRNQp2lNFj/eiKP6rhwAAAC0VG2d70Vo0ZhTVbXl5eZiQjIXEliQkP4jvJRY3m2jRGtfw6NGjKW9OtikZgGl3L0YTRRwRm/LrFnOo7969G1/1TN7oP9VxRFzHHEd8VH3sicnISsyGixlx5klCJ4zOhbzmRw4TYbEKItbW1mr9fl977bXqr63ZvQUAAJsYdvFooqohbUhIDgaDf9jHGZJxnmc+//zz4eJmE1555ZXqq1oTAKATyrL8n6mhOCIqD8+dO7eekNznjU3DOCK+l6biiChsyNbjiB0nIysj8ySvjB0EpoG5kDCZzqbcQqpO0VJhZCfTPY8NAABabLjgNZJIq10kJGOuUoxHSCn9XXxoDy/HkXyOZ2K+UywgNtFWLW2xiAgAU67RTU1VQrJq2ToYDH6dUjo1dsfmHMktU4eJyPhemoojRq7h7pOR6Q/zJGPR8wfmScLUMBcSJtexamd0zIus0+nTp6uvZlYkAABtt95irYlWrZXLly+nS5cupUgIppQu9vv9/xvrbmN3rNfZXI35aiwgRrIwV1fUzoZEADrqTiQIFxYWGosjIvn3xhtvDEcsFUVxKKX0tzlBeGzszvWq4oiFphORG+KI9TzDrpKRlZghl+dJ/sg8SZhY5kLC5BtWRX7yySe1nkgEXhGA5Z1adkQDANB6ZVn+TWq4OjLl2Ds27lXVDSmlX+TFxLqTkme/+uqr/51S+nlUY0ZVZpMLiOnxqkgbEgHolKIofpb2II6IZORI29aFlNKv8u/dxuOIiF+ajCO2Kmx47uuvvx67824tza5Fa4rzKaVvNXYmQF1iLuSqdqww+QaDwT9FQPHyyy/XGkxE26c8L/Kn5kUCADAhogXZP0bVYlNt1jaKCoCInaMiMz2Kz39dFMX/yItwu9n0G9/4qX6//2dlWb4QH4ik53vvvZdu3bo1duc6xYbEjz/+uDqHP2r0HwOA9tnzOCI2AUW1ZK4mTNHlIW+uulFnHPHOO+801lWhsl0cUWsyMj1KSB7JC5Znxg4CbRFzIc9rxwpTIaoif379+vXh/Jo6RcvXgwcPflkUxXdG2yoAAEDLRRLwTLRSrbt7yHYiKRnVANFdpDIYDB4WRXE7Lyje26Lt6Vxe/FzM1RHrIs6Pc2g6CVmJ9xS5GuR1lZEAdNS+xBGRlIzb/Pz8+scGg0Gsy93KccSDLZKTizmOmGtzHFF7MrKyNLs2l5OSC2MHgf0ScyGXV3oz9Q6VA/ZNzKeJtlCx6FHn7qYIfi5evJjy5oWzY3cAAID2irlLv4qZ6iMtR/fM4cOHh9UUcYsE5aFDh3b8T8f3HAuGcYvNgU22Uduoqmbo9/u/Kcvyn4/dAQC6YV/jiPh9XMUQkxRHRIeIq1evbhlHNJaMrCzNrp3KSckXxw4Ce+V+TkLa1QjTJTb+/F20Woh2DnWK4CG3mfrBFruuAACgzYZVDTGTKW77KRYVo/Xa9773vU0XFGPhcG1tbbi5cC8XDTeK65SrMcaqGQCgY1oXR1R/bjQpcUTjyciKeZKwL77ImwFWtWSFqdRI24hqJ1Oupt6bBvkAAFCvYzFz6fe///03o4tILNKxtajAiFlSeU7Vv9zyjgDQDeKIp1B1WPvqq6/+z4EDB/7NZp9ZjH2kISu9meVc3nqlFVcHpl881+biuScRCVMpesGfiYHadfevjyArG9vFBAAAE+JeWZbvRCXihQsX/My2EW1lq/nzZVn++db3BIDOEEfsUFRsVtfowIEDf7HVZ+1ZMjI9Skg+WOnNnM0t326O3QGoQzy3Xo7n2kpvZrPB+MB0iG4DVQVjbWIhIgZNR393yUgAACbcclQ1LCwsjG64Y4PLly9X7WMvGdEAAOvEETsQG5p2EkfsaTKystKbubPSm4m2bz/Ks+yAZxfPpdfjubXSm7nhesJ06/f7P4kTvH79eq3n+dprrw3/LMvyr8cOAgDAhIlKv8Fg8OVbb71VzURnRFQyxHynWGyN9UTXBgD+oOoY8MYbb4gjNvE0ccS+JCMrK72Zayu9mWM5Y/rF2B2AnfgiP4eiJasqJuiGU2VZvhCJyLp71keP98zrCQAA0+BOURR/Gefx/vvvD1uJ8UjE/rEZcTAYPCzL8o9dFgAYE5V+r0flX8QR0VGMR542jtjXZGTFPEnYNXMhoZuGLVrrnhW5uLiYjh49mvJrizbPAABMi9hodyUWEt955x0LiXkB8eLFi7GA+GVRFP82pWRNAQA2tx5HvPfee+KIXcYRz3399ddjH9xPS7Nri7mcc6FV3xi0S8yFXNaOFTopNu/86vPPPx+tYqxF7PCK1goxdzal5PUFAIBpE4uJZ+7evZvefvvt2ruMTIpqATF7XVcUANiR9Tji3Llz6eHDh528ahviiBjFeG3sTptoXTKysjS7djYnJV8cOwjdFS1Zz2vHCp02DHwuXbpUa2VktKv6+OOPY0fTr4ui+KOxOwAAwHQYxtNffvnlcP5TLCh2iUQkADwTccQu44hWtGndTE62zJknCeviuXBMIhI67chgMPhxBDw3btRbuBg93kNRFH81dhAAAKbH2arV2tWrV9Pp06c786NdXl5eb6mWKxmsLwDA01mPI6LDWN1dy9rsWeOI1lZGjlqaXTuWqyTPjB2E6fdRroY0vw2IgOfnH374Ybp8+XJtFyN63UeV5cGDB6PP+3fMiwEAoAOGsXWc5s2bN4fx9bS2bY0uKDEr8/jx4+nrr7/+p+eeey5GI90ZuyMAsFOPxRGRqJvWtq11xRETkYysmCdJx/RyEtLcNmBoMBj8QyQLT548WetCyUiLhSs5mAIAgC6Y6/f7/60sy+9H95GolIyNf9O0mBiVn9FGLio4Yr00pXTK5kMAqMXcYDD4ONbqIo6ISsmIJaZJnXHERCUjK+ZJMuXMhQQ2ExtyfnH79u1hEFCnCJRid1NK6bspJVXYAAB0zXK/33+7LMtvfv7558OEZHQOmeSk5EsvvZQuXLhQxfnhpyml1bE7AgDP4kis5aeUhrv8I46IpGSMV5qyOOJSzsnt2kQmI9OjhGT1Q47bt8buAJMpntSrK70ZuxSBjYYDst9+++1a50VGcPHee++lvLtpcewOAADQDY+NCIoKh0hIRmJyktq3Rnx/7ty5ND8/X33oZu5+YtMhADRHHPEEE5uMrOR5krGz69WxgzA5zIUEthMbcP4xdlfVPRh7pCry5ZSSttAAAHTdcDFxMBj8uCiKYU+yu3fvpuvXr6dPP/10+Pc2WlxcHLZSqxYPB4PBr4ui+E8ppWtd/4ECwB56LCmZxBHrJj4ZWcnzJCMpOTt2ENrrfuwsMBcSeILhUOzYTXX58uXt7/kUYrj2K6+8klRFAgDAmCN5LtL50bWm2CAYnUriduvWrbFP2kuxqTA2K8YC4tGjR4f/cr/f//uyLC/nzioAwP6YyDgiJyH/qok4YmqSkZU8T3JV61ZaLuZCLq/0ZsxrAHbiTgQuJ0+eHLZ2iJYJOxVBzsZ2EKO7nQaDwcOiKP6Vtk0AALClY3nz3qnRzlzRgi0WEuO2F9UOMzMzw/cC1a1aOMyu5IVDm50BoF22jSMiMfnLX/5y6uOIqUtGpsfnSV4cOwj7792ciDQXEtiJCFh+1cSVyrum/zwnOwEAgCerKh0WB4PBvyuK4nD1GbGoGAuJkZiMv8fC4sOHD596cTEWC2OBsPrzxIkTw+qFQ4cObbzrR7l9WtysMQBA+z1VHBGetoJyszgi/tyQfEx7HUdMZTKyYp4kLXMzt2RVfQQ8jbP5NurOUwQJcznQGXVvJNgAAAB2by5XOyz2+/0TZVm+sN1XikXGWGDcaItk47rBYPBlURS3csXCDRWQADAV1uOIwWDww9Hk5GaiA1rcNpqEOGKqk5EV8yTZZ+ZCAgAAQDccyQuL1abAY/lWWdjsKuTxCbdHPnRj5M97xioAQCfsKo7o9/u/Kcvys5EPtS6O6EQysrI0uxatW5fNk2SPmAsJAAAAAAB0WqeSkekP8yQjIfnm2EGoTwx8PW8uJAAAAAAA0GWdS0ZW8jzJD7Yqa4VdupmTkHdcQAAAAAAAoOs6m4ys5HmSkZR8cewg7Nz9nIS85poBAAAAAAA80vlkZMU8SXYp5kKurvRmll1AAAAAAACAx0lGjsjzJFdTSmfGDsI4cyEBAAAAAAC2IRm5iaXZtbmclDRPks3EXMjlld7MjU2OAQAAAAAAkElGbmNpdu1UTkqaJ0nKcyEjCfmBqwEAAAAAAPBkkpE7sDS7FvMAz5sn2Vlf5KT0qpasAAAAAAAAOycZuUPmSXbWR3ku5L2uXwgAAAAAAICnJRn5lJZm1xajVad5klOvl5OQ5kICAAAAAADskmTkLi3Nrp3NSUnzJKfLFzkJaS4kAAAAAADAM5KMfAa5det58ySnxiVzIQEAAAAAAOojGVmDpdm1Y3me5KsTfzLddDOldNZcSAAAAAAAgHpJRtYoz5OMpOTs1JzUdLufk5DmQgIAAAAAADRAMrIBeZ7kqtatrRVzIZdXejOrXb8QAAAAAAAATSpc3fqt9GY+SCkdyzMIaZd342cjEQkAAAAAANA8lZENy/MkIzm5MNUn2n4xF/L8Sm/mTtcvBAAAAAAAwF6RjNwjeZ5kJCVf7MQJt8f9nIS81vULAQAAAAAAsNckI/fY0uza+ZhXaJ5k42Iu5OpKb2Z5ys8TAAAAAACgtSQj98HS7NqRnJB8s3Mnvzeu5GrIB104WQAAAAAAgLaSjNxHS7Nrc1G9Z55kbWIu5PJKb+bGlJwPAAAAAADARJOMbIGl2bVTOSlpnuTu3M9JyA8m8ZsHAAAAAACYVpKRLbI0uxatW8+bJ/lULuXZkFqyAgAAAAAAtIxkZMvkeZJRJXmm69fiCT7KcyHvbX83AAAAAAAA9otkZEstza4tRutR8yTH9HIS0lxIAAAAAACAlpOMbLml2bWzOSnZ9XmSX+S5kKtjRwAAAAAAAGglycgJkFu3xizJix29BO/mRKS5kAAAAAAAABNEMnKCLM2uHcvzJF/tyCnfTCmdNRcSAAAAAABgMklGTqA8TzKSkrNTeor381zIa2NHAAAAAAAAmBiSkRNsaXbtfJ4n+a0pOaWYC7m60ptZHjsCAAAAAADAxJGMnHB5nmQk796c8FO5kqshzYUEAAAAAACYEpKRUyLPk/wgpbQwYWcUcyGXV3ozN8aOAAAAAAAAMNEkI6dMnicZSckXW35m93MS8oOxIwAAAAAAAEwFycgptTS7Fq1bz7d0nuSlPBtSS1YAAAAAAIApJhk5xfI8ydWU0pmWnOVHeS7kvbEjAAAAAAAATB3JyA5Yml2by0nJ/Zon2ctJSHMhAQAAAAAAOkQyskOWZtfOxpzGPZwn+UWeC7k6dgQAAAAAAICpJxnZMbl16/k9mCf5bk5EmgsJAAAAAADQUZKRHbU0u3Yst259teYrcDOldNZcSAAAAAAAACQjO25pdm0xJyVnn/FK3M9zIa+NHQEAAAAAAKCTJCMZyvMkV3fRujXmQq6u9GaWx44AAAAAAADQaUXXLwCPrPRmPkgpHcuzHnfqSkppTiISAAAAAACAzaiMZEyeJxkJxlNbVEpeydWQd8aOAAAAAAAAQCYZybaWZtfmUkpHqvus9GZubHd/AAAAAAAAGEop/X/v3BJVuTshzwAAAABJRU5ErkJggg==){height=\"60px\" width=\"240px\"}](https://pytorchlightning.ai)"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Tutorial 9: Normalizing Flows for Image Modeling\n", " :card_description: In this tutorial, we will take a closer look at complex, deep normalizing flows. The most popular, current application of deep normalizing flows is to model datasets of...\n", " :tags: Image,GPU/TPU,UvA-DL-Course\n", " :image: _static/images/course_UvA-DL/09-normalizing-flows.jpg"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab_type,id,colab,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10"}, "papermill": {"default_parameters": {}, "duration": 25.941431, "end_time": "2022-05-12T12:19:23.098784", "environment_variables": {}, "exception": null, "input_path": "course_UvA-DL/09-normalizing-flows/NF_image_modeling.ipynb", "output_path": ".notebooks/course_UvA-DL/09-normalizing-flows.ipynb", "parameters": {}, "start_time": "2022-05-12T12:18:57.157353", "version": "2.3.4"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"033751fff3854e7fba6e527fc3ca1af0": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "0a53e8d36171453087899d7b05d0dab4": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "0bf513f5ede34bf0aa6095e88896ee01": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_f26ee6645b204d699c8d6a262817a8ef", "placeholder": "\u200b", "style": "IPY_MODEL_101cb19c084044df8919210038562038", "value": ""}}, "101cb19c084044df8919210038562038": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "111dcbf1598644c6b5fe894526a16bc9": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "1474b267cbb442cf939e8c460db28ed9": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "1df07eff5dec46f7b8867701f6c9b5f8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_abdfdb58fdeb414cb9bd9b21fcd9fa99", "max": 4542.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_1474b267cbb442cf939e8c460db28ed9", "value": 4542.0}}, "218074dccea7473e816a8461d3bec333": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "22731f4074314260bf3a8474c377ec03": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8b8edc12175c48719a46a007eb575ec4", "placeholder": "\u200b", "style": "IPY_MODEL_4edbc364f3154c35b86cec4dfd3fe383", "value": " 62%"}}, "245169ab03414848bc71b05cf2129307": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "26c84ae6074a49cd95b93cd701457f0d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_0a53e8d36171453087899d7b05d0dab4", "placeholder": "\u200b", "style": "IPY_MODEL_a1dfc3dd292a4c3f9a440d34dd2bafa9", "value": " 0%"}}, "2739cef7a26c4d36abe27140463f1536": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "280b89d79e1f40bab2d9ca74d447480d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "2b08b3d74913422d99ad29d9b9b39904": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "2b2f05b4a3d44430b7343f7c7dd96d6b": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "2e0f9562244644ccac867ccd8b85d15f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_9bd7ad04b5bc4639bdd0eef00c52e3a9", "max": 9912422.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_9dac662d19214eddaae33c0e40a93cef", "value": 9912422.0}}, "316381f85ead4a8d865c45dc0aaf4c74": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "432fa993cc574f50963c5a558841af95": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "44514b44ec4f42baba6c3b8a0a05cf69": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_316381f85ead4a8d865c45dc0aaf4c74", "placeholder": "\u200b", "style": "IPY_MODEL_280b89d79e1f40bab2d9ca74d447480d", "value": " 1649664/? [00:00<00:00, 48914787.24it/s]"}}, "453d7b4525594a01a1023e7484e02c8c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "4762650b38ca460ebf7a8b7eaf2d2125": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "4edbc364f3154c35b86cec4dfd3fe383": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "560c48ca02ef41c09de8dc7078c4bf24": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "5a571cfc40ac454b8583b4b978c5f5a8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_b845e4639c67471f86e22528305643ca", "placeholder": "\u200b", "style": "IPY_MODEL_e8041c5ef7e74a60b5527813002dcc5b", "value": ""}}, "5b1adf06f69f4853b5475a5eb0b8ed55": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "5cc75509b1e64556a1358f725e34afd0": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_4762650b38ca460ebf7a8b7eaf2d2125", "max": 8.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_245169ab03414848bc71b05cf2129307", "value": 8.0}}, "5cd28bfb90d94e71b16c3cba25520349": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_ca3de77b07124ce386d85a29236f3178", "max": 28881.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_218074dccea7473e816a8461d3bec333", "value": 28881.0}}, "5e798f310d4b42208f60488f4b25167b": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "5ec66f1159e941a0b8a03d507ab59716": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_111dcbf1598644c6b5fe894526a16bc9", "placeholder": "\u200b", "style": "IPY_MODEL_2739cef7a26c4d36abe27140463f1536", "value": " 5120/? [00:00<00:00, 253232.04it/s]"}}, "6512366631a949678ad0cfd766ffba0a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_8b23d7d073dc4fadb963d7611e5bdfda", "IPY_MODEL_2e0f9562244644ccac867ccd8b85d15f", "IPY_MODEL_8e7de690e370429aadb271d44c5dca88"], "layout": "IPY_MODEL_e0dec845c63045a59b58b11cb729b68a"}}, "75ad285edbad41fe83fdf09c4c8cf6c0": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8e0c54ee985f4ea3820c13e8b58a3f2d", "placeholder": "\u200b", "style": "IPY_MODEL_453d7b4525594a01a1023e7484e02c8c", "value": ""}}, "75f3db6cb8e14d1296d489ca26b01f24": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_0bf513f5ede34bf0aa6095e88896ee01", "IPY_MODEL_1df07eff5dec46f7b8867701f6c9b5f8", "IPY_MODEL_5ec66f1159e941a0b8a03d507ab59716"], "layout": "IPY_MODEL_ed18b5e93d8745a68a5041277a459d9d"}}, "7928f983aa4344859749c3737df8c87e": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "80a80e415bb24e459eef8028675f66ad": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_5a571cfc40ac454b8583b4b978c5f5a8", "IPY_MODEL_f77de1e20aa44f558370b6ab4ab63b65", "IPY_MODEL_44514b44ec4f42baba6c3b8a0a05cf69"], "layout": "IPY_MODEL_2b2f05b4a3d44430b7343f7c7dd96d6b"}}, "8a5581805ad545c6a4cc26e86a64d636": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "8afbd845be84491384ba3254f348da55": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8b23d7d073dc4fadb963d7611e5bdfda": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_91acf3221c8a4ac2b3468f63e45358ae", "placeholder": "\u200b", "style": "IPY_MODEL_8a5581805ad545c6a4cc26e86a64d636", "value": ""}}, "8b8edc12175c48719a46a007eb575ec4": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8e0c54ee985f4ea3820c13e8b58a3f2d": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8e7de690e370429aadb271d44c5dca88": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_8afbd845be84491384ba3254f348da55", "placeholder": "\u200b", "style": "IPY_MODEL_2b08b3d74913422d99ad29d9b9b39904", "value": " 9913344/? [00:00<00:00, 102366614.04it/s]"}}, "91acf3221c8a4ac2b3468f63e45358ae": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "91eed8e58a8341e9aadfd56c1abbf3c1": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "94d70f1ca8af4a318c07f34a8ad16a23": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_ecd7d7859e40414dad266aff3cf9908a", "placeholder": "\u200b", "style": "IPY_MODEL_432fa993cc574f50963c5a558841af95", "value": " 0/8 [00:00<?, ?it/s]"}}, "9764df840c324bd79369088ec1295004": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_7928f983aa4344859749c3737df8c87e", "max": 8.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_c3b21e9f688e4d9fb9abb4b2bd35520c", "value": 8.0}}, "9a751f24d3c04ef7b8e5f4763ceccf4e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "9bd7ad04b5bc4639bdd0eef00c52e3a9": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "9da52b7df5e8458e9237d8e64f99813c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_acfcf1a6a77a4f84ba076fa320e440f5", "placeholder": "\u200b", "style": "IPY_MODEL_a46709408d804941a191d4f4e94cdaff", "value": " 29696/? [00:00<00:00, 1513047.27it/s]"}}, "9dac662d19214eddaae33c0e40a93cef": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "a05e28af0506439f8843dceab189d0a8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_26c84ae6074a49cd95b93cd701457f0d", "IPY_MODEL_9764df840c324bd79369088ec1295004", "IPY_MODEL_94d70f1ca8af4a318c07f34a8ad16a23"], "layout": "IPY_MODEL_033751fff3854e7fba6e527fc3ca1af0"}}, "a1dfc3dd292a4c3f9a440d34dd2bafa9": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "a46709408d804941a191d4f4e94cdaff": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "abdfdb58fdeb414cb9bd9b21fcd9fa99": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "acfcf1a6a77a4f84ba076fa320e440f5": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "b845e4639c67471f86e22528305643ca": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "c3b21e9f688e4d9fb9abb4b2bd35520c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "c49cd5fd7f434bd1bc57f2ce1bada81a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_91eed8e58a8341e9aadfd56c1abbf3c1", "placeholder": "\u200b", "style": "IPY_MODEL_e6cb260f2d2e4d9598aa26eded04b62c", "value": " 5/8 [00:00<00:00, 49.34it/s]"}}, "ca3de77b07124ce386d85a29236f3178": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "ce6bcc8f63c346b4a80e0c245eb89704": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_75ad285edbad41fe83fdf09c4c8cf6c0", "IPY_MODEL_5cd28bfb90d94e71b16c3cba25520349", "IPY_MODEL_9da52b7df5e8458e9237d8e64f99813c"], "layout": "IPY_MODEL_5e798f310d4b42208f60488f4b25167b"}}, "d60486171a554fbfb03474a308153d0d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_22731f4074314260bf3a8474c377ec03", "IPY_MODEL_5cc75509b1e64556a1358f725e34afd0", "IPY_MODEL_c49cd5fd7f434bd1bc57f2ce1bada81a"], "layout": "IPY_MODEL_560c48ca02ef41c09de8dc7078c4bf24"}}, "e0dec845c63045a59b58b11cb729b68a": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "e6cb260f2d2e4d9598aa26eded04b62c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "e8041c5ef7e74a60b5527813002dcc5b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "ecd7d7859e40414dad266aff3cf9908a": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "ed18b5e93d8745a68a5041277a459d9d": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "f26ee6645b204d699c8d6a262817a8ef": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "f77de1e20aa44f558370b6ab4ab63b65": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_5b1adf06f69f4853b5475a5eb0b8ed55", "max": 1648877.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_9a751f24d3c04ef7b8e5f4763ceccf4e", "value": 1648877.0}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}