{"cells": [{"cell_type": "markdown", "id": "c331ad54", "metadata": {"papermill": {"duration": 0.030587, "end_time": "2021-09-16T12:41:51.358101", "exception": false, "start_time": "2021-09-16T12:41:51.327514", "status": "completed"}, "tags": []}, "source": ["\n", "# Tutorial 9: Normalizing Flows for Image Modeling\n", "\n", "* **Author:** Phillip Lippe\n", "* **License:** CC BY-SA\n", "* **Generated:** 2021-09-16T14:32:34.242172\n", "\n", "In this tutorial, we will take a closer look at complex, deep normalizing flows.\n", "The most popular, current application of deep normalizing flows is to model datasets of images.\n", "As for other generative models, images are a good domain to start working on because\n", "(1) CNNs are widely studied and strong models exist,\n", "(2) images are high-dimensional and complex,\n", "and (3) images are discrete integers.\n", "In this tutorial, we will review current advances in normalizing flows for image modeling,\n", "and get hands-on experience on coding normalizing flows.\n", "Note that normalizing flows are commonly parameter heavy and therefore computationally expensive.\n", "We will use relatively simple and shallow flows to save computational cost and allow you to run the notebook on CPU,\n", "but keep in mind that a simple way to improve the scores of the flows we study here is to make them deeper.\n", "This notebook is part of a lecture series on Deep Learning at the University of Amsterdam.\n", "The full list of tutorials can be found at https://uvadlc-notebooks.rtfd.io.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/course_UvA-DL/09-normalizing-flows.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/latest/)\n", "| Join us [on Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)"]}, {"cell_type": "markdown", "id": "cb8074e8", "metadata": {"papermill": {"duration": 0.029722, "end_time": "2021-09-16T12:41:51.416816", "exception": false, "start_time": "2021-09-16T12:41:51.387094", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "4c53849b", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2021-09-16T12:41:51.477802Z", "iopub.status.busy": "2021-09-16T12:41:51.477304Z", "iopub.status.idle": "2021-09-16T12:41:51.479831Z", "shell.execute_reply": "2021-09-16T12:41:51.479366Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 0.03456, "end_time": "2021-09-16T12:41:51.479947", "exception": false, "start_time": "2021-09-16T12:41:51.445387", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# ! pip install --quiet \"seaborn\" \"tabulate\" \"matplotlib\" \"pytorch-lightning>=1.3\" \"torch>=1.6, <1.9\" \"torchmetrics>=0.3\" \"torchvision\""]}, {"cell_type": "markdown", "id": "03ad92b7", "metadata": {"papermill": {"duration": 0.02881, "end_time": "2021-09-16T12:41:51.538089", "exception": false, "start_time": "2021-09-16T12:41:51.509279", "status": "completed"}, "tags": []}, "source": ["
\n", "Throughout this notebook, we make use of [PyTorch Lightning](https://pytorch-lightning.readthedocs.io/en/latest/).\n", "The first cell imports our usual libraries."]}, {"cell_type": "code", "execution_count": 2, "id": "6dcb9f29", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:51.604711Z", "iopub.status.busy": "2021-09-16T12:41:51.604228Z", "iopub.status.idle": "2021-09-16T12:41:53.319528Z", "shell.execute_reply": "2021-09-16T12:41:53.320210Z"}, "papermill": {"duration": 1.753622, "end_time": "2021-09-16T12:41:53.320362", "exception": false, "start_time": "2021-09-16T12:41:51.566740", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["/tmp/ipykernel_3359/964175757.py:27: DeprecationWarning: `set_matplotlib_formats` is deprecated since IPython 7.23, directly use `matplotlib_inline.backend_inline.set_matplotlib_formats()`\n", " set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "Global seed set to 42\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Using device cuda:0\n"]}], "source": ["import math\n", "import os\n", "import time\n", "import urllib.request\n", "from urllib.error import HTTPError\n", "\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pytorch_lightning as pl\n", "import seaborn as sns\n", "import tabulate\n", "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", "import torch.optim as optim\n", "import torch.utils.data as data\n", "import torchvision\n", "from IPython.display import HTML, display, set_matplotlib_formats\n", "from matplotlib.colors import to_rgb\n", "from pytorch_lightning.callbacks import LearningRateMonitor, ModelCheckpoint\n", "from torchvision import transforms\n", "from torchvision.datasets import MNIST\n", "from tqdm.notebook import tqdm\n", "\n", "# %matplotlib inline\n", "set_matplotlib_formats(\"svg\", \"pdf\") # For export\n", "matplotlib.rcParams[\"lines.linewidth\"] = 2.0\n", "sns.reset_orig()\n", "\n", "# Path to the folder where the datasets are/should be downloaded (e.g. MNIST)\n", "DATASET_PATH = os.environ.get(\"PATH_DATASETS\", \"data\")\n", "# Path to the folder where the pretrained models are saved\n", "CHECKPOINT_PATH = os.environ.get(\"PATH_CHECKPOINT\", \"saved_models/tutorial11\")\n", "\n", "# Setting the seed\n", "pl.seed_everything(42)\n", "\n", "# Ensure that all operations are deterministic on GPU (if used) for reproducibility\n", "torch.backends.cudnn.determinstic = True\n", "torch.backends.cudnn.benchmark = False\n", "\n", "# Fetching the device that will be used throughout this notebook\n", "device = torch.device(\"cpu\") if not torch.cuda.is_available() else torch.device(\"cuda:0\")\n", "print(\"Using device\", device)"]}, {"cell_type": "markdown", "id": "3cd3ed62", "metadata": {"papermill": {"duration": 0.029414, "end_time": "2021-09-16T12:41:53.382047", "exception": false, "start_time": "2021-09-16T12:41:53.352633", "status": "completed"}, "tags": []}, "source": ["Again, we have a few pretrained models. We download them below to the specified path above."]}, {"cell_type": "code", "execution_count": 3, "id": "78d63583", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:53.446715Z", "iopub.status.busy": "2021-09-16T12:41:53.445099Z", "iopub.status.idle": "2021-09-16T12:41:54.071021Z", "shell.execute_reply": "2021-09-16T12:41:54.070611Z"}, "papermill": {"duration": 0.659621, "end_time": "2021-09-16T12:41:54.071141", "exception": false, "start_time": "2021-09-16T12:41:53.411520", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/MNISTFlow_simple.ckpt...\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/MNISTFlow_vardeq.ckpt...\n", "Downloading https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/MNISTFlow_multiscale.ckpt...\n"]}], "source": ["# Github URL where saved models are stored for this tutorial\n", "base_url = \"https://raw.githubusercontent.com/phlippe/saved_models/main/tutorial11/\"\n", "# Files to download\n", "pretrained_files = [\"MNISTFlow_simple.ckpt\", \"MNISTFlow_vardeq.ckpt\", \"MNISTFlow_multiscale.ckpt\"]\n", "# Create checkpoint path if it doesn't exist yet\n", "os.makedirs(CHECKPOINT_PATH, exist_ok=True)\n", "\n", "# For each file, check whether it already exists. If not, try downloading it.\n", "for file_name in pretrained_files:\n", " file_path = os.path.join(CHECKPOINT_PATH, file_name)\n", " if not os.path.isfile(file_path):\n", " file_url = base_url + file_name\n", " print(\"Downloading %s...\" % file_url)\n", " try:\n", " urllib.request.urlretrieve(file_url, file_path)\n", " except HTTPError as e:\n", " print(\n", " \"Something went wrong. Please try to download the file from the GDrive folder, or contact the author with the full output including the following error:\\n\",\n", " e,\n", " )"]}, {"cell_type": "markdown", "id": "98588147", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.02986, "end_time": "2021-09-16T12:41:54.131396", "exception": false, "start_time": "2021-09-16T12:41:54.101536", "status": "completed"}, "tags": []}, "source": ["We will use the MNIST dataset in this notebook.\n", "MNIST constitutes, despite its simplicity, a challenge for small generative models as it requires the global understanding of an image.\n", "At the same time, we can easily judge whether generated images come from the same distribution as the dataset\n", "(i.e. represent real digits), or not.\n", "\n", "To deal better with the discrete nature of the images, we transform them\n", "from a range of 0-1 to a range of 0-255 as integers."]}, {"cell_type": "code", "execution_count": 4, "id": "47598513", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:54.196645Z", "iopub.status.busy": "2021-09-16T12:41:54.192801Z", "iopub.status.idle": "2021-09-16T12:41:54.232017Z", "shell.execute_reply": "2021-09-16T12:41:54.231591Z"}, "papermill": {"duration": 0.071068, "end_time": "2021-09-16T12:41:54.232133", "exception": false, "start_time": "2021-09-16T12:41:54.161065", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}], "source": ["# Convert images from 0-1 to 0-255 (integers)\n", "def discretize(sample):\n", " return (sample * 255).to(torch.int32)\n", "\n", "\n", "# Transformations applied on each image => make them a tensor and discretize\n", "transform = transforms.Compose([transforms.ToTensor(), discretize])\n", "\n", "# Loading the training dataset. We need to split it into a training and validation part\n", "train_dataset = MNIST(root=DATASET_PATH, train=True, transform=transform, download=True)\n", "pl.seed_everything(42)\n", "train_set, val_set = torch.utils.data.random_split(train_dataset, [50000, 10000])\n", "\n", "# Loading the test set\n", "test_set = MNIST(root=DATASET_PATH, train=False, transform=transform, download=True)\n", "\n", "# We define a set of data loaders that we can use for various purposes later.\n", "# Note that for actually training a model, we will use different data loaders\n", "# with a lower batch size.\n", "train_loader = data.DataLoader(train_set, batch_size=256, shuffle=False, drop_last=False)\n", "val_loader = data.DataLoader(val_set, batch_size=64, shuffle=False, drop_last=False, num_workers=4)\n", "test_loader = data.DataLoader(test_set, batch_size=64, shuffle=False, drop_last=False, num_workers=4)"]}, {"cell_type": "markdown", "id": "a20ba18c", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.030353, "end_time": "2021-09-16T12:41:54.293285", "exception": false, "start_time": "2021-09-16T12:41:54.262932", "status": "completed"}, "tags": []}, "source": ["In addition, we will define below a function to simplify the visualization of images/samples.\n", "Some training examples of the MNIST dataset is shown below."]}, {"cell_type": "code", "execution_count": 5, "id": "9f49ed4e", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:54.360403Z", "iopub.status.busy": "2021-09-16T12:41:54.359918Z", "iopub.status.idle": "2021-09-16T12:41:54.446277Z", "shell.execute_reply": "2021-09-16T12:41:54.445797Z"}, "papermill": {"duration": 0.12252, "end_time": "2021-09-16T12:41:54.446381", "exception": false, "start_time": "2021-09-16T12:41:54.323861", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDMzNS4yOTkzNTQ4Mzg3IDE3Ny40OCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjrEOwjAMRHd/xX1BYsdpk45FlSLGsvABVQVUtAgqwe/jMgDDSb6T7XuCiXwrOK1gTKYXBAW+G5+XYTyUHYaV2PKZVCsXmkaryuz130pKLmbL+DeeiRa6I7nwkQZ2eduOWXOC1Oo44zHiiAW+DRuAGIAYAKPYpQbBVix1/H4ZZvi9oLuhp57eJg8oJwplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0MgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjIyICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vJycnHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKwsLCvr6+urq6tra2srKyqqqqpqamoqKinp6elpaWkpKSioqKhoaGgoKCenp6cnJyampqZmZmYmJiXl5eWlpaVlZWUlJSTk5ORkZGPj4+NjY2Li4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKAgIB+fn59fX18fHx7e3t5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5qampnZ2dmZmZkZGRjY2NiYmJgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NRUVFQUFBPT09MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj48PDw7Ozs6Ojo5OTk4ODg3Nzc1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4sLCwrKysqKipcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDMyMSAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTY0IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMzIxID4+CnN0cmVhbQp4nO2d+19M+R/HZ6ZRKWFGJRoUWdEWspVLqnHLJSqXZV1qI7llv+7azTJZwmZlKXeKWLnLZWklconUisxmhPlbvu85M6dpTufymflMzNr384c8zmVe8/48M5/z+ZxzOiP7H0KH7HMX8K8HDdKCBmlBg7SYDRqdxn8vDg3SxqFB2jg0SBuHBmnj0CBt3JdhsLa2dvTo0adPOymOFDRIi6MGHz16VFrapUsXuVwuk0VHR3d4jVI76fVarRaqmTHDKXHkoEFa0CAtDhnctGnTpEmTFK3ExMR0eI2ie9TV1cXHyxny8ujj7AIN0oIGabHDoMGg1+uLi4ujojw9PRVt6d69+86dHVyj6B5Tpkxh9Pn6+hoM9HF2gQZpITX45MmTtp9bW8g/xfn5+TKZLCODZ5PB0NLSIlyjaHWDBg1iDO60+UUKIB23Z8+ekBBoFyQmJUHJr15JxaFBW9CgOJ/FYE7O5MmThfwpFAUFBUQ1vnoVHh4O++/fb7P62rVry5cvV6kiIiKKi4VqFKnOYhDG94cOHRLdTzqusXH27NnQz7uZgFKZf8PCSktLRePQYCvONvjxY319/dq1a728eLx17Xr+/PmbJj58+EBU49On5lfu2mU06YTsy5cvL1ni7+/PZmZl2dtkBsZgfHy86E5Ecfn5YMzPz++XXzZv3swadHPz8vI6d+6ccBwabAUNEvBJDer1vP3e1KlTU1NTL160t8b9+82v79EjPT29Xz9u7MiRI/V6e5tsNG7ZskWpVMrlixcvFt6JKO7x48chIeDroqlpVVVVB0xotVpGYlBQ0MuXQnFo0Awa/FcYhMFzXt6bN28cqlGvT0pKgkGHNa5z585abUZGBrOQxdsLShisqxs+fDgznK6trRWtSjpOp9MpFMuWLbMZ2sPCli09evSAAtPShOLQoBmnGmxuboZVzNyGRa2OjIx89uyZdSf4X/+S73+2aJPhAJ6b+5UJOMiXlJQYjXPnzlWYDu0wc7SzyUBhodyC2WBR0UETR44cEXqFQFxlZaVarQ4IgEZxN715M2DAAHiH1FShODRoRIOuZxAmgjb9X3BwcFkZ9H+72gJd2sSJE3ftOnr0KHmTuZw4oVKp4B2Ep7SicVFRrMFJkyZNnz7dzY1ZgKktLPAenQXilixZwkyBeV5RVmaeJP/2m1AcGjSiQVc36OMDs8L8/OTkZO7ohgEmtgcPkjaZyzffMH2suqZGaA9Cgzz06lVRUUEYl5KSAp50Ou56mPTPnAlbhg4d+u6dUHVo0Oh8g+YhJItSqdFo2o6E29G1a3l5OVGTuTAGly5dKrwHoUH4PQa0xcfHRyaD3oc70uaNa2zs06cPeGr/Bps2bWJmdcW8Z3/RIAsadD2DUJGILz7OnDlD1GQbioqKunUbNWpUU1OT8E5EBmNjX79+bbOlrKzMzw+2VFdXS8c9f24+lWpzuG1oaMjMdHd3Z7aIVocG0eB/1CCMNLp166ZQzJC4a5LI4E8/cbfU19eHhvr5+XEnusKjGWgFO/WtqSksLJTJZJbGZfDeJGCNQ4PGT2QQpkrDhg0rLHz69Cn4io1tsykg4PLly0RNtrJjxw7mtYcPHxbdj8hgQoLNpxgW4uPj/fzOnz9PGLdq1Sr4rHp7j2Xw94cF+P2mpo4YMcLNDQ0SxKFBVzPIOwVWqVS5JlauXMnd9MMPpE1maWiIiIiAV44eree9QkcY9+237IgafhHM+V+YyJaXjx8/HlYNHkweZzAYFi5cqFabj8mengkJCcxFdn+TzN27d4tWhwaNaND1DHLPsIoyZMiQykrSJrNYOkGFRCcoFffqFXsTNRAXF5eeDh7Y5Zwc++OYy+wHzp41Lz9/Dv9t1OoHDx6IVocG28Q5y+CVK1c0Go20vMDAwLNnDZwb6IkMJibCy3v37l1VJbWnRFxzc/PAgQM551Zh2KDT6d6/tz/Olt9/h4+06O0kaFAcNGh/nC2OGwROnTol4TArK+v48eOO1PjixYuvv4aEcePGSbdCOg4Ow4mJrDx3d/e9e4uKihyPawMzbThwQKo6NCgIlUGgvJzXXN++fc+YeMd78YqkxmvXrjE9AM+lNEfi7II8rrq6WqWCTzEadDQODdLGOcGgg0jHzZo1S6GIjY11UpxdkMfdvn2bmeKhQUfj0CBtnOsa/PPPTp06ma7jK9PT09vf8GhvnJOrs4IGaeNc1+CtW8whvV+/fvfv36ePsxPyuLdv3373nZeX1/XrUnFokB80SBvnuga/vDg0SBuHBmnj0CBtHBqkjcOnytOCBmlBg7SgQVrQIC1okBYczdDGoUHaODRIG4cGaePQIG0cGqSNQ4O0cWiQNg4N0sahQdq4T2rw3r17+SZ8fRUKBfvXk9ynCZPH2fDs2bPw8K1bt5JXd/PmzWHD5HJ5cPC0adM2btz4449Qmk6ny8iYMmVKoum2Tihx+vRFixY1NjYKx6FBNGjhizdYUjJ27NjQ0FBlKwrm7hmlUq1Wjx27evXq5mY74tqzfft2uVylUhFWd+zYsbCwsJ9/3rt3L+8bf/z48fXrkpKSDRtGjBjh7e2dklJfX88bhwbRoAVXNnjhwgXrwn7Og/WFa2xDWVmZWt1Gmo1BdiEzkzSOl4SEBLncw8ODsLrm5mapR8paePfuXW5uLvP3o7xxaFAKaoNRUXBEsi5qNBrII6mxDS9fvuzTBzzFxcXFt8Is9O/fnzXo4cFzJO0wg+QYDIYFCxagQXvjrKBBx+KsUBusqbExCPpslklrPHiQ96/tKyoqVq1aZekU16xZQxrHQa/Xx8TEmB7b2Et0P/sN3rkTHR0Nw+7Zs3kefYwGCXC+QTgWO2RQGDhQW6Z4jhs8efIk87ed27ZtE92PvLq3b99WVq5fv97Dw93dHUarf/8tFIcG+UGDtNU50WBuLhx+bdY43yB1P+hUgydOzJ8/v2fPnnI5/EhLu3v3rmgcGmwHGhThsxiMirKZFxsZgw4+gpsXZxhkv+mFyuCJE2lpacxf/UVFRSUmJp458573gSGcODTYSkcZTE7mroFBtfAJms9jMCIigt5gZKT5gSEpKW2ery0dhwZbQYMuaTAzM7P9iRhY2V4rUY28zJkzxzUM/vPP1atXV6xYERCgVCqDgoJ0uoaGBuk4NNgKGnRJg+0HzzAv1phITk6GjRqNzEKyRapdBpuakpKSmMvukODjw/MlzkRxpaVdunQBf76+Dx8+FN2TKK6uDkyuXLnS2zswMHDbNqFnExnRoBDOMwiyYGhpXYBPr6wVWIBVNTX7rZDXyHLlCnutzsfHZ8wYoRqlYgoKoCAwuHy51J52VXf3bqTp8CxyJwkaFAcNOhrHQm8QeruampoLF1hz0NXl5jJrLgifnSGvsaKiIjiYNThv3jwH41paWlJSQB/0WcInUeyvjqG+vj4yEvpY5gsyheLQoAhokKI6BjqDmZlgMMp0vR1+cOfBjhi8c+fOunXr8vJgHjxnDvzw9fVl/YWF8dwYJRHHAt0UM5geOnSo6H6EcVwaGgICAqKjo4Xj0KA4jhqsYa7QmUbMGs7pVTOc8/5SNRYXF8+bN3DgQN57t8LCwoSfSy3d5NOnT3t5gUEo9a+/RPfkiSP58vLs7Gwo9Y8/hOLQoBRoUCyu4wwy8kRutkxO5l48EaqRXcnOfq1PFWYXBg8enJ8v9EZEHVfv3mBw5syZYl98xxO3dOlSsbvQWGDMAPGmryvhj0ODUqBBwbgONSh1tzQzspaukaG2tjbYMnQWvo9648aNhHFc9u3b5+4OTRQa9QrHTZgwQau9LvqIRhOXLl2C+LVrheLQYMcYdJz2cdXV1bzSbBYcn9WFh4czw+kXL17YWd3Vq1djYz09PaOjMzIycnJyfv21rq6O+4qqqpCQECjw4kWhODSIBr98gy0tLZmZmVZpAwYM+P77Bw8eFBQUxMQoLYNqx+fF7qZekOjbYnjimprOnTuXnR0UFARdXWgoTNXHmNBqtXkm4uLiAgOhvg0bNgjHoUHXN2hkhlT+/gsXLgSD2dk3btywNmA3A82sLikpyXGDFh4+fHj48OFHj27dupWVlWXztVkzZoh8OSsaZEGDX77BLzoODdLGoUHaODRIG4cGaePQIG0cGqSNQ4O0cWiQNg4N0sahQdo4fKo8LWiQFjRICxqkBQ3S8n+rQtglCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMzEyNAplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDkxNjE0NDE1NCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjQuMykgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNDkyNiAwMDAwMCBuIAowMDAwMDAwNjU5IDAwMDAwIG4gCjAwMDAwMDA2ODAgMDAwMDAgbiAKMDAwMDAwMDc3OSAwMDAwMCBuIAowMDAwMDAwODAwIDAwMDAwIG4gCjAwMDAwMDA4MjEgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwNDAyIDAwMDAwIG4gCjAwMDAwMDA2MzkgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNjE5IDAwMDAwIG4gCjAwMDAwMDA4NTMgMDAwMDAgbiAKMDAwMDAwNDkwNSAwMDAwMCBuIAowMDAwMDA0OTg2IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo1MTQzCiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["def show_imgs(imgs, title=None, row_size=4):\n", " # Form a grid of pictures (we use max. 8 columns)\n", " num_imgs = imgs.shape[0] if isinstance(imgs, torch.Tensor) else len(imgs)\n", " is_int = imgs.dtype == torch.int32 if isinstance(imgs, torch.Tensor) else imgs[0].dtype == torch.int32\n", " nrow = min(num_imgs, row_size)\n", " ncol = int(math.ceil(num_imgs / nrow))\n", " imgs = torchvision.utils.make_grid(imgs, nrow=nrow, pad_value=128 if is_int else 0.5)\n", " np_imgs = imgs.cpu().numpy()\n", " # Plot the grid\n", " plt.figure(figsize=(1.5 * nrow, 1.5 * ncol))\n", " plt.imshow(np.transpose(np_imgs, (1, 2, 0)), interpolation=\"nearest\")\n", " plt.axis(\"off\")\n", " if title is not None:\n", " plt.title(title)\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "show_imgs([train_set[i][0] for i in range(8)])"]}, {"cell_type": "markdown", "id": "122fbad4", "metadata": {"papermill": {"duration": 0.031364, "end_time": "2021-09-16T12:41:54.509186", "exception": false, "start_time": "2021-09-16T12:41:54.477822", "status": "completed"}, "tags": []}, "source": ["## Normalizing Flows as generative model\n", "\n", "In the previous lectures, we have seen Energy-based models, Variational Autoencoders (VAEs)\n", "and Generative Adversarial Networks (GANs) as example of generative models.\n", "However, none of them explicitly learn the probability density function $p(x)$ of the real input data.\n", "While VAEs model a lower bound, energy-based models only implicitly learn the probability density.\n", "GANs on the other hand provide us a sampling mechanism for generating new data, without offering a likelihood estimate.\n", "The generative model we will look at here, called Normalizing Flows, actually models the true data distribution\n", "$p(x)$ and provides us with an exact likelihood estimate.\n", "Below, we can visually compare VAEs, GANs and Flows\n", "(figure credit - [Lilian Weng](https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html)):\n", "\n", "
\n", "\n", "The major difference compared to VAEs is that flows use *invertible* functions $f$\n", "to map the input data $x$ to a latent representation $z$.\n", "To realize this, $z$ must be of the same shape as $x$.\n", "This is in contrast to VAEs where $z$ is usually much lower dimensional than the original input data.\n", "However, an invertible mapping also means that for every data point $x$, we have a corresponding latent representation\n", "$z$ which allows us to perform lossless reconstruction ($z$ to $x$).\n", "In the visualization above, this means that $x=x'$ for flows, no matter what invertible function $f$ and input $x$ we choose.\n", "\n", "Nonetheless, how are normalizing flows modeling a probability density with an invertible function?\n", "The answer to this question is the rule for change of variables.\n", "Specifically, given a prior density $p_z(z)$ (e.g. Gaussian) and an invertible function $f$,\n", "we can determine $p_x(x)$ as follows:\n", "\n", "$$\n", "\\begin{split}\n", " \\int p_x(x) dx & = \\int p_z(z) dz = 1 \\hspace{1cm}\\text{(by definition of a probability distribution)}\\\\\n", " \\Leftrightarrow p_x(x) & = p_z(z) \\left|\\frac{dz}{dx}\\right| = p_z(f(x)) \\left|\\frac{df(x)}{dx}\\right|\n", "\\end{split}\n", "$$\n", "\n", "Hence, in order to determine the probability of $x$, we only need to determine its probability in latent space,\n", "and get the derivate of $f$.\n", "Note that this is for a univariate distribution, and $f$ is required to be invertible and smooth.\n", "For a multivariate case, the derivative becomes a Jacobian of which we need to take the determinant.\n", "As we usually use the log-likelihood as objective, we write the multivariate term with logarithms below:\n", "\n", "$$\n", "\\log p_x(\\mathbf{x}) = \\log p_z(f(\\mathbf{x})) + \\log{} \\left|\\det \\frac{df(\\mathbf{x})}{d\\mathbf{x}}\\right|\n", "$$\n", "\n", "Although we now know how a normalizing flow obtains its likelihood, it might not be clear what a normalizing flow does intuitively.\n", "For this, we should look from the inverse perspective of the flow starting with the prior probability density $p_z(z)$.\n", "If we apply an invertible function on it, we effectively \"transform\" its probability density.\n", "For instance, if $f^{-1}(z)=z+1$, we shift the density by one while still remaining a valid probability distribution,\n", "and being invertible.\n", "We can also apply more complex transformations, like scaling: $f^{-1}(z)=2z+1$, but there you might see a difference.\n", "When you scale, you also change the volume of the probability density, as for example on uniform distributions\n", "(figure credit - [Eric Jang](https://blog.evjang.com/2018/01/nf1.html)):\n", "\n", "
\n", "\n", "You can see that the height of $p(y)$ should be lower than $p(x)$ after scaling.\n", "This change in volume represents $\\left|\\frac{df(x)}{dx}\\right|$ in our equation above,\n", "and ensures that even after scaling, we still have a valid probability distribution.\n", "We can go on with making our function $f$ more complex.\n", "However, the more complex $f$ becomes, the harder it will be to find the inverse $f^{-1}$ of it,\n", "and to calculate the log-determinant of the Jacobian $\\log{} \\left|\\det \\frac{df(\\mathbf{x})}{d\\mathbf{x}}\\right|$.\n", "An easier trick to stack multiple invertible functions $f_{1,...,K}$ after each other, as all together,\n", "they still represent a single, invertible function.\n", "Using multiple, learnable invertible functions, a normalizing flow attempts to transform\n", "$p_z(z)$ slowly into a more complex distribution which should finally be $p_x(x)$.\n", "We visualize the idea below\n", "(figure credit - [Lilian Weng](https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html)):\n", "\n", "
\n", "\n", "Starting from $z_0$, which follows the prior Gaussian distribution, we sequentially apply the invertible\n", "functions $f_1,f_2,...,f_K$, until $z_K$ represents $x$.\n", "Note that in the figure above, the functions $f$ represent the inverted function from $f$ we had above\n", "(here: $f:Z\\to X$, above: $f:X\\to Z$).\n", "This is just a different notation and has no impact on the actual flow design because all $f$ need to be invertible anyways.\n", "When we estimate the log likelihood of a data point $x$ as in the equations above,\n", "we run the flows in the opposite direction than visualized above.\n", "Multiple flow layers have been proposed that use a neural network as learnable parameters,\n", "such as the planar and radial flow.\n", "However, we will focus here on flows that are commonly used in image\n", "modeling, and will discuss them in the rest of the notebook along with\n", "the details of how to train a normalizing flow."]}, {"cell_type": "markdown", "id": "3f9b2256", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.030707, "end_time": "2021-09-16T12:41:54.571000", "exception": false, "start_time": "2021-09-16T12:41:54.540293", "status": "completed"}, "tags": []}, "source": ["## Normalizing Flows on images\n", "\n", "
\n", "\n", "To become familiar with normalizing flows, especially for the application of image modeling,\n", "it is best to discuss the different elements in a flow along with the implementation.\n", "As a general concept, we want to build a normalizing flow that maps an input image (here MNIST) to an equally sized latent space:\n", "\n", "
\n", "\n", "As a first step, we will implement a template of a normalizing flow in PyTorch Lightning.\n", "During training and validation, a normalizing flow performs density estimation in the forward direction.\n", "For this, we apply a series of flow transformations on the input $x$ and estimate the probability\n", "of the input by determining the probability of the transformed point $z$ given a prior,\n", "and the change of volume caused by the transformations.\n", "During inference, we can do both density estimation and sampling new points by inverting the flow transformations.\n", "Therefore, we define a function `_get_likelihood` which performs density estimation,\n", "and `sample` to generate new examples.\n", "The functions `training_step`, `validation_step` and `test_step` all make use of `_get_likelihood`.\n", "\n", "The standard metric used in generative models, and in particular normalizing flows, is bits per dimensions (bpd).\n", "Bpd is motivated from an information theory perspective and describes how many bits we would need to encode a particular example in our modeled distribution.\n", "The less bits we need, the more likely the example in our distribution.\n", "When we test for the bits per dimension of our test dataset, we can judge whether our model generalizes to new samples of the dataset and didn't memorize the training dataset.\n", "In order to calculate the bits per dimension score, we can rely on the negative log-likelihood and change the log base (as bits are binary while NLL is usually exponential):\n", "\n", "$$\\text{bpd} = \\text{nll} \\cdot \\log_2\\left(\\exp(1)\\right) \\cdot \\left(\\prod d_i\\right)^{-1}$$\n", "\n", "where $d_1,...,d_K$ are the dimensions of the input.\n", "For images, this would be the height, width and channel number.\n", "We divide the log likelihood by these extra dimensions to have a metric which we can compare for different image resolutions.\n", "In the original image space, MNIST examples have a bits per dimension\n", "score of 8 (we need 8 bits to encode each pixel as there are 256\n", "possible values)."]}, {"cell_type": "code", "execution_count": 6, "id": "f731caad", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:54.645154Z", "iopub.status.busy": "2021-09-16T12:41:54.634808Z", "iopub.status.idle": "2021-09-16T12:41:54.647412Z", "shell.execute_reply": "2021-09-16T12:41:54.647011Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.045385, "end_time": "2021-09-16T12:41:54.647510", "exception": false, "start_time": "2021-09-16T12:41:54.602125", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ImageFlow(pl.LightningModule):\n", " def __init__(self, flows, import_samples=8):\n", " \"\"\"\n", " Args:\n", " flows: A list of flows (each a nn.Module) that should be applied on the images.\n", " import_samples: Number of importance samples to use during testing (see explanation below). Can be changed at any time\n", " \"\"\"\n", " super().__init__()\n", " self.flows = nn.ModuleList(flows)\n", " self.import_samples = import_samples\n", " # Create prior distribution for final latent space\n", " self.prior = torch.distributions.normal.Normal(loc=0.0, scale=1.0)\n", " # Example input for visualizing the graph\n", " self.example_input_array = train_set[0][0].unsqueeze(dim=0)\n", "\n", " def forward(self, imgs):\n", " # The forward function is only used for visualizing the graph\n", " return self._get_likelihood(imgs)\n", "\n", " def encode(self, imgs):\n", " # Given a batch of images, return the latent representation z and ldj of the transformations\n", " z, ldj = imgs, torch.zeros(imgs.shape[0], device=self.device)\n", " for flow in self.flows:\n", " z, ldj = flow(z, ldj, reverse=False)\n", " return z, ldj\n", "\n", " def _get_likelihood(self, imgs, return_ll=False):\n", " \"\"\"Given a batch of images, return the likelihood of those.\n", "\n", " If return_ll is True, this function returns the log likelihood of the input. Otherwise, the ouptut metric is\n", " bits per dimension (scaled negative log likelihood)\n", " \"\"\"\n", " z, ldj = self.encode(imgs)\n", " log_pz = self.prior.log_prob(z).sum(dim=[1, 2, 3])\n", " log_px = ldj + log_pz\n", " nll = -log_px\n", " # Calculating bits per dimension\n", " bpd = nll * np.log2(np.exp(1)) / np.prod(imgs.shape[1:])\n", " return bpd.mean() if not return_ll else log_px\n", "\n", " @torch.no_grad()\n", " def sample(self, img_shape, z_init=None):\n", " \"\"\"Sample a batch of images from the flow.\"\"\"\n", " # Sample latent representation from prior\n", " if z_init is None:\n", " z = self.prior.sample(sample_shape=img_shape).to(device)\n", " else:\n", " z = z_init.to(device)\n", "\n", " # Transform z to x by inverting the flows\n", " ldj = torch.zeros(img_shape[0], device=device)\n", " for flow in reversed(self.flows):\n", " z, ldj = flow(z, ldj, reverse=True)\n", " return z\n", "\n", " def configure_optimizers(self):\n", " optimizer = optim.Adam(self.parameters(), lr=1e-3)\n", " # An scheduler is optional, but can help in flows to get the last bpd improvement\n", " scheduler = optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.99)\n", " return [optimizer], [scheduler]\n", "\n", " def training_step(self, batch, batch_idx):\n", " # Normalizing flows are trained by maximum likelihood => return bpd\n", " loss = self._get_likelihood(batch[0])\n", " self.log(\"train_bpd\", loss)\n", " return loss\n", "\n", " def validation_step(self, batch, batch_idx):\n", " loss = self._get_likelihood(batch[0])\n", " self.log(\"val_bpd\", loss)\n", "\n", " def test_step(self, batch, batch_idx):\n", " # Perform importance sampling during testing => estimate likelihood M times for each image\n", " samples = []\n", " for _ in range(self.import_samples):\n", " img_ll = self._get_likelihood(batch[0], return_ll=True)\n", " samples.append(img_ll)\n", " img_ll = torch.stack(samples, dim=-1)\n", "\n", " # To average the probabilities, we need to go from log-space to exp, and back to log.\n", " # Logsumexp provides us a stable implementation for this\n", " img_ll = torch.logsumexp(img_ll, dim=-1) - np.log(self.import_samples)\n", "\n", " # Calculate final bpd\n", " bpd = -img_ll * np.log2(np.exp(1)) / np.prod(batch[0].shape[1:])\n", " bpd = bpd.mean()\n", "\n", " self.log(\"test_bpd\", bpd)"]}, {"cell_type": "markdown", "id": "bde3e5a1", "metadata": {"papermill": {"duration": 0.030814, "end_time": "2021-09-16T12:41:54.709142", "exception": false, "start_time": "2021-09-16T12:41:54.678328", "status": "completed"}, "tags": []}, "source": ["The `test_step` function differs from the training and validation step in that it makes use of importance sampling.\n", "We will discuss the motiviation and details behind this after\n", "understanding how flows model discrete images in continuous space."]}, {"cell_type": "markdown", "id": "2fbe3846", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.030971, "end_time": "2021-09-16T12:41:54.770996", "exception": false, "start_time": "2021-09-16T12:41:54.740025", "status": "completed"}, "tags": []}, "source": ["### Dequantization\n", "\n", "Normalizing flows rely on the rule of change of variables, which is naturally defined in continuous space.\n", "Applying flows directly on discrete data leads to undesired density models where arbitrarly high likelihood are placed on a few, particular values.\n", "See the illustration below:\n", "\n", "
\n", "\n", "The black points represent the discrete points, and the green volume the density modeled by a normalizing flow in continuous space.\n", "The flow would continue to increase the likelihood for $x=0,1,2,3$ while having no volume on any other point.\n", "Remember that in continuous space, we have the constraint that the overall volume of the probability density must be 1 ($\\int p(x)dx=1$).\n", "Otherwise, we don't model a probability distribution anymore.\n", "However, the discrete points $x=0,1,2,3$ represent delta peaks with no width in continuous space.\n", "This is why the flow can place an infinite high likelihood on these few points while still representing a distribution in continuous space.\n", "Nonetheless, the learned density does not tell us anything about the distribution among the discrete points,\n", "as in discrete space, the likelihoods of those four points would have to sum to 1, not to infinity.\n", "\n", "To prevent such degenerated solutions, a common solution is to add a small amount of noise to each discrete value, which is also referred to as dequantization.\n", "Considering $x$ as an integer (as it is the case for images), the dequantized representation $v$ can be formulated as $v=x+u$ where $u\\in[0,1)^D$.\n", "Thus, the discrete value $1$ is modeled by a distribution over the interval $[1.0, 2.0)$, the value $2$ by an volume over $[2.0, 3.0)$, etc.\n", "Our objective of modeling $p(x)$ becomes:\n", "\n", "$$ p(x) = \\int p(x+u)du = \\int \\frac{q(u|x)}{q(u|x)}p(x+u)du = \\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right]$$\n", "\n", "with $q(u|x)$ being the noise distribution.\n", "For now, we assume it to be uniform, which can also be written as $p(x)=\\mathbb{E}_{u\\sim U(0,1)^D}\\left[p(x+u) \\right]$.\n", "\n", "In the following, we will implement Dequantization as a flow transformation itself.\n", "After adding noise to the discrete values, we additionally transform the volume into a Gaussian-like shape.\n", "This is done by scaling $x+u$ between $0$ and $1$, and applying the invert of the sigmoid function $\\sigma(z)^{-1} = \\log z - \\log 1-z$.\n", "If we would not do this, we would face two problems:\n", "\n", "1.\n", "The input is scaled between 0 and 256 while the prior distribution is a Gaussian with mean $0$ and standard deviation $1$.\n", "In the first iterations after initializing the parameters of the flow, we would have extremely low likelihoods for large values like $256$.\n", "This would cause the training to diverge instantaneously.\n", "2.\n", "As the output distribution is a Gaussian, it is beneficial for the flow to have a similarly shaped input distribution.\n", "This will reduce the modeling complexity that is required by the flow.\n", "\n", "Overall, we can implement dequantization as follows:"]}, {"cell_type": "code", "execution_count": 7, "id": "14292232", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:54.841782Z", "iopub.status.busy": "2021-09-16T12:41:54.841282Z", "iopub.status.idle": "2021-09-16T12:41:54.843421Z", "shell.execute_reply": "2021-09-16T12:41:54.842957Z"}, "papermill": {"duration": 0.040991, "end_time": "2021-09-16T12:41:54.843517", "exception": false, "start_time": "2021-09-16T12:41:54.802526", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class Dequantization(nn.Module):\n", " def __init__(self, alpha=1e-5, quants=256):\n", " \"\"\"\n", " Args:\n", " alpha: small constant that is used to scale the original input.\n", " Prevents dealing with values very close to 0 and 1 when inverting the sigmoid\n", " quants: Number of possible discrete values (usually 256 for 8-bit image)\n", " \"\"\"\n", " super().__init__()\n", " self.alpha = alpha\n", " self.quants = quants\n", "\n", " def forward(self, z, ldj, reverse=False):\n", " if not reverse:\n", " z, ldj = self.dequant(z, ldj)\n", " z, ldj = self.sigmoid(z, ldj, reverse=True)\n", " else:\n", " z, ldj = self.sigmoid(z, ldj, reverse=False)\n", " z = z * self.quants\n", " ldj += np.log(self.quants) * np.prod(z.shape[1:])\n", " z = torch.floor(z).clamp(min=0, max=self.quants - 1).to(torch.int32)\n", " return z, ldj\n", "\n", " def sigmoid(self, z, ldj, reverse=False):\n", " # Applies an invertible sigmoid transformation\n", " if not reverse:\n", " ldj += (-z - 2 * F.softplus(-z)).sum(dim=[1, 2, 3])\n", " z = torch.sigmoid(z)\n", " else:\n", " z = z * (1 - self.alpha) + 0.5 * self.alpha # Scale to prevent boundaries 0 and 1\n", " ldj += np.log(1 - self.alpha) * np.prod(z.shape[1:])\n", " ldj += (-torch.log(z) - torch.log(1 - z)).sum(dim=[1, 2, 3])\n", " z = torch.log(z) - torch.log(1 - z)\n", " return z, ldj\n", "\n", " def dequant(self, z, ldj):\n", " # Transform discrete values to continuous volumes\n", " z = z.to(torch.float32)\n", " z = z + torch.rand_like(z).detach()\n", " z = z / self.quants\n", " ldj -= np.log(self.quants) * np.prod(z.shape[1:])\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "d1d08852", "metadata": {"papermill": {"duration": 0.030723, "end_time": "2021-09-16T12:41:54.905082", "exception": false, "start_time": "2021-09-16T12:41:54.874359", "status": "completed"}, "tags": []}, "source": ["A good check whether a flow is correctly implemented or not, is to verify that it is invertible.\n", "Hence, we will dequantize a randomly chosen training image, and then quantize it again.\n", "We would expect that we would get the exact same image out:"]}, {"cell_type": "code", "execution_count": 8, "id": "728daf90", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:54.972326Z", "iopub.status.busy": "2021-09-16T12:41:54.971852Z", "iopub.status.idle": "2021-09-16T12:41:54.978174Z", "shell.execute_reply": "2021-09-16T12:41:54.977694Z"}, "papermill": {"duration": 0.042049, "end_time": "2021-09-16T12:41:54.978271", "exception": false, "start_time": "2021-09-16T12:41:54.936222", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Dequantization was not invertible.\n", "Original value: 0\n", "Reconstructed value: 1\n"]}], "source": ["# Testing invertibility of dequantization layer\n", "pl.seed_everything(42)\n", "orig_img = train_set[0][0].unsqueeze(dim=0)\n", "ldj = torch.zeros(\n", " 1,\n", ")\n", "dequant_module = Dequantization()\n", "deq_img, ldj = dequant_module(orig_img, ldj, reverse=False)\n", "reconst_img, ldj = dequant_module(deq_img, ldj, reverse=True)\n", "\n", "d1, d2 = torch.where(orig_img.squeeze() != reconst_img.squeeze())\n", "if len(d1) != 0:\n", " print(\"Dequantization was not invertible.\")\n", " for i in range(d1.shape[0]):\n", " print(\"Original value:\", orig_img[0, 0, d1[i], d2[i]].item())\n", " print(\"Reconstructed value:\", reconst_img[0, 0, d1[i], d2[i]].item())\n", "else:\n", " print(\"Successfully inverted dequantization\")\n", "\n", "# Layer is not strictly invertible due to float precision constraints\n", "# assert (orig_img == reconst_img).all().item()"]}, {"cell_type": "markdown", "id": "063694b6", "metadata": {"papermill": {"duration": 0.031526, "end_time": "2021-09-16T12:41:55.042165", "exception": false, "start_time": "2021-09-16T12:41:55.010639", "status": "completed"}, "tags": []}, "source": ["In contrast to our expectation, the test fails.\n", "However, this is no reason to doubt our implementation here as only one single value is not equal to the original.\n", "This is caused due to numerical inaccuracies in the sigmoid invert.\n", "While the input space to the inverted sigmoid is scaled between 0 and 1, the output space is between $-\\infty$ and $\\infty$.\n", "And as we use 32 bits to represent the numbers (in addition to applying logs over and over again),\n", "such inaccuries can occur and should not be worrisome.\n", "Nevertheless, it is good to be aware of them, and can be improved by using a double tensor (float64).\n", "\n", "Finally, we can take our dequantization and actually visualize the\n", "distribution it transforms the discrete values into:"]}, {"cell_type": "code", "execution_count": 9, "id": "9e42101f", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:55.118016Z", "iopub.status.busy": "2021-09-16T12:41:55.117517Z", "iopub.status.idle": "2021-09-16T12:41:55.544694Z", "shell.execute_reply": "2021-09-16T12:41:55.545080Z"}, "papermill": {"duration": 0.469437, "end_time": "2021-09-16T12:41:55.545218", "exception": false, "start_time": "2021-09-16T12:41:55.075781", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDM5My43MzEyNSAyMjIuOTQ4NzUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicvZ3LrmXHcabn5yn2UBroaOU9cyjBNgGPWmoCPWh4INOUWoLKhinZRvvp+/8jYuXOCJ5NFkm0BNhgfbVq7Vx5iVtGZqTHn95++av0+MNfHvh/j+vxJ/zff+G/v+Cf3y786dNbWeV9lJQb/vTn40855/dV52igl/vT/3l7+z3QF29vFQ+PyafLeG+14K/xwjHfW6B/Pmm+rveeFT/fcFL5hV/+KrPl13tqaHxC4xMaL3/8wujbv78lNPsXF/6q1PcrX9b099UeX316+/WXj1/+Q3qk6/Hl7+Vzv/yXt//9+Nkv8OjPH//0+PIf3/7+y7ffnG9J+XovZeVr9pz7978pva9Xb2oN/xZfdM3Wxue8Kb1607zec1s9pVnX97/oem8vXoTuea+ltpT6XPN733S97KWc23vGWNVR01U+40UvW1TXe8sj5VKvWb/3Ra/7KM/2nmpNtfWavn8GvB62MjgNv/8Nr6cQ5s57azIR13vWyf36Pf/98i3vcyQuuIIlUjv/60cN1X4PPmymC4Nfeuk/brT2u9I18J/o7jHyKN/Z0d/fMIzb+yoQA/Va1/rOl31Gy2Z+v65U81Vb/a4uy7FlfM8v+MZU38eiIFr5vZcpAuz1e/7Hzx9pvMtv/+ybnz9yepf//Ld//t0///HPf/zr/3U/8vhAYhZIrvlIvbxf8/HN14//9fjXxy9/VUz25dQ6ZBGeu95rt/+Nt+t9XG3OXFpej2/+8PrJh3vyt1+4J99eP/nNH16I9yes8/2CbGqTcry+pzVSnaQprbqa0p56Gpk05zQXYXu/SktLXlByw+8pxYDltEhrXq0UpXOUXOUFrVT5qf6eF4Sz/Hus9qs1pSOlPOXfjzrzlH+PkSlUCqSz9XYtpa2uUuQNq7dRptLVZ2l4w3q/Rlujk6KNaMLMpBhlSCql8+r16qR5rZakDfjvjI+QN1Q0pirstdYmL4CsyxcfbVixvdQhL+g196wQDWcbAdGAaw2l60qtyAvm6L3xy1pC3+HT5QVoQB5L6Sir4b/xr1Kqa/HLWoa2nB2SGhTdPaUXQNsYHVoOtOA/S1K6Vu9V3lDnNaQJ0BuYM0Ne0K+6stFZal/yglEufDtphbIqI8kL0OVdPw0zAaKiyhvWrLnKGzAVVsInPdBwjN5aRhsEzCLE0C/pW2rTvGYqpNA8V5JH+zsmxSzygoYevYyOPqbAXtZYHMjGRdrxLOkYFR+htF/o6UG60gWRRoqZDdWERyEl6mgtKawVnSs0zbZKV4qltNBNoJjNrcvoYCZgygBhfeGnFI0rXVn+eYd9Ada5GC/8Atlo0M9ZaSsLwgBwrj5kvAFXg/yuD3T7VTM7DZR9AfU5SNO67LX4QJgLjbAUzlnC/I51n4a8AH3WZ1KKaYilS4q+WvoC/AQmXCYcK3eZ84B15FzlBYvflZXOlTLEB4YYn6fj0ut7wR9Wesh0wKydSkdeBV0MWlNfSRpGzT1LkTe0Yd0FygnX5Q2DA16VooWQyaQT02VIG/p7vVrlIDYuKyxopROtxaxutFbmTNKGgTkFG6iSYjSbtgHToGNOTlIMYRcztGMazNQuecMoBfNTabuuluUN66o12bMLMg0SCrMPq+bK0pGLA9BGIs3QvbUqHQ3ru5Fi7Q8ZCuhnLoUkb8BiXWUpxYO9yRvGzL3y1wb0aqpsJejqEHtFaS2lY81zstc6Lnt2Uu4k0pLnpb+G6TAwGxopOuGSBQnasRi6vKFDrIj8GJhbkEVYMKBzDjReaWOv4g2TX9xEjg/OjTEhUUEz7MzKfhiYD506h7T2tLQNGNkJjSNv6BAVMpoDowURc8kbZiuXTF9Q2NILCwQLCVbMJbpgdK4FjCIpJOfQX+vyG5QgkMMtJ5mpoGjtdckbesMAyhvQJxegvAEyUgUeKCQYWvHoFNSYG9JeqLYCkZhJIczWVZTCwL8wmp0Ssy6Zv6BQINCEpJCR1rLFOcf2cD3Ddtb34pcT1kN/YOVCs+rso9VfCnUmV3mlPBHYG5yxTNigWic/YmI6QApUeQEmSetG0cTc8QLaRC3dFIoFy5A0Z0wefsSk8JolddLa2spVKWx8+EOkfYqNDorp0LFG5A0LDZcFANowWpAfEAmYm5dM1Ak7GUsECge0DBUvgBV6FBoeEHIK8kEp9UKTF+BNWbQy7PrcaVdAeKDZU38LkwGGB7qfgoZvkr7BZKAm7KTQZ1NZg6xuRLBssshsQNp3kBKQJ9eAIpffx0zAQoOyAC0UywoxszFvKY9g3+i/xzzAvK7y7yFjtFcwhWFaQxBC8FzzUpU+MVmLDAE0FUV1VwgnbEDygMLOyPpR1O4XRB3ppMwEXJgEk5YIJFSCyyUvXbTUJhYZaZnzEkmxOHcGpicp5veQtQva4D3IowsTQ4wlQJg9E0Kw81NGF4MA5i4kK5Y3aR10EpXKD3SaVWPK8lgFMydzjkCU8UtFa4LCHMPqJy1XH7I8FsYN78eCB20QVAZhW7CL+1JHIiuFTL+gzSgLsXgSV8fCaNDMTKS14ee6UnQVFjwpXgB3nbTD0um0myghYWXqJ6CXR4PZS1rKhW9XulalRAalCO3yEQOaqXCpgGKZJO1FyD2sVegzyk38tXbDpEznx4O2AnuwKMWnc/aCwnsoMj0WOmpCleINWB7NdCpouyYNRVDMqCTWAygsrAIJOGhs26qB6IIY6BAJj8G1UJPBAYsQwzqoU9GeKhiTgZ7LJKbqat0wNAuMpwfk6dX41YIhH6DZIQSBy7TPI6ZLjcUHDAknC5IUchLz40HpW1aVsYMsg0gUux8Y9t+0pzEp1sRAEGOp9UubDWMhwbDKD4jaCw0twzD6sdFb5cwuuXbDkPFURhTM7PMiuNES5MhD2l4dBtkyDP+NihIYXXplbSA0MAwVPAOMPrvmNNwxoFBuEMMpQdNqS9As6BOsTGBo0rKMwuCFiU0KZczlrnhCL2JxQj5D3BRtNVwHqBOoN1AswjqW4QHFiCU3RQdcSTF+vTZ88gMSGubddQ3DrVfY0cR0JPL99EIP4430dzGfdMQSVgjMLug4CnRoVlEaxBOzF5KREh2CqymmfX1RX1GkY/g3hhiEWCJut9CEeQ5ZifXZiRe9l2a4rsmPgADnetGRTFQNAyNL3McS0yklGqQds+AxRfKqm0iMuYD1QYwpXsV5g7UEi6hysIHxLWpLEDdakXgJjSqYFfpuvBGLEX8AxhyCJ2YYNiCa9YAshyRKuRuekCWQx5NjShkpGG/sFGxziOep4wsKzwFr9gEpDzXZqv4iZDKsAaxOCn9o0lENc32nx+SESvc3Qv5WeLWFGB52sh/EJOGADGIMRcnNMOQ7lhBkPZ0psRPwFTCNMsUMVcDqJhiAKYIhNyHYaaHrL2aYLpVqeHFAq3ZepgyFjC6MbsBdmKImkwRQYLsMYlhMQ+VFxvsWHBNSGIMcTqXQDnjjAyK/LMxC6Y4MtwiCCF0GzGCOfiFwa/jNB8R7hipuN1007yvxoHzSNvOF6Dy8A27BZEzB8Ew0kEAhASFnhOKF0DP4w6LKnbZkgOG/wrgl5neryMn0DahrHpDmF5TDujF1HoQ/RT/+oc6PTNselvkght3R9cs7fTw05QEpD9EHgWYYDiWcYWI8ei19GvOj0/x/LPoBZaYbY13CFiWeaFPVr5kMXKGPHxTqMPcgrYzXTNNHONyGeeNZGHl4UKrDNB7WcBrCMBmqcNgWq1TjEOsMCFCC54vd6PkRrX9N7d2O7pYEas0O1D7S0d0jgVr3Obr7OlAbGEf3IAZqIx6oTQ9H91wK1Caeo/ckDdAmtKN79geqKyVAW1WO7iUYqC5XB/fSDtTkQKAqMk74FC+Bmixy9JZbAZqMc3QLxEBVeAZogtbRLZUDVQnu4Jb2gZpqCNT0iKNb6QRqGsrRrc4CNd3n6NaTgZpSDdQ0sKNbXQdqut3RbQgEalbDSZ8mRqBmjwRqxouj29IJ1MwiR7cJFajZW4GabeboNuQCNavP0W0iBmr2pKPb+AzULNVAzax1dNvAgZrB7Oi2rgM1SzxQM9sd3TZ+oOYQOLq9h0DN0Tjo0ynx0PwXD83VOeH2ijw0B+qE29fy0NyyE24PzkNz9jw0v/CE24X00LzNE27H1EPzYT00b/eE2zH20HzoE5qz7ZF55SfcDryH6up7ZkGBE+74gYcaaTjYMyThoUUvPNQ4x8l2RMRDi52c8I6yeGbhmBPuyI2HGuPxzGJBJ9xhIw81wHSyHYry0KJWHlp864Q7FOahRc1OuANsHlos7oQ7bOehRfg8tGDgCXfc0EOLMB7wGYz00OKWJ9whTg8tGuqhBU5PuGOsHlo49oQ7cuuhBXk9tHjwCXfo2EOLMp9wB6Q9tNj1CXeY20OLiHtowfMT7ji7hxaSP+GO3ntogX4PbU/ghHv7wEPbaTjgc1PCQ9u/OOHe6vDQdkU8tA2UE+69Fg9tW+aEewfHQ9vsOeHeF/LQtpA8tN2mE+6NKQ9tD+uEe7vLQ9sZ81D30E62d9s8tI25E95beJ7ZZt8J976gh7qD6JntNR7wuSnpITcbTrI3OT20/VAPdef0ZHuP1UPbjj3hvXHrmW3xnnDvBnuo+8ae2Q7zCfdmtIe2b33CvcXtoe2Gn3Dvm3toW+we2m78CffGvYe2x3/CnQ7goWUOeGg5Bgd8piN4aIkLJ9w5Dh5aOsQJd+aEh5Zk4aHlY5xwp254aFkeJ9zpIB5a5oiHlmRywp2O4qFlrpzw25kvkpX467ffPH5oEk9iPs6iWYdlzswdSC+M4Op88g/hrx/ur3/7xXf99RtzdNCdmOGwrA8x9emk9BL6glKV2BN4S8zgeND/YIBN9ojJV4FPVYRjdSwNYRVMwcZQ3kN8Gwhu2SUhZ8JIFgxVMmQHKRXMbTS08jUQtE0MFuXczuldOKy6S4NPBf1V6ObST0O7pob7gJmfkYrwrv6mcsgMjj99wEQjVn+WkUXGv4ghQFa/8bg6M4HodMLOuTSuXBp3WZkrIi4qvf1kvFUaePRn0d1FQ6bAC95QUz4YBNDXQzLNDJH8oAcNqdeum8N1r0JHX2i8UGi4TOPkIb75wmuT8V4WFyQ5teXSX4UkbbNiOOn3w5/rGvcr3JsbdS3h6I5r3BzfDRvvIZGGNbI9DxWeGFYSvkB146jwU2Di8j2wbOG0yM5cqkytxVrOwpclYhG3wZ2+BwMkreDfD+NrJYiUByMvV19JNxEqTBQMXh/CMbK133xmjA/fwySFVXQLqkILwsuRbmPqAZzVZhwvoQvCMBKTl7Q5XGgd1udDQ05JczPI29XG5GsWzeh83RyuFhUG+YTTq7tWWP0w6WFsS+Yv87A0igoOuU0Hi5x5Jk1/V6LSDJoz2NZgauqMBe/XhYX5YMAO8yzpzIQYw/ReEBXknTvgN8fPYm48GCFMqyz7Lm79Dka9yeHHJt0PgODkFg8jj4wu28eyozq1N/nkRoiOCVPw2oXZK7FORnaSccxpTnsGRrEGr2kYo8yYNcOorTElTzg6EIZn4vODb+x1Gh8rcaQyV9vqKlqYW5Yu7mAyyku7Tdc4U8a4L5SFTzjHupcG7XZBafFL+UPdGgkMIzZzFWbuiGgWHfGEDQIRqDIMhpl0DLOuYHaiHwv9npl1PJhPBYOSwfTCfbOu6QCJaVJMF+BrxO4aut0HzlztJhg+gGYvENPKwFhS9kASDt1UYv4SbCl4JOSYWUP3yJiVBPeWww3hw1QQ3dZgBhKsU4rdwjgznMNkvDd4gFl4x7K7ZFpKDhGMCYwgpEnGP0g3r8xZWsIHrME2jK9ELfigPCkUEIIHtx04RYmxpmx2gw8M82LvTCaDta7NwcQZiY4BxQPETx/dOFrWaxLOdaF7CkzFuWj1PERsQE3pVivTbjKtO+Ew6ZJuBjPFBm2DtKUcYMt0EJlOg1VeqvAlys54x3LglIIcqJAOktmVmCYDS3byPdQg04QIc2LQz1cTzhSUzTmROEuo7vtauoEE0xvuweQeChc8xeE0PualuckYucXMEeFcqou58+Sw4Va9OV7DRsvCRi+r1GHOSuE3EmPeJ509wBS8VLZiJcH0Ssbxm0yClCV8Xd0+l21onGzMeIWTprtOGx82ygu4XxyoNSNQbbOD+wMDte5wdHdeoNbVju6BCdSGMVAbdEf3FAnUJtRJn9MvUJusgdrUdnQvhEBt2Ti6F1mgtiQd3es3UFvtgZpscHRLkkBN7ji6pVSgJtMc3RIwUBWXAZpsdXRL4kBVbDu4ZXygphACVe3h4FY1gZpiOunWYgGaynN0K8hAVZsGaKrX0a2oA80j/POt/gM1YyFQMy0c3YZIoGa2OLqNnEDNJHJ0G1CBmrkVqBlnjm5LLlCz+xzdVmKgZlM6ui3QQM1eDdSM25M+TeFAzXB2dJvZgZpRHqiZ8I5ugz9Qcw8cNVciMHM7HN1OSqDq0QRo7o+j21kKVD0rB7cbFqg5bYGqh+fgdgcDNefR0dvTDNDcUke3ExuoubyBRrf5R3v4DKdmc8vfKIJGhRc3IfLOv7FjOs8/v37yt1+8feaTEgKAzscCcnvIn05KsZZhTavR1uFpF6ZnCh+9rxvD9IfdQnlZoB7UMAaGOEp0nHieAz2m7iwzTxfPWT1EQKfVNI0NHCY+flz46vSuhfMA0OReA2U/zNmukYHOMzyDqS7UHxd8DOOLeddikVPZwHhSkxYcDjU3F6mwsMQ0eZZ8YfnMJrwPnpckH/gY6A6MmShDCGfJbE6SNcrdD+HMKLHnEzNPuSNKRZsHLMibt3lx/5Qc41A3pzVJ40xUOxqtHPYqfFh6AkyaZ3CtGeen0OSE2YChKpe2pzCVrTJth9HWVew1DMZgsPmz+HKJZQqXDNBGi7NxYzZZJAS8ofV0x5mNDjdUk58Gj9hJXjg5nO5mvGEhJaZC0b6qVaLYygfEAZ08s9E0bWt0xlFo8DNPvF6apE3MIy3GMdGKhpWY3JlHn0MWDnrDPnbg33a6JaALnq4mNQJPnsbiW6CEMILWSCbdVZ46UEu2Z82/Yx4nh4Tv4XdkPcKUmMiJyUbnTxK0W+3dOCxthoM6o4u1pmmY4bSmHDOnSj44j93SX6fb3UU6N6PixJNBnZmvzJxNvIiuOFOqC49vGO+lc+uXHKr9kmNHckqxNVr6kkFdWrlxpRqpwisdwmkc+l8CZUyYhtbXoM+EU8qzG/hZTnT8lwbKJk9xZ+aayERvQw99YEJC/yUucs7zYcG/yXT5i2n8Ms2pR6pxfCkU4YPT9rrm1LXMJE28sOeHTvNe583pf88uHCZmVneQaZq1M8gm0xyfUG/eu5z/k3nOvGzlmDVcOlM4U1tUtoC3izGdB+cznZR1c/wUpxz5aBBp+l1YYBVqhe+hzpvmOE3m/15MgiWnl67zjPmaevKTfLY2UzeOcWBGBSc6vsMWyZOfmuUltXc7ulsSqLXb0f2VgVqfBGo96Oju70BtdBzdYxmojbyje54EarMqUJ2BDu7pGqhNbkf3UgjUFk6gusoc3EsyUFvAjtpiD8zEwkmfQiRQlTgBmnhydAuzQE30OboFZaAqVQM0CezolteBqnB3cGuCQE1vOLq1TKCmkwI1Debo1neBmnJ0dKvSQE3xOrrVdKCm1AM1E8DRbTAEauaFCz9sYyRQM10CNUPH0W0WBWpGlKPb5ArUDDRHtzkXqNp+AZqd6Oi3bc2fYDdP2dbqsxcavA06Gb53b3K9hxxR5+iK3fx88vH6SbGbP+tJsZsXfADMwzNn4dNJuSAajBA1xRbtqc4cPHKozqyqf/GYc2PGHhdbxjNqojEXn8eMu3CGqPQwAbPxV5EYLviC8NZg4co8Ai2mG1d9rinfvBXuNwtvRU/NJsnIv8RyA57VksuZkd8lc0DkDAPl1fiY6JMlHBbO0BMjTMq/RhddTjd4FjUkwXvunfY0BN6VZbucvL3DyOB2nwjC6xq657B4cIOpTMJ5+kI3LsAn3knbbXJsxtQY6IIXC8GEzmS+EvyEfGPMeongTp56XEn3KBb3ii6J4E6aKNV+FR3blgRwgWep95igYwfTSfU+iN51d0WS9plBKzgvO0VNPpMkDZO3lJZ1zuKOk1hvU8JYVQ2RxQOkVXZpJs3d65L38AobeJ+LAV+me+U55bgdOdwaMd+o1NDhYuiQw/q96ERNjaTI1ki+GBxPYr6Bo/GjZ+OjXZe+Bh2pJ9vzlRnTlh0Z5qMxzjOMN9hsZQqHzNYjArwGBuMm9hs4D63b8zTRGaQVXuHir2V8VvUNeZjnKlOOTqHh77lXxsbJocGzuJLkaDOPnJFzs2Dp+xtPiGdtDo/ESnibGGpA9mrkVFDV7VhySGwx33i7y7CFBTsBuocrSHhmHmQyDuEj5tukerpG1d4f3OEvMurcDpu5L+M9sf3KYbNNfX4yd6DKqDOFKJVquG6KfzjL/fSEJl7aemaAZH2cF0NkJv2SD3i2PRmHAJatGvIydac9p4tuA4/BCMd7ZPeZHFONGxnCmY4yjS+Is6StH1ClslF28ENAv6b27kCtJSd9tjtQ+0pHd58Eqh0YoHW2o3toArWBdHQPe6A2SRzdUypQm4CB6mx1cE/tQG0hOLqXTaC2yAK1JenoXsCB2nJ39JYNAZogcXSLnUBNSAVqIu20Ap4CMFATl45u4RqoiWJHt9wO1KR8oKYSHN0KJFDVNg5u1RSoKbJATe05upVkoKZSHd0KOFBT145u3R6oGQKBmtng6DYyAjWTxNFtwARq5k6g0WT6CeZfs7ymvngT0XUbbg850Q0rS87Xifn3fPLx+kkx/z7rSZp/0DLMZXNHsz6dVOXq1MAd+eRNEFHc8hIF/ECN4jkzz25cM4rzwqS8UW7xr6EH4gp/u/WtLcY0jifGWFu7NNFdiXr7wqcGbcS7qwraHHRXYjLWnKYyn6qOqTwYMVOZT9WYmB8il8TdqlRu/MmpM/Vx5RxUL/gs8oleVafBpMo1gmZnho+mmHlLIE3e8cArZMxw0HuVyGGUXcw/cIYGuBhfNRgmTP1hbvcMhkxivHum9DR89HMzT22PVHowlDJvO0LnBrsqw2KlzivbDtPByoyg1swEG2e28U6wDjcxBTMvZznm2VowC8EZNhfj+DQjmSkETSGDfpqd4I357d2bqeRQ7qUub9ZmJhFxI656M5gcFqUO19Nqlsv+qPiDkU3eC3NJt03elHdmPjNJ7rbhrTc706R575Kz+cnnYga0dxHgcjABmylYp0dBjLnDe5acBwJNzczuJoN+eCzkjZuRaXs4OknA1+SJjO0R6ULPjLPz5In3oMhnlvM1t8MlHlcuzP0rstIP/4wYKoAR6dudk6hyZprSTMwjut0/o413z0zvLJLzCoorOJcwNZiYPznkpzNKPrgfu7zzevBTub+k9m5Hd0sCtXYHat/o6O6RQK37Tvrs7EBtaBzdAxmoDXugNkkc3VMqUJ1/Du7JGqhN7UBtITi6l02gtsgc3SsyUFu/ju7VHqjJhkBNkji65U6gJqUc3TItUJOAgZq8dPQWrgGaJHYeyZbbgZqUd3TrhEBNgwRq+sbRrZ0CVVXm4NZ7gZqWdHTr1EBNAwdq+trRrdwDNVPA0W04BGpmRqBmkzi6LZhAzd5x9DaOAoz21U+xFS3tHbOTO126BZ5h8FmAj/armoofPvhwD8o9mM8H314+KIZiKTwvNc/cqk8nvXeF1EYq3K6GT939LlLmBcUJynDdu05zZeO1MCHE71KRz5Z5iYbtag1TLI3HxNYKu2Dk8Aqv1PyuWS4dDsJ1SeTv2GUjx+vFeLJdObViwNE2MZ5sF082bTO+HdoniR49dv3IByMLbe8SZlU50HoXDLi5dxXTMN6hDblre2xCZiacVh532VuWvRrnkWzZxNMtTjWewLkf2arfEs1VbAJJlj63UMnRa4Wa1LZc5aYktIRZBIWxv3OHlrzBUGt+Q5d4cU6XvQGso1UzT0iK7fTcLCadiRckuZ1l6DsevOSNLG4nmhyDI5aTbVxbWyqPdDIzwm10k2PKNhqo58Y4OT6jhX30XHmv2RTL6dx3J4cF3rhva7v0TZvDMxkMW92b+varncdbxXA6kwBy5bUqVQwnyxmw1gwenOXxhTvHQH0V8Fkz3TGXk5B5VyxP6aw7h6FbL0we36WNfOc8qOHH61p5FWTfORLW/MVzwbyc5kypIF6Fl+TdGRhZo6hyCyoE29wZG2ot83JTdM4IGR6ZV5ZyETefEUKO1SDr3DJI1HrnXaLocMmYOTJOyCscdcmYOTJUyOe4JNXAMlrUnm2YPOuSda4ZMEm7gXd1XrAn586YafpdlULlkowZybCxbuZtmxTvzWfkkC98omTMHBk8mfdlQiOVkPFDjo9PkjEjGULl0vd03hGeJGPmyCgi75gDks78zD/KjbczJd4he6cr2edufuqWF3S/OlBriKO72YHaRzq6uyRQ68BArbsd3YMTqA2lo3vgA7VpEqhNKkf3FAzUJqyje3oHaovhpM+lE6its0BtVTq613CgtuId3fIhUJMmgZrscXRLqkBVrDm4ZWCgJjEdvcVrgCaLAzXJ7eiW84GaVnD0ViEBmrZxdOumQFWRBWhaz9GtIwM1jer8tq1/AzVtHajpdke3IRComQ2ObiMjUDNJHN0GTKBm7gRqxpGj25QK1AwvR7eZFqgZdYGaCejoNhgDjUbnT9lov03d+q0734WWBDNdN9r3k9+6R/75pG60f86TYkBDWmNA8ilNPp3UnWshF3PueQxGLUFqawm62aEZjZyADvoudR+y0V2/xtgNjz/7Mznk+CWGd88jPMQLGofbeHbkR95CAyQVai53Qoicq6fqgaJx6fFaCDFeaCMG3H0AKRnvvO0w+wNLmRfzQiDTgLMDTmpDgDcmE619IKrcz68sN6XYAaqptjkv14X0owVnB67U0AQfnRff7ANaVZtfeRtQpy63A10aousSRRWT7zwAlnlPLnpQzsEdB8bIYbvyotf7gJnGm7t0rZzILc/zaJnG7lijhuNr5J0ZVWUfd5Mx4Z23dGH7Ph2n4WPwpvbkeZiOeMHb4nw6D9/lTiuPZwz2Yb1688Hjp80f7oMdJOaHnILTw4DqWYB3MWP22UHttMFD2XKprDtrSI6ZsObzbKIOOgtxcO1kf5Qx83LaS7JW74OPy/CgFiz7oKTaY3C2eM0yc2vvc5WGed9H8qcwidfEVC73qc02tfGFmyQcB3fIk5w3EE5/JjTD42RiijTyOENK3nnAJe8jpzoxec3s0OOvxwnVLA7w5H2Z+0Sr4UUzYOzTrzqdeKNskdkqh2XrpUuZN8q2VrgK7XCtunqDZ8MZKvWHccnbglRO+/CuDQgLWWQGze/DvjpBeH1sUffNDgfbx/Je8otJxfdhYrXYx+LVc5UW+3n4mLwzmFOeh5VlwvKyWNhCdKTus83LONZjoyN1noUmh0bhCWh3djozyo5pxzE/z1qTj1m6HDnXs9nanQz6X5l1JOwst15hT95z4pI/j37nyYID3NQ4z4mTVt5dXfy5cnLuk9C3Pc+hZ7kddshheT22vowO/CI3muyUu65Z7iKhP+S4+XEqnry1OuU0+3GKnnyNMhksOU/d59l5/eKU0+zHGX3yiUGiE3We6c9z8GZH3nR83wHQbt7rWhS8dmWAus6TFw9LQtF5wwAxhOQlFxgcNxKQr4tpnv4CAzQQ06XTEz7vOyDmsQ9uLtr9CLp1sXg9dk1ygcFxnQI5Zkfaty+ov74gfC4Gl/1lDeTckJt53+2gSgN8VlZy2HdB6I9mVqHKVLR6dcTU3TBWyYFso6K1qyY03vXkp1nzktq7Hd0NCdSaHah9pKN3hwRovXfSZ18HqgPj4B7FQG3MA9UJ4uCeTYHa3HN0z9RAbV47uldBoLZmArUV5uhej4Hq2nVwL/RATSwEetXY4VvaBGqyydEtyQI1uefolpKBmkwN1CTwSZ/yOlCT7o5uXRCoaY5ATc84urVSoKbDHN0aL1BVjw5uXRqoKd5AVUs7uFV6oGYAOHpbCwGaaeHoNkQCVaslQLNwHN32UKBqPDm4La1AzS4L1Ky4kz5tvkDNQnR025OBmvXp6LZVA23fCm08rWBHt8kcqBnYjm5rPFCz3QM1S9/R7RcEal6Eo9vnCNQ8FEe3PxOoeT+Bmq/k6PasAjU/zNHbaQvQPDwX/9ruYKDqOwZojqaj2y0NtMR90Ke7G2h0mX+C+3/cIPWHN/fHL+LfQj8WHo3N54z9dNL7/nANa4NjTJiQ4+4bJ1+lcc/nvp+8Kq90tQq9KLvOvNwcs5klHOz686ReEe+BX7y+8b4tXW1bXgSfeE+lu1s98yJ43SI6b2InrtymLe7eduLJIGe/L3k3A2jw0DMtRbsSXs1y/GHQ2yzuAnlYRzxOzVva7Lp5NaaB8Yc277vps8HVCy/MOS+yz/yEmXlPnN16P28MewIS6bwiv7B3slwNcl6oT9zL4lV15+37MMboxsOBPa/qJ+UJgFrui/3F3iRmwbGxqwBICL5gvAsDclYxQD6bcJQKh/8uLzD0WVY3KbP6WgTEcJd5m74WLtgPL0jRVe4yB3KBP8aP5/yhve6aCOLwEjORq7oCCgWWLlYJTy2d5RaIO+2Y4WozlIuDwTOxdyEHg+3incmu6gPx4l7YuEtENO07BjR4Cs4VlCAeUBXQQFZ+wl5CtyDz0sazWAVx59m45EpbYLq+Swm1ehfCWNlwZTh/WNkMrVFCzAofMOGY/TYtCFXo6NCHrXdFDvEeidkP6DWr36HzJrHyHrvTVfsghp3BKzGtNogEfIjpFkIsnZVECn032HwQDlZ3ROcI8OC5wOqqlMAe54Uc3PO8a5oY7bIV6gqgoAW86APd78qlFPFFoVfnLq5idE6xI45CLJi7vGyE9cnOqi3E9PTRDUeJlwJPOeGzMLxWD6YYhZbqV7uLx+h0Sow4QvrNu9RM0e8evEqFUSwrTGPfPbidzhtazzI2RXz/2bW6DYPyZRjGNBp5l8jRKZJYnLIzhGUFdVI1jJk/WN1Gy+/cD69Uefms1eoRb7lIkePCi2qtsI9EZYhHywxCWBkg8aFL5i59otzVokFFhCdxY/ZNcyWGCiMu12KM2woS6UcCV/grGOpn8SLCyUMs3RU6Qgfw9h5ugFlVJIm/EWP50Jo7SyhBxcKGb7BOXL0l4pbrhTWrxZnKujH0+gUb667kJBTfC2cEH3ZWfSIeTDtarkQU5AP0H+/gdPWkiHuSA/pn8amSBwO+CYLXKlXpHAGukBLoHStrNZph2cfvrgZWYSyP11mvu2CWffvkaTwp5mbFtfQnF4/uSTW3oxIXcWuM0FrVrnzTxYyrcpf4kleUi8cHpZzbUQyMeMroWeEwyVvAL/FcohR0O6qMFYmGYqbeFcnmpVhyS1lb76heRlqnlGO0Umc6nYDXVduzKpqKbAZs5RGroHYto4Onnosrt0bLCD3Zr2dptmkYTerPKm7D6IJrztpoWvJNpWdpPFbKwlh3dbhueBaFLCRnjRNJz+xWqzp3Uyz9wfJouzwd2sjFMIYrZEfakhgHVvXOPm/weO2E8L8r5MncKJNncScH5SinRzwG9HU9S+9hgjBMxdCNlekr0zAcLcbXz5p+pTKRdtCiswKAKsiAYecsDqxWC0w3hvCVUm9aWtBwopkixd60DqEuZ+DB66bmLlooX1Ph9PGUe3IVDokbEwvrXQ2x3E8zvWXNXTpRegr2cZm819PVWSSGAcQNIivKqNqrQk0mijtXwZG40yhJd7lHlbRVK6eltmtDdsN1SFHJu5BkMkypxwqgWnVS9UntPNvOwP9dorIbHhzWedez1FEYTEPGt7ral8QNK7s3VyiTGKKTSc5nVc3CssoXJokvwUnMcEBpu16n0MU7AWi9nrU9iTvm+NyFQMUWL7xoeEKENVc1lLhdUkz6LjFaDK8stcbOeqSFlx3XxIIaVrzU4ODd0cvVOS28QhlqgwVqj6KoxPhp/sEqqNoPFp5ZYQmQu9xqN1zh0q68a7NOwxBDlBxWyFU/nddD85jCuqu+6qJmZs6SPH8tEavbIXCNeChHShAf9WSJ4RHNtFzxWeLFyGNxlWoLb75mia/uytoW+WT8b7kauPhbnjpi8vlZMJe4M7On39V1df5xWGHPtrVL8RqluzOLK9tLPCcNGlfjF/4PdAuTfc56wKSD6W/FFQ8uvH6cMdfuCg0TN9bnumsSN4NrMD3tLmCsJh8XPmZt7a7YMTFEYhq7MLJ8ICUQ/kprOO8qysRchOlZcVl/scjplHrXZlYLh8lwTJ0udyFnFReaO0dn5i75rE9TgieWG7by0MsoPElWObprSStuPMPHkkhn3WniNmmA3TWqtXWdhwP5yFnQmhizuZbhil8TYzHW7gplFyYl8jRX3VW1p2EeHdL617sCd6F2vyCqkivXTYy+ayyerLW97WMmT1U21sC2QuDaT4tHMGV0j6rhxFB+Pde7wrh2H82dxaRbV46cGKqbt1pY7XL1BdXCYurYXei8G648yTZdVXTiySWR7hLqKuCHHX6pVm9d0ySJRxK9c9RmL7Q5sbpTcnXciXldRalnzXfSxRDyvOvDa48wX3cO1jm7i8kPw7BDmAd7V56XsWGJCu6KzLtMfTXKEtlQlmdJ+0L7HlNjSk17+EV6TRdxnQVLhZibHjqF5cxIpreitxhqGkGh/wEruMlLeGOVbPURYx4xoZrXv8Oiz/qTk3tMmCfEjP7etGloSu5CXM06deMjnvWa2osd3a0I1Jrs6P6+QK0zArWec3R3c6A2JI7u8QvUBtvRPTMCtVkUqE05R/f0DNTmsqN74gdqqyRQW1KO7vUXqC3Wkz5XdqAmBhzdMiNQEzCBmjRydIuuQE3OOXrLxABNfgZqwtbRWzAHaELc0S3xA1Xt4ODWJIGa2gn0igdvD3UWqOk+R289GaDpVEe3Ag60fWsYt1o/6dMGCLTEM0eHbRGoGSKBmtHi6LZwAjVzyNFtOwVqhpaj2yoL1Ey4QM3ec3Qbh4GaJenoNjsDNRs1UDNoHd3Wb6BmKju67epAzQR3dNvrgZpxH6h5Aid9ug2Bmo/h6PZHAjXnxdHt6QRqblGg5kM5uv2tQM05c3R7coGa2xeo+YiObocyUPM+Hd2uaqDm1zq6neBAzWMO1NxrR7cvHqg57o5uLz9QCwkEavGDkz6DDYFaZMLRHcUI1EIeju74SKAWTAlUAy8O7hBNoBbOcdQiP4FZkMjRHVEKVKNPAVqkytEd1gpUQ2AO7nBZoBZbC/SKFYuOmF2gFuBz9A4GBmiBw5M+g4yBakAyQAteOrojnYFaWNTRHUMN1AKugVp01tEdyg3U4r6O7iBxoBZRdnRHnwO1UHWgFtd2dAfBA7WIuaM7vB6oReIDtbC9ozvGH6htCDi6dw8Cta2Gkz73JQK1PYxAbcPD0b07EqhtpTi6910CtU0aR/eOTqC2/ROobRU5ureVArU9KEf3flWgtrcVqG2EObp3zQK1HTZH93ZcoLZ35+je6AvUdgUDtS1ER/d+Y6C2OXnS505moLbtGajtkTq6N1QDtd1XR/dWbaC2r+vo3gQO1HaMA9XNZQf3RnSgtmvt6L3DHaDthju6t84D1W32AG1L3tG9fx9oCWdvjqyAQC2FINArHiY7UhMCtTyGgz5zHgK0/AhHdzJFoJp3EaDlaDi6EzoCtewPR3eqSKCWVxKoJaE4ujNWArX0Fkd3Mkygljrj6E60CdTScgKNqT0/PKsoPf7xATnw+C9eqZntUs32rXNFmAeYXfQQfvvF45d/9/V//vGrr3/7xa8fX/3lo/J6nw5YmSc59RDy2//82zaNZyww19dyZ5UOemc/Sejux7buu2oCfrtJP6jE39+mSd8uM/TpVZ2inzCCn1NV4cO2fX45hL956z64HvzTyyvGf3zrPutm3W+37gddifu3bt1Ht1h+enkT5o9v3WddS/dB637IfXJ/89Z9cNfTp5f3Rf2E1n3GpSofNO6H3LDyt27cB3cNfHp5X8FPWBOfc0T3w9Z9/tnav3nrPjju8+nlkaEf37ozofktNuGHpDP/f2nCR9tcn17tlP3oFjxi539oUX06ykIf25vPgtrnrvGuQm778JJg8izVfiYxPCvanzkg3LOD9mCul+XK8BOZOsobVtedVSTyeg5eOdGgNO8MK8DFG0gaLwS0hDPZ7WSi9WgMg54JeMwzT5iJY7qMRJ6FG6swMG5JmlqVJFF6sjbqnbOqRUzgqDPACtPb8nj1vvLEY9KpZp/XzNOrGX4fzGxL85Y9eJqLuc01QlL8q0LFr2ruvigG+6o66KtaTS9rEX1cnedlLZtX5WBelFl5eQH+qyvnX9zx/uqy9Ve3nr+6ffzVNeCvLuR+dbPriytWX1x1+urK0Vd3f764g/PjuzBf3T356mqyV1eBvbhj6+U9VS9udHp5xdGLGx1enPt/dR771WHhF8dePzwh8vKcxMfnBV7l0n+cZ/5hDvar/OSPU3dfJLZ+nPn5Kvvx4wzAV3lwHyd9vcpyepG98yL95PuSQT7QfBkq7Xr8iTYH1NoPChzwtvKeNaqBt32v3g3Pf/j2j3NU3NPP13zX2y9+F6bn4w/Q2n/C//2X/fHt39/gfD9+cYGx6Eou8iO8wl7/+Vef3n795eOX/0A98Pjy93jT9fjyX97+9+Nnf/f1v//H7/71r3/879/99Y//9q+Pf/njX/76zR//+T/kD7//t28ek+irb36Ol8Hnuv/3+NnXf/368Z+/+/N/fP2Xnz/+6e3Lf3z7+y/ffvP2y19VHhdLaOE9AF/wT28yLVljTg4MsF6nfmq6P3XTP0smCqUyj1XcVPJb9pMLwlaiVU847MGv3g7Ky9Hv/nMYeogVzegF7t/iDe368NGsJ/zq+IQn1Z0tqH654OX5Bm532aPP3zpgOd560/0Nfz7h82ufP/Tslm/36lcajXsWltxBrG/dsfMMYmlhyY+ffIvhrs9+55vsZ9onU8MWjnkL6M8n4iF8QeVbyD4r22fdq0D+84sPFgLlj/uhYxFcfhFcmMOP5xwuOoedRe6iWd/o9P6uR9BP6e273uA6RzKcZMK5Dnpi10nUDRuXD/FP6Kznj77usPRBh312vVETDZ8b/foBdUyPzmDmDCtB+w7d1PUnTwDftHxEf0Jv7l983Zn5w878zCJUt5z9zHDYDyhudXQFkzDSrc3aB9h1J2/S27h8iH9Chz5/9HWPlg979DPrOliPfm6Q7AfUizg6g5v7Nc7OG7re5F3qNc7NJ/wJPXn/3Ot+rB/342dcj/zsxs8LmH3uW10ncn+YFadDNz6x60i5lPv61jI/8U/ozOePvu7O9vFC/7xL8O6F/plRtB9wud6zMxYv5JphmW/oehO+86blI/rj+3L/4Oue7B/25HkLgXXXd95T8Gw474HJPX75k56fzmDFpuUj+uM//fmLr799nN/+m7f/ByOuM8sKZW5kc3RyZWFtCmVuZG9iagoxMiAwIG9iagoxMzA4NAplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagoxNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2NCA+PgpzdHJlYW0KeJw9kMERQyEIRO9WsSWAgEA9yWRy+L//a0CTXGQdYPepO4GQUYczw2fiyYPTsTRwbxWMawivI/QITQKTwMTBmngMCwGnYZFjLt9VllWnla6ajZ7XvWNB1WmXNQ1t2oHyrY8/wjXeo/Aa7B5CB7EodG5lWguZWDxrnDvMo8znfk7bdz0YrabUrDdy2dc9OsvUUF5a+4TOaLT9J9cvuzFeH4UUOQgKZW5kc3RyZWFtCmVuZG9iagoxOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3MCA+PgpzdHJlYW0KeJw9kEsSwyAMQ/ecQkcA/4DztNPpgtx/W8uZdIMUY8svRFd07JWHx8aUjfdoY0+ELVzldBpOUxmPi7tmXaDLYTLTb7yaucBUYZHV7KL6GLyh86xmh69VMzGEN5kSGmAqd3IP9fWnOO3bkpBsV2HQnRqkszDMkfw9EFNz0HOIkfwjX3JrYdCZ5hcXLasZrWVM0exhqmwtDOqNQXfK9dR6rvMwEe/zA99BPmQKZW5kc3RyZWFtCmVuZG9iagoxOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMwNyA+PgpzdHJlYW0KeJw9kktuAzEMQ/c+hS4QwPrZnvOkKLqY3n/bJyXpihzZFkVqlrpMWVMekDSThH/p8HCxnfI7bM9mZuBaopeJ5ZTn0BVi7qJ82cxGXVknxeqEZjq36FE5Fwc2Taqfqyyl3S54Dtcmnlv2ET+80KAe1DUuCTd0V6NlKTRjqvt/0nv8jDLgakxdbFKrex88XkRV6OgHR4kiY5cX5+NBCelKwmhaiJV3RQNB7vK0ynsJ7tveasiyB6mYzjspZrDrdFIubheHIR7I8qjw5aPYa0LP+LArJfRI2IYzcifuaMbm1MjikP7ejQRLj65oIfPgr27WLmC8UzpFYmROcqxpi1VO91AU07nDvQwQ9WxFQylzkdXqX8POC2uWbBZ4SvoFHqPdJksOVtnbqE7vrTzZ0PcfWtd0HwplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ0ID4+CnN0cmVhbQp4nEWRTXIFIQiE956iL/Cq5Fc9z6RSWUzuvw3NvCQrWoXmA9MCE0fwEkPsiZUTHzJ8L+gyfLcyO/A62ZlwT7huXMNlwzNhW+A7Kss7XkN3tlI/naGq7xo53i5SNXRlZJ96oZoLzJCIrhFZdCuXdUDTlO5S4RpsW4IU9UqsJ52gNOgRyvB3lGt8dRNPr7HkVM0hWs2tExqKsGx4QdTJJBG1DYsnlnMhUfmqG6s6LmCTJeL0gNyglWZ8elJJETCDfKzJaMwCNtCTu2cXxppLHkWOVzSYsDtJNfCA9+K2vvc2cY/zF/iFd9//Kw591wI+fwBL/l0GCmVuZHN0cmVhbQplbmRvYmoKMjEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzIgPj4Kc3RyZWFtCnicNVFJbsQwDLv7FfzAANbuvCfFoIf2/9dSyhQIQCW2uCViYyMCLzH4OYjc+JI1oyZ+Z3JX/CxPhUfCreBJFIGX4V52gssbxmU/DjMfvJdWzqTGkwzIRTY9PBEy2CUQOjC7BnXYZtqJviHhsyNSzUaW09cS9NIqBMpTtt/pghJtq/pz+6wLbfvaE052e+pJ5ROI55aswGXjFZPFWAY9UblLMX2Q6myhJ6G8KJ+DbD5qiESXKGfgicHBKNAO7LntZ+JVIWhd3adtY6hGSsfTvw1NTZII+UQJZ7Y07hb+f8+9vtf7D04hVBEKZW5kc3RyZWFtCmVuZG9iagoyMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzMSA+PgpzdHJlYW0KeJw1TzmSBCEMy3mFPjBVGNtAv6entjbY+X+6kplOkPAhydMTHZl4mSMjsGbH21pkIGbgU0zFv/a0DxOq9+AeIpSLC2GGkXDWrONuno4X/3aVz1gH7zb4illeENjCTNZXFmcu2wVjaZzEOclujF0TsY11radTWEcwoQyEdLbDlCBzVKT0yY4y5ug4kSeei+/22yx2OX4O6ws2jSEV5/gqeoI2g6Lsee8CGnJB/13d+B5Fu+glIBsJFtZRYu6c5YRfvXZ0HrUoEnNCmkEuEyHN6SqmEJpQrLOjoFJRcKk+p+isn3/lX1wtCmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicPVA7jkQhDOs5hS/wJPIjcB5Gqy1m79+uA5opUEx+tjMk0BGBRwwxlK/jJa2groG/i0LxbuLrg8Igq0NSIM56D4h07KY2kRM6HZwzP2E3Y47ARTEGnOl0pj0HJjn7wgqEcxtl7FZIJ4mqIo7qM44pnip7n3gWLO3INlsnkj3kIOFSUonJpZ+Uyj9typQKOmbRBCwSueBkE004y7tJUowZlDLqHqZ2In2sPMijOuhkTc6sI5nZ00/bmfgccLdf2mROlcd0Hsz4nLTOgzkVuvfjiTYHTY3a6Oz3E2kqL1K7HVqdfnUSld0Y5xgSl2d/Gd9k//kH/odaIgplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzk1ID4+CnN0cmVhbQp4nD1SS27FQAjb5xRcoNLwm895UlXdvPtva0NSqSq8iTHGMH3KkLnlS10ScYXJt16uWzymfC5bWpl5iLuLjSU+ttyX7iG2XXQusTgdR/ILMp0qRKjNqtGh+EKWhQeQTvChC8J9Of7jL4DB17ANuOE9MkGwJOYpQsZuURmaEkERYeeRFaikUJ9Zwt9R7uv3MgVqb4ylC2Mc9Am0BUJtSMQC6kAAROyUVK2QjmckE78V3WdiHGDn0bIBrhlURJZ77MeIqc6ojLxExD5PTfoolkwtVsZuUxlf/JSM1Hx0BSqpNPKU8tBVs9ALWIl5EvY5/Ej459ZsIYY6btbyieUfM8UyEs5gSzlgoZfjR+DbWXURrh25uM50gR+V1nBMtOt+yPVP/nTbWs11vHIIokDlTUHwuw6uRrHExDI+nY0peqIssBqavEYzwWEQEdb3w8gDGv1yvBA0p2sitFgim7ViRI2KbHM9vQTWTO/FOdbDE8Js753WobIzMyohgtq6hmrrQHazvvNwtp8/M+iibQplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTM2ID4+CnN0cmVhbQp4nE2PQQ4DMQgD73mFn0AgQHjPVlUP2/9fS9h20wseyYBsUQaBJYd4hxvh0dsP30U2FWfjnF9SKWIhmE9wnzBTHI0pd/Jjj4BxlGosp2h4XkvOTcMXLXcTLaWtl5MZb7jul/dHlW2RDUXPLQtC12yS+TKBB3wYmEd142mlx932bK/2/ADObDRJCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicTVFJigMwDLvnFfpAIV6TvKdDmUPn/9fKDoU5BAmvkpOWmFgLDzGEHyw9+JEhczf9G36i2btZepLJ2f+Y5yJTUfhSqC5iQl2IG8+hEfA9oWsSWbG98Tkso5lzvgcfhbgEM6EBY31JMrmo5pUhE04MdRwOWqTCuGtiw+Ja0TyN3G77RmZlJoQNj2RC3BiAiCDrArIYLJQ2NhMyWc4D7Q3JDVpg16kbUYuCK5TWCXSiVsSqzOCz5tZ2N0Mt8uCoffH6aFaXYIXRS/VYeF+FPpipmXbukkJ64U07IsweCqQyOy0rtXvE6m6B+j/LUvD9yff4Ha8PzfxcnAplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTQgPj4Kc3RyZWFtCnicRY3BEcAgCAT/VEEJCgraTyaTh/b/jRAyfGDnDu6EBQu2eUYfBZUmXhVYB0pj3FCPQL3hci3J3AUPcCd/2tBUnJbTd2mRSVUp3KQSef8OZyaQqHnRY533C2P7IzwKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDU0ID4+CnN0cmVhbQp4nDM2M1QwUDCxVDAyNlEwNjQCYhOFFEMuoAiIlcsFE8sBs0CqcrigynNgqnK4MrjSAAUYDjIKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcyID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiGeAmCBtEMUgFkSxmYkZRB2cAZHL4EoDACXbFskKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQ3ID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXJYQVi4XTCwHzALRlnAKIp7BlQYAuWcNJwplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYzID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMjIgPj4Kc3RyZWFtCnicNVG7bcUwDOw1BRcwIH4lzeMgSJG3f5s72qlI07wfVV4ypVwudckqWWHypUN1iqZ8nmam/A71kOOYHtkhulPWlnsYFpaJeUodsZos93ALNr4AmhJzC/H3CPArgFHARKBu8fcPulkSQBoU/BTomquWWGICDYuFrdkV4lbdKVi4q/h2JLkHCXIxWehTDkWKKbfAfBks2ZFanOtyWQr/bn0CGmGFOOyzi0TgecADTCT+ZIBszz5b7OrqRTZ2hjjp0ICLgJvNJAFBUzirPrhh+2q75ueZKCc4OdavojG+DU7mS1LeV7nHz6BB3vgzPGd3jlAOmlAI9N0CIIfdwEaEPrXPwC4Dtkm7d2NK+ZxkKb4ENgr2qFMdyvBi7MxWb9j8x+jKZlFskJX10ekOytygE2Ieb2ShW7K2+zcPs33/AV8Ze2QKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxOCA+PgpzdHJlYW0KeJw9ULmNBDEMy12FGljAeu2pZxaLS6b/9Ej59iLRFkVSKjWZkikvdZQlWVPeOnyWxA55huVuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rnKPn16LGz7D8UMUkZIO5jX/WP3ycw2vU48nkW5vvuJenKkOAxEckpq8I11YsS4SEWk1QU3PwFotgLu3Xv4btCO6DED2icRxmlKOob9rcKXPL+UnU9gKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgzID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4m9j5RlMLevw0QJW64J909XB0JmSluM8NDBp4MLIZdcYH0ljALXEdQjp3so2HVvuoEjfWmUvPvD5Se7KzihusBAkIaZgplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNTEgPj4Kc3RyZWFtCnicMza0UDBQMDQwB5JGhkCWkYlCiiEXSADEzOWCCeaAWQZAGqI4B64mhyuDKw0A4bQNmAplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQzID4+CnN0cmVhbQp4nE1Ru60DMQzrPYUWOMD62b55Lnh4xWX/NqScBKlEQxRJycNTumTKYX1KRkiOLg9tGktsujw3QlOHioKpa4nqlKuZpsxTLE3Q895ZruYY4HtVN9Tf9IheApFRglVhgQ6QO7hg+NlrJmxRCyIxhlAzgGnCCnO4EjEEGYy1ZxiUKgxO1c8qV/svp2XYKrB4MJ0iP7KaaKdfuhx46ykHQtjclbt6IU0I7o0GY8wsXHepsp0AHEx0mYmMWLwNx9MhDA1emgascNaNmCCxGyOlD14HGdOwd0UedbcY8b5bxpS71c99UX3mXe0fCMEbJ/h7AcobXV4KZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM0ID4+CnN0cmVhbQp4nC1SS3LFIAzbcwpdoDP4B+Q86XS6eL3/tpKTRUYOYPQx5YaJSnxZILej1sS3jcxAheGvq8yFz0jbyDqIy5CLuJIthXtELOQxxDzEgu+r8R4e+azMybMHxi/Zdw8r9tSEZSHjxRnaYRXHYRXkWLB1Iap7eFOkw6kk2OOL/z7Fcy0ELXxG0IBf5J+vjuD5khZp95ht0656sEw7qqSwHGxPc14mX1pnuToezwfJ9q7YEVK7AhSFuTPOc+Eo01ZGtBZ2NkhqXGxvjv1YStCFblxGiiOQn6kiPKCkycwmCuKPnB5yKgNh6pqudHIbVXGnnsw1m4u3M0lm675IsZnCeV04s/4MU2a1eSfPcqLUqQjvsWdL0NA5rp69lllodJsTvKSEz8ZOT06+VzPrITkVCaliWlfBaRSZYgnbEl9TUVOaehn++/Lu8Tt+/gEsc3xzCmVuZHN0cmVhbQplbmRvYmoKMzkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MCA+PgpzdHJlYW0KeJwzMzZTMFCwMAISpqaGCuZGlgophlxAPoiVywUTywGzzCzMgSwjC5CWHC5DC2MwbWJspGBmYgZkWSAxILoyuNIAmJoTAwplbmRzdHJlYW0KZW5kb2JqCjQwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIwID4+CnN0cmVhbQp4nDVSS24FMQjbzym4QKXwT87zqqqLvvtvaxO9FUwwYOMpL1nSS77UJdulw+RbH/clsULej+2azFLF9xazFM8tr0fPEbctCgRREz1YmS8VItTP9Og6qHBKn4FXCLcUG7yDSQCDavgHHqUzIFDnQMa7YjJSA4Ik2HNpcQiJciaJf6S8nt8nraSh9D1Zmcvfk0ul0B1NTugBxcrFSaBdSfmgmZhKRJKX632xQvSGwJI8PkcxyYDsNoltogUm5x6lJczEFDqwxwK8ZprVVehgwh6HKYxXC7OoHmzyWxOVpB2t4xnZMN7LMFNioeGwBdTmYmWC7uXjNa/CiO1Rk13DcO6WzXcI0Wj+GxbK4GMVkoBHp7ESDWk4wIjAnl44xV7zEzkOwIhjnZosDGNoJqd6jonA0J6zpWHGxx5a9fMPVOl8hwplbmRzdHJlYW0KZW5kb2JqCjQxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTggPj4Kc3RyZWFtCnicMza0UDCAwxRDrjQAHeYDUgplbmRzdHJlYW0KZW5kb2JqCjQyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMzID4+CnN0cmVhbQp4nEWPSw4EIQhE95yijsDHH+dxMumFc//tgJ1uE2M9hVSBuYKhPS5rA50VHyEZtvG3qZaORVk+VHpSVg/J4Iesxssh3KAs8IJJKoYhUIuYGpEtZW63gNs2DbKylVOljrCLozCP9rRsFR5folsidZI/g8QqL9zjuh3Ipda73qKLvn+kATEJCmVuZHN0cmVhbQplbmRvYmoKNDMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDAgPj4Kc3RyZWFtCnicNVI5bgQxDOv9Cn0ggG7b79kgSJH8vw2p2RQDcXRSlDtaVHbLh4VUtex0+bSV2hI35HdlhcQJyasS7VKGSKi8ViHV75kyr7c1ZwTIUqXC5KTkccmCP8OlpwvH+baxr+XIHY8eWBUjoUTAMsXE6BqWzu6wZlt+lmnAj3iEnCvWLcdYBVIb3TjtiveheS2yBoi9mZaKCh1WiRZ+QfGgR4199hhUWCDR7RxJcIyJUJGAdoHaSAw5eyx2UR/0MygxE+jaG0XcQYElkpg5xbp09N/40LGg/tiMN786KulbWllj0j4b7ZTGLDLpelj0dPPWx4MLNO+i/OfVDBI0ZY2Sxget2jmGoplRVni3Q5MNzTHHIfMOnsMZCUr6PBS/jyUTHZTI3w4NoX9fHqOMnDbeAuaiP20VBw7is8NeuYEVShdrkvcBqUzogen/r/G1vtfXHx3tgMYKZW5kc3RyZWFtCmVuZG9iago0NCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1MSA+PgpzdHJlYW0KeJwtUUlyA0EIu88r9IRmp99jlyuH5P/XCMoHBg2LQHRa4qCMnyAsV7zlkatow98zMYLfBYd+K9dtWORAVCBJY1A1oXbxevQe2HGYCcyT1rAMZqwP/Iwp3OjF4TEZZ7fXZdQQ7F2vPZlByaxcxCUTF0zVYSNnDj+ZMi60cz03IOdGWJdhkG5WGjMSjjSFSCGFqpukzgRBEoyuRo02chT7pS+PdIZVjagx7HMtbV/PTThr0OxYrPLklB5dcS4nFy+sHPT1NgMXUWms8kBIwP1uD/VzspPfeEvnzhbT43vNyfLCVGDFm9duQDbV4t+8iOP7jK/n5/n8A19gW4gKZW5kc3RyZWFtCmVuZG9iago0NSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3NCA+PgpzdHJlYW0KeJxNkEkOQyEMQ/ecwheohDPA5zy/qrpo77+tQwd1gfzkIHA8PNBxJC50ZOiMjiubHOPAsyBj4tE4/8m4PsQxQd2iLViXdsfZzBJzwjIxArZGydk8osAPx1wIEmSXH77AICJdj/lW81mT9M+3O92PurRmXz2iwInsCMWwAVeA/brHgUvC+V7T5JcqJWMTh/KB6iJSNjuhELVU7HKqirPdmytwFfT80UPu7QW1IzzfCmVuZHN0cmVhbQplbmRvYmoKNDYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3NSA+PgpzdHJlYW0KeJwztTRSMFAwNgASpmZGCqYm5gophlxAPoiVy2VoZApm5XAZWZopWFgAGSZm5lAhmIYcLmNTc6ABQEXGpmAaqj+HK4MrDQCVkBLvCmVuZHN0cmVhbQplbmRvYmoKNDcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNDEgPj4Kc3RyZWFtCnicPY/BDsMwCEPv+Qr/QKTYKaF8T6dqh+7/ryNLuwt6AmOMhdDQG6qaw4Zgm+PF0iVUa/gUxUAlN8iZYA6lpNIdR5F6YjgYXB60G47isej6EbuSZn3QxkK6JWiAe6xTadymcRPEHTUF6inqnKO8ELmfqWfYNJLdNLOSc7gNv3vPU9f/p6u8y/kFvXcu/gplbmRzdHJlYW0KZW5kb2JqCjQ4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzYgPj4Kc3RyZWFtCnicPYw7DoAwDEP3nMJHaH4kB0KIgd5/pSm0i/30JNvF0WBakQK3wMnkPqnTcs8kO3wQmyHkVxtata7K0poMi5qMvw3f3U3XC6Y4F8AKZW5kc3RyZWFtCmVuZG9iago0OSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxNSA+PgpzdHJlYW0KeJw1UTkOAyEM7PcV/kAkjC94T6Iozf6/zYzRVh7BXIa0lCGZ8lKTqCHlUz56mS6cutzXzGo055a0LXOAuLa8L62SwIlmiIPBaZi4AZo8AUPX0ahRQxce0NSlUyiw3AQ+irduD91jtYGXtiHniSBiKBksQc2pRRMWbc8npDW/Xosb3pft3chTpcaWGIEGAVY4HNfo1/CVPU8m0XQVMtSrNcsYCRNFIjz5jqbVE+taNNIyEtTGEaxqA7w7/TBOAAATccsCZJ9KlLPkxG+x9LMGV/r+AZ9HVJYKZW5kc3RyZWFtCmVuZG9iagoxNSAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNiAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0NSAvaHlwaGVuIC9wZXJpb2QgNDggL3plcm8gL29uZSAvdHdvIC90aHJlZSAvZm91ciAvZml2ZSAvc2l4Ci9zZXZlbiAvZWlnaHQgL25pbmUgNjggL0QgODAgL1AgOTcgL2EgL2IgL2MgL2QgL2UgL2YgMTA1IC9pIDEwOCAvbCAxMTAgL24KL28gMTEzIC9xIC9yIC9zIC90IC91IC92IDEyMSAveSAveiBdCi9UeXBlIC9FbmNvZGluZyA+PgovRmlyc3RDaGFyIDAgL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udERlc2NyaXB0b3IgMTQgMCBSCi9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdIC9MYXN0Q2hhciAyNTUgL05hbWUgL0RlamFWdVNhbnMKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTMgMCBSID4+CmVuZG9iagoxNCAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE5hbWUgL0RlamFWdVNhbnMgL0l0YWxpY0FuZ2xlIDAKL01heFdpZHRoIDEzNDIgL1N0ZW1WIDAgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9YSGVpZ2h0IDAgPj4KZW5kb2JqCjEzIDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE2IDAgb2JqCjw8IC9EIDE3IDAgUiAvUCAxOCAwIFIgL2EgMTkgMCBSIC9iIDIwIDAgUiAvYyAyMSAwIFIgL2QgMjIgMCBSIC9lIDIzIDAgUgovZWlnaHQgMjQgMCBSIC9mIDI1IDAgUiAvZml2ZSAyNiAwIFIgL2ZvdXIgMjcgMCBSIC9oeXBoZW4gMjggMCBSIC9pIDI5IDAgUgovbCAzMCAwIFIgL24gMzEgMCBSIC9uaW5lIDMyIDAgUiAvbyAzMyAwIFIgL29uZSAzNCAwIFIgL3BlcmlvZCAzNSAwIFIKL3EgMzYgMCBSIC9yIDM3IDAgUiAvcyAzOCAwIFIgL3NldmVuIDM5IDAgUiAvc2l4IDQwIDAgUiAvc3BhY2UgNDEgMCBSCi90IDQyIDAgUiAvdGhyZWUgNDMgMCBSIC90d28gNDQgMCBSIC91IDQ1IDAgUiAvdiA0NiAwIFIgL3kgNDcgMCBSIC96IDQ4IDAgUgovemVybyA0OSAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE1IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL0NBIDAgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PgovQTIgPDwgL0NBIDEgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PgovQTMgPDwgL0NBIDAuNSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAwLjUgPj4KL0E0IDw8IC9DQSAwLjggL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMC44ID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8ID4+CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iago1MCAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MTU1KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDUxCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDI0NDM0IDAwMDAwIG4gCjAwMDAwMjQxNTQgMDAwMDAgbiAKMDAwMDAyNDE4NiAwMDAwMCBuIAowMDAwMDI0MzcxIDAwMDAwIG4gCjAwMDAwMjQzOTIgMDAwMDAgbiAKMDAwMDAyNDQxMyAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDA0MDAgMDAwMDAgbiAKMDAwMDAxMzU4MSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMTM1NTkgMDAwMDAgbiAKMDAwMDAyMjcwNSAwMDAwMCBuIAowMDAwMDIyNTA1IDAwMDAwIG4gCjAwMDAwMjIwMzEgMDAwMDAgbiAKMDAwMDAyMzc1OCAwMDAwMCBuIAowMDAwMDEzNjAxIDAwMDAwIG4gCjAwMDAwMTM4MzggMDAwMDAgbiAKMDAwMDAxNDA4MSAwMDAwMCBuIAowMDAwMDE0NDYxIDAwMDAwIG4gCjAwMDAwMTQ3NzggMDAwMDAgbiAKMDAwMDAxNTA4MyAwMDAwMCBuIAowMDAwMDE1Mzg3IDAwMDAwIG4gCjAwMDAwMTU3MDkgMDAwMDAgbiAKMDAwMDAxNjE3NyAwMDAwMCBuIAowMDAwMDE2Mzg2IDAwMDAwIG4gCjAwMDAwMTY3MDggMDAwMDAgbiAKMDAwMDAxNjg3NCAwMDAwMCBuIAowMDAwMDE3MDAwIDAwMDAwIG4gCjAwMDAwMTcxNDQgMDAwMDAgbiAKMDAwMDAxNzI2MyAwMDAwMCBuIAowMDAwMDE3NDk5IDAwMDAwIG4gCjAwMDAwMTc4OTQgMDAwMDAgbiAKMDAwMDAxODE4NSAwMDAwMCBuIAowMDAwMDE4MzQwIDAwMDAwIG4gCjAwMDAwMTg0NjMgMDAwMDAgbiAKMDAwMDAxODc3OSAwMDAwMCBuIAowMDAwMDE5MDEyIDAwMDAwIG4gCjAwMDAwMTk0MTkgMDAwMDAgbiAKMDAwMDAxOTU2MSAwMDAwMCBuIAowMDAwMDE5OTU0IDAwMDAwIG4gCjAwMDAwMjAwNDQgMDAwMDAgbiAKMDAwMDAyMDI1MCAwMDAwMCBuIAowMDAwMDIwNjYzIDAwMDAwIG4gCjAwMDAwMjA5ODcgMDAwMDAgbiAKMDAwMDAyMTIzNCAwMDAwMCBuIAowMDAwMDIxMzgxIDAwMDAwIG4gCjAwMDAwMjE1OTUgMDAwMDAgbiAKMDAwMDAyMTc0MyAwMDAwMCBuIAowMDAwMDI0NDk0IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gNTAgMCBSIC9Sb290IDEgMCBSIC9TaXplIDUxID4+CnN0YXJ0eHJlZgoyNDY1MQolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["\n", "\n", "def visualize_dequantization(quants, prior=None):\n", " \"\"\"Function for visualizing the dequantization values of discrete values in continuous space.\"\"\"\n", " # Prior over discrete values. If not given, a uniform is assumed\n", " if prior is None:\n", " prior = np.ones(quants, dtype=np.float32) / quants\n", " prior = prior / prior.sum() * quants # In the following, we assume 1 for each value means uniform distribution\n", "\n", " inp = torch.arange(-4, 4, 0.01).view(-1, 1, 1, 1) # Possible continuous values we want to consider\n", " ldj = torch.zeros(inp.shape[0])\n", " dequant_module = Dequantization(quants=quants)\n", " # Invert dequantization on continuous values to find corresponding discrete value\n", " out, ldj = dequant_module.forward(inp, ldj, reverse=True)\n", " inp, out, prob = inp.squeeze().numpy(), out.squeeze().numpy(), ldj.exp().numpy()\n", " prob = prob * prior[out] # Probability scaled by categorical prior\n", "\n", " # Plot volumes and continuous distribution\n", " sns.set_style(\"white\")\n", " _ = plt.figure(figsize=(6, 3))\n", " x_ticks = []\n", " for v in np.unique(out):\n", " indices = np.where(out == v)\n", " color = to_rgb(\"C%i\" % v)\n", " plt.fill_between(inp[indices], prob[indices], np.zeros(indices[0].shape[0]), color=color + (0.5,), label=str(v))\n", " plt.plot([inp[indices[0][0]]] * 2, [0, prob[indices[0][0]]], color=color)\n", " plt.plot([inp[indices[0][-1]]] * 2, [0, prob[indices[0][-1]]], color=color)\n", " x_ticks.append(inp[indices[0][0]])\n", " x_ticks.append(inp.max())\n", " plt.xticks(x_ticks, [\"%.1f\" % x for x in x_ticks])\n", " plt.plot(inp, prob, color=(0.0, 0.0, 0.0))\n", " # Set final plot properties\n", " plt.ylim(0, prob.max() * 1.1)\n", " plt.xlim(inp.min(), inp.max())\n", " plt.xlabel(\"z\")\n", " plt.ylabel(\"Probability\")\n", " plt.title(\"Dequantization distribution for %i discrete values\" % quants)\n", " plt.legend()\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "visualize_dequantization(quants=8)"]}, {"cell_type": "markdown", "id": "b4e465fe", "metadata": {"papermill": {"duration": 0.034926, "end_time": "2021-09-16T12:41:55.616011", "exception": false, "start_time": "2021-09-16T12:41:55.581085", "status": "completed"}, "tags": []}, "source": ["The visualized distribution show the sub-volumes that are assigned to the different discrete values.\n", "The value $0$ has its volume between $[-\\infty, -1.9)$, the value $1$ is represented by the interval $[-1.9, -1.1)$, etc.\n", "The volume for each discrete value has the same probability mass.\n", "That's why the volumes close to the center (e.g. 3 and 4) have a smaller area on the z-axis as others\n", "($z$ is being used to denote the output of the whole dequantization flow).\n", "\n", "Effectively, the consecutive normalizing flow models discrete images by the following objective:\n", "\n", "$$\\log p(x) = \\log \\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right] \\geq \\mathbb{E}_{u}\\left[\\log \\frac{p(x+u)}{q(u|x)} \\right]$$\n", "\n", "Although normalizing flows are exact in likelihood, we have a lower bound.\n", "Specifically, this is an example of the Jensen inequality because we need to move the log into the expectation so we can use Monte-carlo estimates.\n", "In general, this bound is considerably smaller than the ELBO in variational autoencoders.\n", "Actually, we can reduce the bound ourselves by estimating the expectation not by one, but by $M$ samples.\n", "In other words, we can apply importance sampling which leads to the following inequality:\n", "\n", "$$\\log p(x) = \\log \\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right] \\geq \\mathbb{E}_{u}\\left[\\log \\frac{1}{M} \\sum_{m=1}^{M} \\frac{p(x+u_m)}{q(u_m|x)} \\right] \\geq \\mathbb{E}_{u}\\left[\\log \\frac{p(x+u)}{q(u|x)} \\right]$$\n", "\n", "The importance sampling $\\frac{1}{M} \\sum_{m=1}^{M} \\frac{p(x+u_m)}{q(u_m|x)}$ becomes\n", "$\\mathbb{E}_{u\\sim q(u|x)}\\left[\\frac{p(x+u)}{q(u|x)} \\right]$ if $M\\to \\infty$,\n", "so that the more samples we use, the tighter the bound is.\n", "During testing, we can make use of this property and have it implemented in `test_step` in `ImageFlow`.\n", "In theory, we could also use this tighter bound during training.\n", "However, related work has shown that this does not necessarily lead to\n", "an improvement given the additional computational cost, and it is more\n", "efficient to stick with a single estimate [5]."]}, {"cell_type": "markdown", "id": "decab6f5", "metadata": {"papermill": {"duration": 0.03541, "end_time": "2021-09-16T12:41:55.686454", "exception": false, "start_time": "2021-09-16T12:41:55.651044", "status": "completed"}, "tags": []}, "source": ["### Variational Dequantization\n", "\n", "Dequantization uses a uniform distribution for the noise $u$ which effectively leads to images being represented as hypercubes\n", "(cube in high dimensions) with sharp borders.\n", "However, modeling such sharp borders is not easy for a flow as it uses smooth transformations to convert it into a Gaussian distribution.\n", "\n", "Another way of looking at it is if we change the prior distribution in the previous visualization.\n", "Imagine we have independent Gaussian noise on pixels which is commonly the case for any real-world taken picture.\n", "Therefore, the flow would have to model a distribution as above, but with the individual volumes scaled as follows:"]}, {"cell_type": "code", "execution_count": 10, "id": "a67d1060", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:55.760055Z", "iopub.status.busy": "2021-09-16T12:41:55.759590Z", "iopub.status.idle": "2021-09-16T12:41:56.167204Z", "shell.execute_reply": "2021-09-16T12:41:56.167593Z"}, "papermill": {"duration": 0.446187, "end_time": "2021-09-16T12:41:56.167737", "exception": false, "start_time": "2021-09-16T12:41:55.721550", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDM4NC4xODQzNzUgMjIyLjk0ODc1IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSCi9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTIgMCBSID4+CnN0cmVhbQp4nL2dS69ut3Gm5/tXfEN74C3eL0MbSQRk1E4L6EGQgS3LigydBJHtGJ1f3+9bVYuLRe3vnO0jtAx0R3q0Nr+1yGLdSBbj408vX/w6Pr798wP/3yM8/oT/9zf885f895eAf/vwkkd5jaPkXvGv3+//mlJ6nWXgn77Hs/u//fvLyx+Bvnx5yeU15ZjqI/fXKn+FFtt8DR5+v8EUwmtLRq8/36G0/sWvE987vMaKV4949YhXl3/90ujLf71EvPOvAv5TKq+lZ3vt11kfX394+c1Xjy/+KT5ieHz1R/nYr/7w8q+PX/yqvIZfPv7t8dU/v/zjVy+/3VuJMbz2mXLOLaX26Zbi63zWUmmvJcxc66i1v6el+KyljneKqec2yvx0Q+G1Pmto5tcxesutjTk+2VJ42ksp1tfW2+ipxJDf0dCzN0p5vs4RUhkljPLJhp73Uer1tc6eemglfloCng9bbvE19CES+YlGPiJFE6IeGtqYr0kbe97M/zz7pPA6euR0y/m1l8Z/+kgff7KVOl5rzHGGEMf4WN98sqWR8VWzhhryR98pfbKlGMZrKHVg8teWP9JU/nRTOb2ip2Ouodf4sWH7dFMVWquPUgPm78e66qlY302N9JpL6a3GCH3yvKnmm2Irv2J7sUAfsZtnem1Z1fLzVv7XL6EtXuWXf/HDLzFPX+Uf//P3v/v9d99/95f/637k8WPVnYHGI7b8Gsbjh28e/+fxH48vfp1ND2Pat9ES7cRrafa//gLtFOoYKdc0Hz98+/zJh3vyX750T748f/KHb5+YmcXmayu5QILEoLQaoC2Fttyh6UDra2gzzS4U3ZiK0jJGToT4vVCiwhlgSYrQVCeULq3Xa07QnNJAL3g6Ku2l5imwjawQI9BKidoANG1pSuugfSIdMdaSxNhCUSQ8QIjez0NhSRE6UWitFX8kFFoS0i20Qz4JoWDqrDkSYmrnMpS2jj+UBmDNQqNVLVDos9cuDcxS54xKa2wtaAttxjSUwnS1rC3MVDpfocTXXEuDOWNbsYWSlXYapyg0T4waaXqNE/q4CIXopqS0xdjxG6RjBH5EgbhlmbWAmCdlRKWlzN6lgZh7lG4EHVCFUxqILYUsnwZJGH0kbQFzr2alsAejSAsJ7U6OZKmQuopGhOYRxOUAraWMKS1goMcwOluesCikAx0hnQNZGImqATTzJabSEeJs0kIuqSeh/TXhvw9pAS4Q5E1pKyEEbWE2EbECWagzQBwBC1whGXbQ0keo0kCpMekrQBhmh6ESCiFr0sLk68QgLVQIfDXaIZlJWqiYUYEjUelCQUilBYx0k3kCWnvGAAnFuItEg+LTU5AWGkSzSQvxtWC08JmkEMxsdOSQqrTQYNsbh7imV3Q/FUKhMwPjGJXCFUlTWuglRJGRSu2DxqSFDgmSwQTFF+UiLaCjw0Vnqnh5obBuiUJSC5VuntLCaJCtrBS+T4lN6IRwybMQh5HwMCn+PGo/wMEJ8p2kdcTOnqztNaRQhrQwZyilKsWfVwgBZkJAf8ysdMD142iCwgsUcYBlSb3X2oSOMWtV2iamlbSAToCwkw5MhtrgYJHiH4bRClWXpAVacxFUUGgByhFowucMyk6FPPTUurQAmRf5B+wz4vcEQoonB6hxhoQub5vTpFsnsEFnFvl7tN4ie4weEsKALg1AiktRWBrFEf8341OysoF5luTP4ZyL3DUYRKj2In+NTutDIXxJvDMZXB2RW1ihiA+Z8uf4iyR9BVrxUlGehSzi9ZTOEfElhI0uDmHh62ECC50DbSnFk1N/DHNMO7BV6IIJLSjaDDqqKGx1hCwNDEwXURUNUgCxrNIAtGyU0QKFQAXpAQxr0g5sr1AeMch7MRbo0oH9FaoeHr3Qzq9R2qHIId8JuhUzVnoLMtBTpBIErbDmTWmdEZ9OislURQExvsLcTwJLssGaUCWYx9JAHFWnAugoA5JBmuCiirh0ar7OtyHFLK1DKSx0jtIC/k/VZyF8mIJFWsiVrSmtEL4uLRRYYbGooLPkPKUFOENZVELH3zUEEEMorJQMOigcZlhPUhi+Loa6w+OEoDVpgWZU1Acopis1Ks1KmlVEocMvzwP+ltDeQutKS0X/SgsdkzRPpYiHMDmFdqhv9g5iCTRWh7SAqY/fU0obFGSABuSC3dsbTBM+QxqY6HWRENBaU6vSAOQvaOdAGHps1IHQcFTk8mMQhhmofUkbjNdUOjA3aNAwCyKoPDswWKMXaSEyQp1KG+xzkxbgANsHQxrgSVEHgja8mcICrRSlAciaGKNOz6GMLH8PD4dtDpgl2MYmf10Sp4DAXhItIyHNhFBIAoYhyJ/jq0uOSjFzZpYGKvV1VzrxvyotQBfrLIGPDDGYVH+gkMWkEK8UgjQAFdTEu2PoUXtI0kCHChatCsqBLtLAqJBFjsKAIEAtd2kB00UtBmiNCFub0IZBn0pnhi+JFvBXUEX6YhWiDQ81CYUiM9g7fJhGGCHkU36sURnHKQ3AhkCWlLYAyykN4MVFg42OXxh0+gBhr4d4UaCYGhxRUgjlUDgaJDsTog96o9ANiAEsS5QG4CIVUaygsJU5SwOVM1JamDAokFBpAZ5KEDkAhf+Zh7QADa0N0P/DB0kDveBtACcEYbBZQvhK4qBOJkngQcqf41NF6CesSRz0J9iXRd4fDB54DfLX0LlD3MhJVdYqTRjdcrxcVgr9StGggwbDERXCPtGckOKfKqfipPBCX0gDqYUep1J4DRRZDFsyBToxGvBvizQAFSROJCCECGNPCFuqfizoHFNYzdSkZPjvcEaS/Hmd0INN6YAZL/IsJlkTEzgbfBxrFHYl6qOQAFr2Qor+w1wmhQigq6I8O+gQTKXoqiGPQq0V8UQAIUCDqg92BwImpm0OeDiRapsUTHxuUEwD6u/GECSJEZuQAIxWlgbgWoi8A+JrZpW/zzBQlQMDU0LTM4e0wGgzZsNQ6iFIExjZJqNAPOCWJGmkwqSJxMUAQcCYV2kEFnGIISKG+xu6NAKhb6In0M3QbzlMaYRvX4ph+E4xSSMDdqBkwxOqvUgjMK1BH4Y8YMJRAzZ6qsOazgy/IvsD/c2u1zYwFTGZadAQqaGXZzKMODRRgOmzTzZHDBGFiHRpBJp85m4YgVCa0kjmr0zDg90mjZQ2p346OjPQ5JPCYYBBNwxLBG+JuMHrkjANmhJvFfOQNhDKpBgNo7NLlEY6tGUZhvnPWRpBX8ehXQIBQeRZpREGGDEZhtTRWDPaAM3aCMJIvAdt26BBGzUbhnGstG5QOdBvooOZI8Uw1iqN5FD6wgU+TpdGYNPUtyXGj9cpjdBOieXFYMAJog0jrq2YlABDdlqRRtDBpSuGlMBlpXYcjBRMeCK9gdGmNAIPWw0ixhlRNZS0NILJkWc2XOgNoxHm58S4KEZHdSoO4DnC1LYLzWantcMMwq/o6EQqzAwPkjhhHGQ+wlGHP5TgpBJj8qdyYYwf/pYYIp/qNIypAPefGD0sjkuMEBNG5dIGeqTNahjR/szSBgxwTNolEBOMfJU2RsY8jIbh+c0hjUyI3dBvx3DjGZg9BHMMz8IwjIGkFWU8iAjPugS9iV6GFFTGBXDT9E0mA4bQpRFYy2IfOZmfDlMagf2weZYYtmOIpBGMadKeuvG2OPCE3i0f1F7D0fXOB7UPPKj1hqOr6w5q/ezoGpSD2gg6ukb7oCYaBzU5cnQJ3UFNQh1d4nxQk/2D2kRxdM2qg9oUdHTN14Pa5HZ0aYKDmto4qOmYnd4K6aCmvRxdqu6gphcdXUr0oKZxD2rq2dGlyw9qit/RZSQOahbloGZ+HF226qBm2BxdVvCgZjIdXeb1oGaLD2qG29Fl5Q9qLoGjy384qDkbBzXPZKO3F+Oh+Ts7vDwjz8yH2uFytzxUx8wzc+F2uLw9D9Uv3NnyID00Z3OH5pV6ZO6rh+bo7vByiT0z53mHy8/2UD1yz8x33+Fy8z3UeGBnK3Dw0EKMDd7BiIcWtnhoEc4OVzDkocVNO1whlocWjHlocdsOV4TnoQWDO1xxo4cWYe5wBaMeWtzqoYW4O1zRsIcWOO9wxdgeWjS+wxW4e2gxvoeWDtjhyhx4aDmGDd7JCA8tbeGhJTh2uHIhHlraZIcrw+KhJWN2uPI2HlqKx0PLBu1w5Y08tBTTDlc2ykNLXHloOa4drnSYh5Y52+FKsnlo+bgdrtSdh5bl89ASgjtcuUMPLc24wTsj6aElLz3UNOfOVkLUQ8ud7vDKsnpm+dgdrtSth5rk9czSwTtcmWMPNce8s5WN9tAS1zu8UtyeWTLcQ8ub7/DKsHtmqfgdrqS9h5Ld98hWAXa4Fgw81KWFjd2LEB7acsUO18qGh7YI4qGtl+xwLa14aKswO1wLNh7a2o6Htgy0w7Vi5KEtLu1wrUN5aEtWO1yrWx7aQpiHtma2w7W85qGtxO1wLdp5aOt7O1xLgR7aqqGHtsC4wXst0kNbttzhWuH00BZDPbR10x2uFVYPbTF2h2vd1kNb4t3hWg320BaOPbQ15h2u5WgPbeV6h2uR20NbD/fQls53uBbZPbT1+B2upXsPbZV/h/eGAAevvQMOXtsMbrjtSHDw2rywwXubg4PXjggHbe/ExvraZeHgtSFjg2vrhmPXJo8N3vtBHLSdI45de0w2eG9HcTCd2yP//eXlNy+/ffydu3IiN9hMGJoJJ5JbcUIt9Dxpo3749vjPD/ef/+XLj/3nF266YVamVmq1+2U+7JQ+A0RqXIkd7h+BgHbuVWLaTTDEaQwuq3cJMnSdj3jAKEDw4aJE2EpZboC6wRSVfFNnxiINzaACtwJ/axC3wf8J5pJZoSsBPKn7m2G4kLTycJS4Gm5wBgQKg7DMzKwHMRRvihzzTr0MMZuGe6GP9YCrFnIdSX8QE7symUOcA1ywaZgbxiAP3Ic46G0anpNZDOJRuYlTMIxKZOLmoQ5jyEZHZqqKFIHPrPqL0FkVn16JqRSSUdhJzid6p9zPKRnEhCkBi4G5B082cc9gNAxfsRHW1EfOBmcO6OqHucI2LlDG8HEwGl0aa/YassGpCc3oJn2NLLttEPATtxZTvTCXXaM8PQsmhbxd5vaI2ik3zNhc6wXAXGyClQOu6cobAw9u30rEg58lr50hIBgMfM2gohlF88bAPYjIMazA1FS5yZlZTcYYwAgWdIGXGFMffg6xWJliGL0+Ye0RxST6rfomXJDrNAaMeOBwtWa4Q+DhRQzuAwiapocWwGRi+DS4DA+3KRluuQYMNeOrDBuqL9I45wIcGWAM6NAkMzD3S8hWSxg2y7OCjpm4gg/M5QJ7GO5rjHAXiVvqxT4dnm4OVCRD3IiZ9UUG1zxolofMjanTIDMJP7iVgYEmfrE0w/BN46zE8GSK9R8cc03oDrF0uguLeEAESiIuCTIg3cq9UwX/Wok7XA9dQuH+oJYRFT4GHapZZNU2ysYYaMBOnOlVXBhdnPFDDLm5U24YntwEJo1gUmcVTG5KyBgxvAknAvqsGu6V5oUYZjvrshdX02FY4ekwG8AwJBuuk2rnMdBpVYWYK8G6E2Qwtdy6yoiszxauRzH1gJmgHcX1UW65wscwXK3T3hpC0rmfZdDlDUPHgIuJGA4o0EFzWXX3BDeSc/F6ShuTSjMbrpABKNDJvWmMEQ0jBuSK5OQOrjA0o89VKi4fd+LBHU3DcB+NfjjTNQETQZ9GxIyIBsptsturSRqXZFKhez+5H0aXTSNXTWAaMCkmU4J4fBpm4nJM4sLA9MKwBiNmYnyKTg+uJ8BZg/qekHg456EZxktwOxCzVlnyQcSINbPYgCnbCHQbIXGFpx+kEcjCaEYnlAt+r1ASqk4DuMHw6+F8ENeSZBsSKeJiPE4KT62qXoB3neD5TMh6qVNW9slahc+ciTnNZYMDJAPRIIKDRgxVN+uFMdO4UD4bE7rNvq8gyESkN4ghPk1nRq2yBwvePBOK1BYX7mVQKU0OctQtERve3Inn1Fp2dL3GQe2dD2of6OjqjYMm7zFuPXxQGw1Hr5E7oA3yQU0iHF3ic1CTtZ3ecnlQE+KDmsQ7umbHQW0qObrm3UFtkjq6ZvRBbfof1HSFo0uxHFSVkINLYR3UtNtBTRU6urTmQU3FOrr08UFNeTu6NP1BzSwc1GyIo8vgHNSs005vU3ZQs3uOLiN5ULOoBzXz6+iy1Qc1w+7o8gIOqh7DAc27cHS5Igc1v8XR5eMc1BwiR5f3dFBztQ5qfpmjy4k7qHl8ji738KDmSx7UHE9Hl5d6UHNpd7rc3wOaq+zo8qsPKi74wcxbd3S59ge1MMDRFTMc1OILR1cwclCLXA5qYY6jKyY6qAVQjq5o66AWmB3UojhHV8h3UIsPHV3B5EEt8nR0hakHPSPdz43JA9POFkg/gnjkldtKyov7L3pS5u0n/X/RkzLr31+ePylBe+Xmr+6Wdj/slP848f7mNMGOjBYxj+TYU2KUKry/hsClKXAmVlse1XhJXIQT3uFPaZhf6TlnJvAhOa9pUlqFw/41JhGJeWxlGoVu5MJAbDz4g7mkDtjkfuYMNyhKgjk1DWzBEWrDZBOjuXBRKCGuB8bG9NbU/aRsMov3TAyrY3F343GCxoVGjluGz6ZOcONG+spVSXKmpdPFW2TsFZmQH1cohzbxhlztjI2r89N8aXBO0NSF45OyxongPJGCjmyNg6W7LflTaQbuCCfv6LF88Y4YeKLHGiY9pov9bOG2yJay8MEdltk4bHArbGcwa9nz9fzkYaopfMKJsd+FCzgq3arI9RX4IBpmNO5PLJAQOXAauJFAeXvliQSMMjnkqWiSpdE1TR12EzYOPmG1QcHLhRY5zuQt5GK9CcEZCHUgaD3xU2yfFT9mTqgx4Q2SrIOFb4mDvltkgqi1pKETeIdrP5JwhPS6e5TfAlmmX4ipDRXOrjWO0GhSAMEhTrqJn3zOwlxD7NxaWlW88YkFZh66k7hNS850ni7JXCmNzNngtTQt0nmuInFZlZzBoGyfli/vCJDYTGd6y7Ir+MIwAxdDyHmgr18c8xnDDs7ZxsMWxiGMMU7hcJWiRgid8tu5USNyHbRMi+7B4T/KLGcyBc6Dir50SeUmEPLZmLQz3jDLOMsHv3xoPos5r8QzcqRwPfW4HDE8JZ7owYNcO8rrcagQJr3IMd+i9jG6BMMMH0bOQaJTrl5AV+GNOZ3B4S1O3eXHropDMHfIt2Ef23kSgBvZyAva082W4PBMENwLhxsZNBckKSoIKjBDPmYiDXcMf1Uuf0c6uWee59vQQZJs0e114NyuJxT9klQPMedUI7c/kPc5dPf3xnej8ja9Wj6YvYWj640Pqp/n4OqLg1rPHVS72cE1Jge1EXR0jfdBTToOqqLk4BK7g5qQOrpE+qA2ARxd0+WgNrkOalPR0TVxD2rT3NGlFA5qGmSnt7o5qCmng5oqc3QpvoOamnR06dSDmgY+qOlrR5d2P6jZAkeX5Tio2RlHl1U6qNmwg5rFc3TZx4OaNXX0Mr0HNDt9ULPqjl4uwAHNX3B0eRcHVVfEZSCW33JQ9XEOaA6Ro8t9Oqi6Wg4uv+yg5sUd1Hw+R5eHeNDTy/x8h3nIChS3ikhlDyhbbpTimXA7Hs5zqOIwP3ny4Z4Uh/l+8uX5k+Iw0/J1Ln05h/mm3M46KstUmFLnuU3E39wSCy1m28jFIg7uqyNPGAf1e2gRMZFSFF6431bb4fJAY1pHNtxCo2vKmBaxVfHbwHnUXE01OI+S01GPPGvGFxUOkxiy+G3czjswmafxkfCVgjHWVd02MYnoCOUlRD2+KCaxhU4bDl7xCurV0ijKvktiHpbMRuH38mCc7DdG0KA778ERWqGzHtydzOVZtaaSym/itUWemiqWYgfn4ZdQhSPMN79nwER1dggxXrfa4xifmZknJK9lTOuzwQN19CJkqzRC9WkYog+3QPDAI7kYh53heJJzO4PuWB/c0R+CDDl9gRl1KYXpeDiU9Nq4PxuqQw4nxQlZQofLkPMgUmr6Osy8J8ZRxPhsbQV0liY+GzD0pZYAgL3DMzUycuKe8G5pdWAE5uKysdxKQzfJxJsQLrgpdNnAOzf1FeOY/Cnry49SrBOYUEcE1LJwCKXFDeClBXHaYqO/b6EWi8T0maNinkJSjOGZLFEgnMlEjW7AO0+rajMw/3rSLzJZnuHgajvQzXr6mLzWWmTEuVsect2Mz16YICTndnl1yJkY53GyrLylUJtxWLIiQ96YW9EwYHYqHm0cQmzBLCj3hyd9+Tp5llf44JJ9pZseuapUrz4bXN6vMuDc0oteuzjEgAMhfNr5bdZ7gYrlxJODAQl9k4xDzXBTrvDStMADax3BurWmncZVZwlWyDE6zZrpQ8QS0smZ1KO+PVyAIM4+OXwoHskTjokhmmXjm2p+Tq1tR6/3OKC99G4H7k88qHWIo6v7DmqdfVAbGkfXQB40nUmlWz4OatLk6JK9g5qkHtTk2tE1Cw5qc8bRa4Id0GbjQW3uOrpm+kFNLzi6tMhBTeU4uhTUQVWZHdA0n4t3lp48qGlVR5cOPqhp7IOaend0GYODquVwcJmZg5pRcnSZsIOawTtoOA8lbJb0oGZ3Hb2M9AHNoju67P9BzVs4qPkWji5P5KDmtxyhq3k5Bz09pc93+qptPMK789jY5a6xnpC4aoWLwOL0PXny4Z4Up+9+8uX5k3T6qPgGwuV9BeLDTpl7azkn1deJO9V40kpw0nVh4kZ1PBWHLKYJs5A74LjNhhiDU7JhTDkoOsWYZ8XwzHVAiIi5eKHKvXAX3gxFMGarmE/izk0OQzAcjqpPM4WV+C/EiARSN9y4LboJhnYw3FiJjwyTM0qJFzIWvEry0lBlLVyPwjHg5ixiiNrUb2FmkseZBWfWMzLcA6MzwRCtrh0yEMjhReSlAw/dT8OIomKRlw6ditkwc5JNXhpqoxmGSRmIdaWz8Z7D3oSbg1KCpeSObbStNjMG9A78vS4YwZKs2hM3LohFwXnoidYUWaKFrpHggOmbDBeMI2QXeCD6Fk+LeMwO2RDMQ+X6NMtj0IcQzNpo1TAMQ5Y2Ootb6cPc9FYgoYLh+IZiGHOOGVMG4NA39cJTi/FIXN6jlD1I4hvHAvVBHEq1FynczctqR8C1hSq5dRbtSwUiIq9dY806wuJcjlrktQsXuRTT+etV+g8xztWtjZuP65RfZE2mfOEJQY/yNPT3kFgD3gY3Nbciv5i4SS0ZliKE0tvceVi1kcHN0vho4sj8UjLcQuw6wDHU6yMnN2FDTokRIllHTW6uonveuNwFg9oMQ6FzlxcxPlxcafj93Ac+OL48hi/JIMUNIsP5y8zGUNcS4sTdLYPjy70VWk2KtObCqh1M+kOgdEICwzLNIG00zDEV4sQNkzynTAznxkQHmHvTqrx2GaFOfZr1PAK9OmI4eSF7vLtnT+hq+aD2Go6udz6ofeBBrTMcXT13UOvmnd5jclAbwIPaYDu6JOOgJkaOLpk7qAmoo0uaD2qif1CdJg6uKXVQm3+Orsl6UJvZji41cFDTGQc1BePopYwOaIrL0aXlDmoq8aCmPx1dyvagppldcLHU+EFN5zu6DMRBzZoc1EyPo8tOHdSMmqPLAh7UzOVBQz7+fJngg5q9dnQZ94OaJ+DochsOaj7GQc0hcXR5Lwc1V+cIE09v6Sc4frbJvA/8C5c3CRIcNsvRwb6Z3/fmgw/3oLp968GXpw+K15e4Q3bOfX/Uh53KYZCpe55TKnQ/gtYgQY/MMA3DTAatQQJNpVuyiQdr4QjNPdvD3CbeAiv4NNYy0KwJMV4yxq7FUHjqSXBjbixqiRS0HJvR2jBDFWfbu0k8R+LkkDorU5ce4eIxq8eZxC1iPP5zYdb5YRUJyYyEoXjwTFpSCukvF23c8q5toLWhinzyqFsO8jSrLaubBFxkGZa0VB4OUgrfEspGMGarbGJMmQOEjp9aYqYPtdDAPOTIck/MCg1NkkKquGJe9P1ymln7CbhO9JQ0ksTrE8zKaYELvMS0RMMwYnuW05EKOFNzksQDfkOSn0zBzhSkzHJZnbsXuJlOzJZSuEa1a20dhBFB2yjikkwZGp4+Uw8iyx7qFuUXA4sbX3iWzPV24monE1JmkjDRNhU5WRZllZS4j8hv4MEj9JRmfXKTjbqsAwNca+rZcIOayVI/aaJp+8nOk6K0p6w0BA9M5Q+4VKkhSlxbykbx6AhalijRaAsePKzKojqsbISIIV4YDiAdL2K8iLpxeUoFGauDBNNg3cpCJ0kq7NFPatMGYfJwLQ8IEF/1XpKUxaR+JsUgqJsklR8xTbUaE0+siTRIycMxrXQTa3cNw61w0wMpVyulV1lHj8vf8oOly/l2xWVIrbgiy5hB5Y/126S8HmloJevD3H5P+yxFpVhzuhvuJRvl6qQ+XFiqLTZ56TRHs2+BiLBaj9WwSlfT9TUEUTmV4V3QScNfjyxLLmWwxhUScZTyYO6QuGadeJTmKiVWq+zbuWCDAxSkBfq/2gDDsZqTNMDdNioHrFIUi4oSKzOH4uhuGJ5Ba9bR9QoH1bd1cH3YQa0XDqo95uDq3YPaUDh6DdsBbYgdvcThgCY6BzU5c/SSyQOa/Dq6hP2g+VzK3ibRQW3GHdSmp6NrLh/UJr6jS0sc1DTKQU39OLp01UFNsTm6tOBBTWU6uvTrQU0ZH9Q0t6NLzR/UTIKjy34c1IzNQc0yObrM2EHN5jm6DORBzZq6gGuZ3oOqmT6gmXRHl/0/qPoKDi6/4qDmhDi6PJaDmntz0Hie2dzcpoOaj+Xo5Y8d0Hy3g5qjdwTUp6/4E1a4Lwe1/Kh4utDMjV26wv32kw/3pK5wryd/VDx9b/NFFPLM8Iu3HvqwUykP3KIapsGSJOKJRdYLrGbGBsuX8IxWkcIuo6mRnSx1UqzwcEA4fmGW8tN6wnHoQbokpcfgYkob0MRtXJi78qKWPy5We5uYRdeKtpG47CM48rRTbdqIrXaTslK0VgBmER31eit3Guam1YIDa2glw2hXHDHuMp1FtkASzwZPVRthZTttJLNATdOKwSFGLcxI3CEdMWo16HThwu7pWaqOQ8kV+5rCGjlda6fD/GrFxVQr6+l0KymNOE99q0rfhIdiBUc9F0QKD25o9XQG5zo0lSUcWQ5EcIfdLIZ7Zt0TwVVKQBEzjxuHVlAfRU4dKK6IiLWEOiapVugmnoMle6zstkUTdXCJVjwxSBt3xyXDI3VWPSRuWnA5cT8wPGat512K1sMnhbRysZsYf6oZxsa68DU0bSLwzKbhGqQ2KUuKdD2nRgp/gPXm5GzH0N3FiPdZgJ3ST8wNuskwKzs3LWOebS9D4tbhIZdiECNW115t3JmI/rhKoU8dXk5vltvUauq9RHW6Gjcm8isEw+5qBrRl8a6HNpJb1sRjK6yHlbWgeo1d480m+1OzFlTHW/epbdCNy1klB86D7g8ghovFqVxkB3VOF541Zq2pLvsstJEmJ7XTVVxeV/WJO0uhayOhauXxJKsFXGyS+vSjdc1HAsv50qTF7IetWjeWkuRyola+n7o7ghi0avn9DAkN19MIM2vrWj0/DXVx22SptGoF+CFwGu003sGAkdMa/ly8EtwDi7A1LcGP19ByzsTwVpvW4MdHmgveORERDWojSRf/U48sBde1CH8KuUhVTuIRWRVB7xKoWZdpeuIqTK96HUHrWacNtyxX+j2CWchOXySzfN0IeiFBjnqCl7jwYFPVmw6y5Xl5eB0dobX4JZCXMeuF3cYz3MSse9EN94rJV/VmhZA1ZOIu5ibniolZiboarhL3CYaeVAns3PLRp97kEHqzX2Sun9tA9doHyd0phjCEqDdHwFnXkI5nzJscCpsMROsF0dFcV8m6y03lj7uaoVGHNoGo0PppwDfjv+n9FSVYP/HYeGRxUp7DYmZeXxoygtdoSa/ACFr9mbhDTYyq92XkqDNhBBZi5uUMcrlGsyQGD4LDv9JbPwZT4/KTgxq/yXE4SUXqfRXErELY9YoPboPrhqFYs17Gwd7TOTkSK1PmJPd5sM5Qu3Dn3iNthFKkb5IpGtzeLzeNyOz0ePNGntC7ZU+v19jp/c6eXh/o6dUbO727ztOrnze6DYqn1wju9B5uTy/Z8PQSpJ3eUuepCegOb2H29JJ8T3WWOLhm1EFt+jm65upBbWI7urTAQU1lHNT0i6NLGR3UNJejS80d1HSio0t/HtSU7UFNM+/0VuMHNZ3v6DIQBzVrclAzPY4uO3VQM2qOLgt4UDOXji7belAzxAc1q+3oMvEHNXfA0eU7HNQcjYOaV+LocmEOav6Oo8s5Oqh5Uo4ut+ug6qId0Ny5nd6+30HVT3RwuZQHNf/T0eWsHtQ824OaG+zo8pkPag62o8sbP6h57gc1N9/RFRMc1AIIR1e0cVALTRxdccxBLeg5qEVIjq5w6qAWejm64rSDWlB3UIsAXVJrRYsHtdDS0RWHHtSCVkd/HPZ+fgS/FUr69sX965fnfxVrXWvIu9R+cJQXjpXYLoM/oRXi1MvJYG/Uxyg8vsodAlI9K4524RFk922WashazgfjwnxxmeKNNSrxarjBvkfxpBpcYvU4WXEFvMitX7xZSAMn4DJybdIIr3+yF+SmmMTbQoh7s1BtdJ78bVEaYfpAo2hgqdCujSDO1TUlqaEiS2vELItcDVfMi6GNxItB1HsQX6zIqWXBk+tIkitmQdBS2zQ8WD9aWoB6L5q6Z1WUygS/YAbc0zCcd6mtJlsNtKxLmpFnsaUOG4tOFu1rVj9hRX1tI0yLVSYv8wlS3m2ycIWteE3e9UevWfDaxcE6J21Mvacus/SHPp150JxV/IlhwtQPZUWT0KY6hbKUKUMweS0JlLk2ktLU9AswPizo8NIrsvcuPLUXVHTwb03d6ll5npS/TzwsxGehkpC4HkgKvay72Gbj7XwxahtSN8pwLSFmbSPbDnziCUNetRE8rst6EyLSIV3aCMyL/aTcYJCCyDCcR1tRmkOKzSdpBN3ebMgGVz2lOiFrO8NH0kYmqxukro0gaLKfnFyalKqHkylEW0kEHjxiq40w3Sj35wXGoDHrDXyRBzu74Q4RbdoIL6eU2/pClBhP4yOerdS7AYFrk0pUxKxwfOFJj1saCU3r9uSQZCVJLyjEB6iUEDNxo7qBp0qU8uhGqaoaeNVM64YbuqRoGzxYqVhuqamqGgIrVA7DVRYVWahi6oVshBMDQBvO8hVMggnm0tVk0oAYH5WmYRi1phT6QSYkJIlFPdrQNoroE8WVR0z16Zz0ehYEHiwX0vVhhAsStpKWXLgFjDhMzWkRD17/1Ikx7XrWzhusWTIE8mRYNNj0CpQsdwDokm0OHFA4fNoE9FFrhmuSgqvcgWynsUlZOpfqkwUIo94glmOQG5GiPp2mXqNB3OW8BGm0owqQI5Z6mU0x5CBfuDE5KG1A7LUqVpbDQ4Ue/GBjSTIvpKVlSVDT0FVdniQeCLGatsHrPGTAI8x/iFLXHBjyYE9nhpOyeMXbMvQyuMyDN3mKL8ejhFmVOzELETdtI2XdW0iMXudtIrI2lPQGqswDLbOloI2E1GYyPGLleGQORtLbsTJPfsB/rNII80X2fo3VglhziZh3bOpPchdoytoljT3SDfOikqSNcEXreprX1BVtpOTYtVuH7BPo2giMTMyGIRhZh4x2MOmbTOZypOjxkJN+0WhlwR1tg1tQ5UUSUyWsjElcZ62SNCEuLNEmbeDv0ryeHhiQKI1UVrSS105RNhBnbYSylQzDBkrt58Hx0FMxUFdyvxv1J7c3TdF8pJUVS7UNOMIqUbywOA8JvVgfl7dQCc48gsgTKcQpq2SDYuoyA0/KzcjaNG+vqhIADlb+6j0ZRu+wsgBLcU4bmcRDUpnZjCxnODTXSFxgL/pUHNbDvBQ3aBNUWfqDejgoaRvtugIVuLO6n7bBS/b0PbrcDdW1kSJ2UnHlLZjaCMv/DcO8sCJqI7wxRDHLJFSmrKWuaOrWfZy0GGtRWyxcq3fEJtoz+FPaSGh67w4xhC5EaQRulV73lDNTcimo/uRs0zHIzNyyMJPgkbtqHeDJIrXaCFdtREYyoq44oyrQjDHSIcu8r3BIXXhi6Ch9OtH6RdWgubJWpeHK3UnaiBycMgwvQmrTA+cxtI3ML06qQnO2XQPEcC+lxiqnle5Yp5cMY5xEBfASuHBRHrea2gQrdOtLVx6zlqL7vCwoyy01pDXIHQ1cYsZ8uCgvb1ITAb/OXoKl0bp4zaSphm4Y8quDiOi5W8Pc087KfoIbr2w1jHlcVH9ym4PSwY3kKmPQTd0+hJKVqlpGFkgsFx6Ycqo+uTybtefEk6r6crnFCzYW8NQm0tT9xlmSvqOp8kRnF0lEZ1ka4CVKpLHrxVNZlokYVykO1huszttrU8OYuKR8YfiSAjHvm/Y9r3GMsmtdcKr2eyzPmro9PZqm2jNrCdfYjeKTolHE5F0VZ+xFt68QQ3SGGsbYbL9W5t11AUFLVJy0/B1xh7pXyxirOQTcn1LaUMOIYLyoieF2ncZy7oILYulqmPpALWOE4dSZxc1lDKlEYdGR07nCjXy8ilUboc3Qp3nEAqZRG0kseG8YnmlQzclzC2qQWFl6yF21gi0nTEw3X01jRCw2o2F4IVFVJ+Vf3ZAibnxU0xhmsxdhfrhFVZ0w4HokIrNoNiIZtYzQ67rRh5j6Si0jy0Ua5nIk1L42wm2O03DJKallDLx8YBiGJklqGUNretdTZkHwLrcYCo51GMXkyWoYmbqwhzPPfWWddZAWjUmJuQ9NLWModiMgMWKPrJYRPq2umWTWOodcq+rk8oUOGeuij1LUMkJRTFmeg0sjizNqGUPScJe06SY3ncRdtTJLs5dY1TJCPybVFSzj3oJUeyYuWl6SGBFYVcvISwX0pm6Wh5+sBS842GEaYlitFrSREIs9TbFjXTmG+szQhAtXpkqa4qQ1KokRzjZNt084jnbnOOPIbLmvyaX7bhj6vmsiH056VIeXFfXp72gjvU4JzYh5DbOmUXgxmRpHFlJqzN8K5uGbaRgm2+6Ugsev546Ip2weEFznhRMPdNptVZMVGKdhBOd2tRWXUtSo8woCOqnaSLE72HlbAe9i0zbgcOjka3JQ3C7YmpjMIRtmMWbdgzJ5XLEbpnuuG1YYXEg0nZtcBKl7y2aauo5Eiq62O8GklL4+3OSeN83GTarcC/MyX8mPYSYH+0QeFGpR983AGFzvgcgWbomNL6v0Xbh3u0RjBk1cZN5EgVe1t6O2unCLKek2OfgBVV1BXnBx5aVYAl+1y06vVNYTuDe70/sdbrq97g7vT7vp3g87XX22w7t/b7oPxk7XwN1wH+Sd3hKx01t8brrL2k5vubzpLsQ7vSX+pvv02Ok9l3Z6T7yb7rN0p/eUXtTN/53eyuKmu2bZ6a2GdnrrrJveCs7TSxvu9Fadnl561tNLKe/01uCeXtp+p7dp8PSyIzu9jY6nl4Xy9DJnO71tn6eXndzpbVQ9vSywp5e53ult2z29HIGNbl6Dp5eHsdPbHfH08l08vRydnd5ekaeXC7XT29/y9HLOdnp7cp5ebp+nl4+409uf9PRyPnd6e6qeXm6tp+YC7/B2lz29fOudXm64Z5fHvtPbvffUQgEPr7Bho1uM4amFIzu8QxdPrzjH03BuKNjDJ0+vWGunKy7z8IrhdnrHe55abOjhFUbu9I45PbX4dId3KOvpFffu9I6RPb0Cak+v6Hund6ju6RXX7/ROAnh6ZQw8vdILG91yEZ5eiYud3lkOT6+UyE7v/ImnV7LF0yszs9M7jePplfPZ6Z0g8vRKJnl6ZZ52emepPL1SWju981+eXrmynd6JNU+vLJynV8Zup3d6z9MrF7jTO3Ho6ZVl9PRKSW50y196euU6d3onRj29sqg7vVOunl75WU+vZO5O78yvp1eaeKd3TtnTKwG90ztb7emV2vb0yoPv9M6Ze3ol2Hd6Z+M9vVL3nlqaf4f3koCn1/rBTtdag4fXusRGt0UMT23Bw8NrcWSn90qKp+k4obOvz3h6LeZ4ags/O7wXiTy9VpR2ulafPLxWqnZ6L2t5aitgHl6rZTu9l9Y8vdbhdnqv2Xl6LfDt9F4M9PRaOfT0Wmbc6b0m6em1gHnTfbXT02tp1NNrHXWn96Krp9cK7U7v5VxPr7Xfnd4LxZ5eq8qeXkvQO73Xqz291rZ3ei+Ee3qtmnt6LbHv9F6P9/RavN/pvdLv6bUtYKf3HgJPrw0Hnl6bE3Z672Tw9Nr2sNFtj4Sn14aKnd67Lzy99ml4em3q2Om9A8TTa7vITu+9JZ5eG1E8vXat7PTe4uLptR9mp/fmGU+vnTY7vbfleHru7Pm79xTFxz8/MNCPv7EYZbJylPVHh3gClwExByeP+3zxD9/893dff/MvX/7m8fWf37hB7sPGxB0ssvf45X//rO/FOvoIRONuQT7slNGu3qDx2e/2sTvv3nyhd19h97O80BsX83x4drfP54/ee24LeOPd/p6rA37ul3ujmvaHpxW5P/vl3lU29s2Xe38N2Z/55d4q/fjhafnIz365d9Vee/Pl3l2I7ed+tzdqKn14Vpbp89/tHcVL3ny1d1cy+Zlf7cc1AT48qSnw+RPhPUdf33yz956D/blf7Y3zPR+eHRH67HfbNy2//PgF3r1l+f/HC7yxmPXhyWLY5/784+z1t3ymD9uF0npWssvp23WNOfO5iXmQx37rPS8tq0Orrgyebo6TJwu57MDNaaDwzLk1tLYHL1mzNfKI+C1EqSEq9XAoHoK5sbCzOb4N8ytSjTtyZbxxYUx20lfdY/DEv3p6Ue/bN80+uf30yZ2Zzy4jfHLN0NNLc57cIvP0wpUnF5Y8uQrkI/d1vF3c/Vl19P524fGnNb2fVct+uxD155R4frNO6dvVOp8VrHy7POPfXYzwrdpZTyogPa0s80Yhgrd3wj/bgfrmPqxnC++fXGE/VU2CEgmPP1G9Q5H8HbEYq2a3pFEi2vqEkjuffqvlt9b892fvNj7ScuD3QEIevCP8T/h/f7N/ffmvF4Qzj18FMF78gJnZ5M+b/fnXH15+89Xji3+KD4zSV39ES+Hx1R9e/vXxi3/45r/++rv/+Mt3//O7v3z3n//x+MN3f/7LD9/9/q/yL3/8zx8eg+jrH375SDK37X+PX3zzl28e//277//6zZ9/+fi3l6/++eUfv3r57csXvy56wfm3q+N5Yfm3L5Kk53GYLtdqpzLsU5N96oLfy4J/bnJT1qKyYeB6kNcoMIl6s26Pff1yQ9bhvjpvp+VV1hUgXffvsEC4Pnu/0c2+3t/+xrpukPOg+thaSPF+9v6xneat4YXXR3zv6P2594/d3fJGr36tiY37XruVE/hRaZI7J6D32r395MuZPXh3my+y5nN99uAYyDQ72feOMXEmLP2Y2acl+7RrKsg/fvnGbOBmAf9b21QIfioESPLjluSskuzcH5cl+EGF/GOPoLPiy8da8D3Eo7V1UPh8L93c91RvG09v85/SY/fvPu+1+EavvfOKxB8uLfHe3MK72/XdyhObo/2oVxf2ncrbgy6c3sQ/pUvXjz7v0fRmj77rDp3Vo+/NOby7Xd+jrD8Rl9V8i/s+ZW25xdPb/Kf06v27z7s1v9mt76pSv7r1vRmJd7fru5U1scqP5PSivktzWjS9RX9Kd16/+Lwzy9ud+Y7Sr3dfvi9N8d5WfU+yijF6+Ed9eXPfm7FvPL3Nf0qP3r/7vE/r2/P+PZXF7nn/zjzGu9t13corV+Y4Z/2ivktDunF6E/+EDl2/+bw725vduR/5tj776KHw7dV55wYCrvPzb+y+f4wNpzfxT/j++0efd0DfO+C3L/8PePyJmgplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjEzMzkxCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTY0ID4+CnN0cmVhbQp4nD2QwRFDIQhE71axJYCAQD3JZHL4v/9rQJNcZB1g96k7gZBRhzPDZ+LJg9OxNHBvFYxrCK8j9AhNApPAxMGaeAwLAadhkWMu31WWVaeVrpqNnte9Y0HVaZc1DW3agfKtjz/CNd6j8BrsHkIHsSh0bmVaC5lYPGucO8yjzOd+Ttt3PRitptSsN3LZ1z06y9RQXlr7hM5otP0n1y+7MV4fhRQ5CAplbmRzdHJlYW0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTcwID4+CnN0cmVhbQp4nD2QSxLDIAxD95xCRwD/gPO00+mC3H9by5l0gxRjyy9EV3TslYfHxpSN92hjT4QtXOV0Gk5TGY+Lu2ZdoMthMtNvvJq5wFRhkdXsovoYvKHzrGaHr1UzMYQ3mRIaYCp3cg/19ac47duSkGxXYdCdGqSzMMyR/D0QU3PQc4iR/CNfcmth0JnmFxctqxmtZUzR7GGqbC0M6o1Bd8r11Hqu8zAR7/MD30E+ZAplbmRzdHJlYW0KZW5kb2JqCjE5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzA3ID4+CnN0cmVhbQp4nD2SS24DMQxD9z6FLhDA+tme86Qoupjef9snJemKHNkWRWqWukxZUx6QNJOEf+nwcLGd8jtsz2Zm4Fqil4nllOfQFWLuonzZzEZdWSfF6oRmOrfoUTkXBzZNqp+rLKXdLngO1yaeW/YRP7zQoB7UNS4JN3RXo2UpNGOq+3/Se/yMMuBqTF1sUqt7HzxeRFXo6AdHiSJjlxfn40EJ6UrCaFqIlXdFA0Hu8rTKewnu295qyLIHqZjOOylmsOt0Ui5uF4chHsjyqPDlo9hrQs/4sCsl9EjYhjNyJ+5oxubUyOKQ/t6NBEuPrmgh8+CvbtYuYLxTOkViZE5yrGmLVU73UBTTucO9DBD1bEVDKXOR1epfw84La5ZsFnhK+gUeo90mSw5W2duoTu+tPNnQ9x9a13QfCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDQgPj4Kc3RyZWFtCnicRZFNcgUhCIT3nqIv8KrkVz3PpFJZTO6/Dc28JCtaheYD0wITR/ASQ+yJlRMfMnwv6DJ8tzI78DrZmXBPuG5cw2XDM2Fb4DsqyzteQ3e2Uj+doarvGjneLlI1dGVkn3qhmgvMkIiuEVl0K5d1QNOU7lLhGmxbghT1SqwnnaA06BHK8HeUa3x1E0+vseRUzSFaza0TGoqwbHhB1MkkEbUNiyeWcyFR+aobqzouYJMl4vSA3KCVZnx6UkkRMIN8rMlozAI20JO7ZxfGmkseRY5XNJiwO0k18ID34ra+9zZxj/MX+IV33/8rDn3XAj5/AEv+XQYKZW5kc3RyZWFtCmVuZG9iagoyMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIzMiA+PgpzdHJlYW0KeJw1UUluxDAMu/sV/MAA1u68J8Wgh/b/11LKFAhAJba4JWJjIwIvMfg5iNz4kjWjJn5nclf8LE+FR8Kt4EkUgZfhXnaCyxvGZT8OMx+8l1bOpMaTDMhFNj08ETLYJRA6MLsGddhm2om+IeGzI1LNRpbT1xL00ioEylO23+mCEm2r+nP7rAtt+9oTTnZ76knlE4jnlqzAZeMVk8VYBj1RuUsxfZDqbKEnobwon4NsPmqIRJcoZ+CJwcEo0A7sue1n4lUhaF3dp21jqEZKx9O/DU1Nkgj5RAlntjTuFv5/z72+1/sPTiFUEQplbmRzdHJlYW0KZW5kb2JqCjIyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMxID4+CnN0cmVhbQp4nDVPOZIEIQzLeYU+MFUY20C/p6e2Ntj5f7qSmU6Q8CHJ0xMdmXiZIyOwZsfbWmQgZuBTTMW/9rQPE6r34B4ilIsLYYaRcNas426ejhf/dpXPWAfvNviKWV4Q2MJM1lcWZy7bBWNpnMQ5yW6MXROxjXWtp1NYRzChDIR0tsOUIHNUpPTJjjLm6DiRJ56L7/bbLHY5fg7rCzaNIRXn+Cp6gjaDoux57wIackH/Xd34HkW76CUgGwkW1lFi7pzlhF+9dnQetSgSc0KaQS4TIc3pKqYQmlCss6OgUlFwqT6n6Kyff+VfXC0KZW5kc3RyZWFtCmVuZG9iagoyMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OSA+PgpzdHJlYW0KeJw9UDuORCEM6zmFL/Ak8iNwHkarLWbv364DmilQTH62MyTQEYFHDDGUr+MlraCugb+LQvFu4uuDwiCrQ1IgznoPiHTspjaREzodnDM/YTdjjsBFMQac6XSmPQcmOfvCCoRzG2XsVkgniaoijuozjimeKnufeBYs7cg2WyeSPeQg4VJSicmln5TKP23KlAo6ZtEELBK54GQTTTjLu0lSjBmUMuoepnYifaw8yKM66GRNzqwjmdnTT9uZ+Bxwt1/aZE6Vx3QezPictM6DORW69+OJNgdNjdro7PcTaSovUrsdWp1+dRKV3RjnGBKXZ38Z32T/+Qf+h1oiCmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTUgPj4Kc3RyZWFtCnicPVJLbsVACNvnFFyg0vCbz3lSVd28+29rQ1KpKryJMcYwfcqQueVLXRJxhcm3Xq5bPKZ8LltamXmIu4uNJT623JfuIbZddC6xOB1H8gsynSpEqM2q0aH4QpaFB5BO8KELwn05/uMvgMHXsA244T0yQbAk5ilCxm5RGZoSQRFh55EVqKRQn1nC31Hu6/cyBWpvjKULYxz0CbQFQm1IxALqQABE7JRUrZCOZyQTvxXdZ2IcYOfRsgGuGVRElnvsx4ipzqiMvETEPk9N+iiWTC1Wxm5TGV/8lIzUfHQFKqk08pTy0FWz0AtYiXkS9jn8SPjn1mwhhjpu1vKJ5R8zxTISzmBLOWChl+NH4NtZdRGuHbm4znSBH5XWcEy0637I9U/+dNtazXW8cgiiQOVNQfC7Dq5GscTEMj6djSl6oiywGpq8RjPBYRAR1vfDyAMa/XK8EDSnayK0WCKbtWJEjYpscz29BNZM78U51sMTwmzvndahsjMzKiGC2rqGautAdrO+83C2nz8z6KJtCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzYgPj4Kc3RyZWFtCnicTY9BDgMxCAPveYWfQCBAeM9WVQ/b/19L2HbTCx7JgGxRBoElh3iHG+HR2w/fRTYVZ+OcX1IpYiGYT3CfMFMcjSl38mOPgHGUaiynaHheS85NwxctdxMtpa2XkxlvuO6X90eVbZENRc8tC0LXbJL5MoEHfBiYR3XjaaXH3fZsr/b8AM5sNEkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OSA+PgpzdHJlYW0KeJxNUUmKAzAMu+cV+kAhXpO8p0OZQ+f/18oOhTkECa+Sk5aYWAsPMYQfLD34kSFzN/0bfqLZu1l6ksnZ/5jnIlNR+FKoLmJCXYgbz6ER8D2haxJZsb3xOSyjmXO+Bx+FuAQzoQFjfUkyuajmlSETTgx1HA5apMK4a2LD4lrRPI3cbvtGZmUmhA2PZELcGICIIOsCshgslDY2EzJZzgPtDckNWmDXqRtRi4IrlNYJdKJWxKrM4LPm1nY3Qy3y4Kh98fpoVpdghdFL9Vh4X4U+mKmZdu6SQnrhTTsizB4KpDI7LSu1e8TqboH6P8tS8P3J9/gdrw/N/FycCmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA5NCA+PgpzdHJlYW0KeJxFjcERwCAIBP9UQQkKCtpPJpOH9v+NEDJ8YOcO7oQFC7Z5Rh8FlSZeFVgHSmPcUI9AveFyLcncBQ9wJ3/a0FScltN3aZFJVSncpBJ5/w5nJpCoedFjnfcLY/sjPAplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNTQgPj4Kc3RyZWFtCnicMzYzVDBQMLFUMDI2UTA2NAJiE4UUQy6gCIiVywUTywGzQKpyuKDKc2CqcrgyuNIABRgOMgplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzIgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZcQL6piblCLhdIDMTKAbMMgLQlnIKIZ4CYIG0QxSAWRLGZiRlEHZwBkcvgSgMAJdsWyQplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDcgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZclhBWLhdMLAfMAtGWcAoinsGVBgC5Zw0nCmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjMgPj4Kc3RyZWFtCnicRZA7EgMhDEN7TqEj+CMDPs9mMik2929j2GxSwNNYIIO7E4LU2oKJ6IKHtiXdBe+tBGdj/Ok2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlDcPVf9b9i3TmbiYHJyh0IzepT3Pk2O6K6usn+pMfcrNd+K+xVYWlZS8sJt527ZkAJ3FM52qs9Px8KOvYKZW5kc3RyZWFtCmVuZG9iagozMiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMyMiA+PgpzdHJlYW0KeJw1UbttxTAM7DUFFzAgfiXN4yBIkbd/mzvaqUjTvB9VXjKlXC51ySpZYfKlQ3WKpnyeZqb8DvWQ45ge2SG6U9aWexgWlol5Sh2xmiz3cAs2vgCaEnML8fcI8CuAUcBEoG7x9w+6WRJAGhT8FOiaq5ZYYgINi4Wt2RXiVt0pWLir+HYkuQcJcjFZ6FMORYopt8B8GSzZkVqc63JZCv9ufQIaYYU47LOLROB5wANMJP5kgGzPPlvs6upFNnaGOOnQgIuAm80kAUFTOKs+uGH7arvm55koJzg51q+iMb4NTuZLUt5XucfPoEHe+DM8Z3eOUA6aUAj03QIgh93ARoQ+tc/ALgO2Sbt3Y0r5nGQpvgQ2CvaoUx3K8GLszFZv2PzH6MpmUWyQlfXR6Q7K3KATYh5vZKFbsrb7Nw+zff8BXxl7ZAplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE4ID4+CnN0cmVhbQp4nD1QuY0EMQzLXYUaWMB67alnFotLpv/0SPn2ItEWRVIqNZmSKS91lCVZU946fJbEDnmG5W5kNiUqRS+TsCX30ArxfYnmFPfd1ZazQzSXaDl+CzMqqhsd00s2mnAqE7qg3MMz+g1tdANWhx6xWyDQpGDXtiByxw8YDMGZE4siDEpNBv+uco+fXosbPsPxQxSRkg7mNf9Y/fJzDa9TjyeRbm++4l6cqQ4DERySmrwjXVixLhIRaTVBTc/AWi2Au7de/hu0I7oMQPaJxHGaUo6hv2twpc8v5SdT2AplbmRzdHJlYW0KZW5kb2JqCjM0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODMgPj4Kc3RyZWFtCnicRYy7DcAwCER7pmAEfib2PlGUwt6/DRAlbrgn3T1cHQmZKW4zw0MGngwshl1xgfSWMAtcR1COneyjYdW+6gSN9aZS8+8PlJ7srOKG6wECQhpmCmVuZHN0cmVhbQplbmRvYmoKMzUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA1MSA+PgpzdHJlYW0KeJwzNrRQMFAwNDAHkkaGQJaRiUKKIRdIAMTM5YIJ5oBZBkAaojgHriaHK4MrDQDhtA2YCmVuZHN0cmVhbQplbmRvYmoKMzYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDMgPj4Kc3RyZWFtCnicTVG7rQMxDOs9hRY4wPrZvnkueHjFZf82pJwEqURDFEnJw1O6ZMphfUpGSI4uD20aS2y6PDdCU4eKgqlrieqUq5mmzFMsTdDz3lmu5hjge1U31N/0iF4CkVGCVWGBDpA7uGD42WsmbFELIjGGUDOAacIKc7gSMQQZjLVnGJQqDE7VzypX+y+nZdgqsHgwnSI/sppop1+6HHjrKQdC2NyVu3ohTQjujQZjzCxcd6mynQAcTHSZiYxYvA3H0yEMDV6aBqxw1o2YILEbI6UPXgcZ07B3RR51txjxvlvGlLvVz31RfeZd7R8IwRsn+HsByhtdXgplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYwID4+CnN0cmVhbQp4nEWQORIDMQgEc72CJ0hcgvesy7XB+v+pB9ZHoukCNBy6Fk3KehRoPumxRqG60GvoLEqSRMEWkh1Qp2OIOyhITEhjkki2HoMjmlizXZiZVCqzUuG0acXCv9la1chEjXCN/InpBlT8T+pclPBNg6+SMfoYVLw7g4xJ+F5F3Fox7f5EMLEZ9glvRSYFhImxqdm+z2CGzPcK1zjH8w1MgjfrCmVuZHN0cmVhbQplbmRvYmoKMzggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMzQgPj4Kc3RyZWFtCnicLVJLcsUgDNtzCl2gM/gH5DzpdLp4vf+2kpNFRg5g9DHlholKfFkgt6PWxLeNzECF4a+rzIXPSNvIOojLkIu4ki2Fe0Qs5DHEPMSC76vxHh75rMzJswfGL9l3Dyv21IRlIePFGdphFcdhFeRYsHUhqnt4U6TDqSTY44v/PsVzLQQtfEbQgF/kn6+O4PmSFmn3mG3TrnqwTDuqpLAcbE9zXiZfWme5Oh7PB8n2rtgRUrsCFIW5M85z4SjTVka0FnY2SGpcbG+O/VhK0IVuXEaKI5CfqSI8oKTJzCYK4o+cHnIqA2Hqmq50chtVcaeezDWbi7czSWbrvkixmcJ5XTiz/gxTZrV5J89yotSpCO+xZ0vQ0Dmunr2WWWh0mxO8pITPxk5PTr5XM+shORUJqWJaV8FpFJliCdsSX1NRU5p6Gf778u7xO37+ASxzfHMKZW5kc3RyZWFtCmVuZG9iagozOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDcwID4+CnN0cmVhbQp4nDMzNlMwULAwAhKmpoYK5kaWCimGXEA+iJXLBRPLAbPMLMyBLCMLkJYcLkMLYzBtYmykYGZiBmRZIDEgujK40gCYmhMDCmVuZHN0cmVhbQplbmRvYmoKNDAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMjAgPj4Kc3RyZWFtCnicNVJLbgUxCNvPKbhApfBPzvOqqou++29rE70VTDBg4ykvWdJLvtQl26XD5Fsf9yWxQt6P7ZrMUsX3FrMUzy2vR88Rty0KBFETPViZLxUi1M/06DqocEqfgVcItxQbvINJAINq+AcepTMgUOdAxrtiMlIDgiTYc2lxCIlyJol/pLye3yetpKH0PVmZy9+TS6XQHU1O6AHFysVJoF1J+aCZmEpEkpfrfbFC9IbAkjw+RzHJgOw2iW2iBSbnHqUlzMQUOrDHArxmmtVV6GDCHocpjFcLs6gebPJbE5WkHa3jGdkw3sswU2Kh4bAF1OZiZYLu5eM1r8KI7VGTXcNw7pbNdwjRaP4bFsrgYxWSgEensRINaTjAiMCeXjjFXvMTOQ7AiGOdmiwMY2gmp3qOicDQnrOlYcbHHlr18w9U6XyHCmVuZHN0cmVhbQplbmRvYmoKNDEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxOCA+PgpzdHJlYW0KeJwzNrRQMIDDFEOuNAAd5gNSCmVuZHN0cmVhbQplbmRvYmoKNDIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzMgPj4Kc3RyZWFtCnicRY9LDgQhCET3nKKOwMcf53Ey6YVz/+2AnW4TYz2FVIG5gqE9LmsDnRUfIRm28beplo5FWT5UelJWD8ngh6zGyyHcoCzwgkkqhiFQi5gakS1lbreA2zYNsrKVU6WOsIujMI/2tGwVHl+iWyJ1kj+DxCov3OO6Hcil1rveoou+f6QBMQkKZW5kc3RyZWFtCmVuZG9iago0MyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM0MCA+PgpzdHJlYW0KeJw1UjluBDEM6/0KfSCAbtvv2SBIkfy/DanZFANxdFKUO1pUdsuHhVS17HT5tJXaEjfkd2WFxAnJqxLtUoZIqLxWIdXvmTKvtzVnBMhSpcLkpORxyYI/w6WnC8f5trGv5cgdjx5YFSOhRMAyxcToGpbO7rBmW36WacCPeIScK9Ytx1gFUhvdOO2K96F5LbIGiL2ZlooKHVaJFn5B8aBHjX32GFRYINHtHElwjIlQkYB2gdpIDDl7LHZRH/QzKDET6NobRdxBgSWSmDnFunT03/jQsaD+2Iw3vzoq6VtaWWPSPhvtlMYsMul6WPR089bHgws076L859UMEjRljZLGB63aOYaimVFWeLdDkw3NMcch8w6ewxkJSvo8FL+PJRMdlMjfDg2hf18eo4ycNt4C5qI/bRUHDuKzw165gRVKF2uS9wGpTOiB6f+v8bW+19cfHe2AxgplbmRzdHJlYW0KZW5kb2JqCjQ0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjUxID4+CnN0cmVhbQp4nC1RSXIDQQi7zyv0hGan32OXK4fk/9cIygcGDYtAdFrioIyfICxXvOWRq2jD3zMxgt8Fh34r121Y5EBUIEljUDWhdvF69B7YcZgJzJPWsAxmrA/8jCnc6MXhMRlnt9dl1BDsXa89mUHJrFzEJRMXTNVhI2cOP5kyLrRzPTcg50ZYl2GQblYaMxKONIVIIYWqm6TOBEESjK5GjTZyFPulL490hlWNqDHscy1tX89NOGvQ7Fis8uSUHl1xLicXL6wc9PU2AxdRaazyQEjA/W4P9XOyk994S+fOFtPje83J8sJUYMWb125ANtXi37yI4/uMr+fn+fwDX2BbiAplbmRzdHJlYW0KZW5kb2JqCjQ1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTc0ID4+CnN0cmVhbQp4nE2QSQ5DIQxD95zCF6iEM8DnPL+qumjvv61DB3WB/OQgcDw80HEkLnRk6IyOK5sc48CzIGPi0Tj/ybg+xDFB3aItWJd2x9nMEnPCMjECtkbJ2TyiwA/HXAgSZJcfvsAgIl2P+VbzWZP0z7c73Y+6tGZfPaLAiewIxbABV4D9useBS8L5XtPklyolYxOH8oHqIlI2O6EQtVTscqqKs92bK3AV9PzRQ+7tBbUjPN8KZW5kc3RyZWFtCmVuZG9iago0NiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDc1ID4+CnN0cmVhbQp4nDO1NFIwUDA2ABKmZkYKpibmCimGXEA+iJXLZWhkCmblcBlZmilYWAAZJmbmUCGYhhwuY1NzoAFARcamYBqqP4crgysNAJWQEu8KZW5kc3RyZWFtCmVuZG9iago0NyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE0MSA+PgpzdHJlYW0KeJw9j8EOwzAIQ+/5Cv9ApNgpoXxPp2qH7v+vI0u7C3oCY4yF0NAbqprDhmCb48XSJVRr+BTFQCU3yJlgDqWk0h1HkXpiOBhcHrQbjuKx6PoRu5JmfdDGQrolaIB7rFNp3KZxE8QdNQXqKeqco7wQuZ+pZ9g0kt00s5JzuA2/e89T1/+nq7zL+QW9dy7+CmVuZHN0cmVhbQplbmRvYmoKNDggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3NiA+PgpzdHJlYW0KeJw9jDsOgDAMQ/ecwkdofiQHQoiB3n+lKbSL/fQk28XRYFqRArfAyeQ+qdNyzyQ7fBCbIeRXG1q1rsrSmgyLmoy/Dd/dTdcLpjgXwAplbmRzdHJlYW0KZW5kb2JqCjQ5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE1ID4+CnN0cmVhbQp4nDVROQ4DIQzs9xX+QCSML3hPoijN/r/NjNFWHsFchrSUIZnyUpOoIeVTPnqZLpy63NfMajTnlrQtc4C4trwvrZLAiWaIg8FpmLgBmjwBQ9fRqFFDFx7Q1KVTKLDcBD6Kt24P3WO1gZe2IeeJIGIoGSxBzalFExZtzyekNb9eixvel+3dyFOlxpYYgQYBVjgc1+jX8JU9TybRdBUy1Ks1yxgJE0UiPPmOptUT61o00jIS1MYRrGoDvDv9ME4AABNxywJkn0qUs+TEb7H0swZX+v4Bn0dUlgplbmRzdHJlYW0KZW5kb2JqCjE1IDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2FucyAvQ2hhclByb2NzIDE2IDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ1IC9oeXBoZW4gL3BlcmlvZCA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXgKL3NldmVuIC9laWdodCAvbmluZSA2OCAvRCA4MCAvUCA5NyAvYSAvYiAvYyAvZCAvZSAvZiAxMDUgL2kgMTA4IC9sIDExMCAvbgovbyAxMTMgL3EgL3IgL3MgL3QgL3UgL3YgMTIxIC95IC96IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNCAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxMyAwIFIgPj4KZW5kb2JqCjE0IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTMgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTYgMCBvYmoKPDwgL0QgMTcgMCBSIC9QIDE4IDAgUiAvYSAxOSAwIFIgL2IgMjAgMCBSIC9jIDIxIDAgUiAvZCAyMiAwIFIgL2UgMjMgMCBSCi9laWdodCAyNCAwIFIgL2YgMjUgMCBSIC9maXZlIDI2IDAgUiAvZm91ciAyNyAwIFIgL2h5cGhlbiAyOCAwIFIgL2kgMjkgMCBSCi9sIDMwIDAgUiAvbiAzMSAwIFIgL25pbmUgMzIgMCBSIC9vIDMzIDAgUiAvb25lIDM0IDAgUiAvcGVyaW9kIDM1IDAgUgovcSAzNiAwIFIgL3IgMzcgMCBSIC9zIDM4IDAgUiAvc2V2ZW4gMzkgMCBSIC9zaXggNDAgMCBSIC9zcGFjZSA0MSAwIFIKL3QgNDIgMCBSIC90aHJlZSA0MyAwIFIgL3R3byA0NCAwIFIgL3UgNDUgMCBSIC92IDQ2IDAgUiAveSA0NyAwIFIgL3ogNDggMCBSCi96ZXJvIDQ5IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMTUgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMyA8PCAvQ0EgMC41IC9UeXBlIC9FeHRHU3RhdGUgL2NhIDAuNSA+PgovQTQgPDwgL0NBIDAuOCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAwLjggPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgPj4KZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjUwIDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTA5MTYxNDQxNTYrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My40LjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My40LjMpID4+CmVuZG9iagp4cmVmCjAgNTEKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMjQ3NDIgMDAwMDAgbiAKMDAwMDAyNDQ2MiAwMDAwMCBuIAowMDAwMDI0NDk0IDAwMDAwIG4gCjAwMDAwMjQ2NzkgMDAwMDAgbiAKMDAwMDAyNDcwMCAwMDAwMCBuIAowMDAwMDI0NzIxIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDQwMSAwMDAwMCBuIAowMDAwMDEzODg5IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAxMzg2NyAwMDAwMCBuIAowMDAwMDIzMDEzIDAwMDAwIG4gCjAwMDAwMjI4MTMgMDAwMDAgbiAKMDAwMDAyMjMzOSAwMDAwMCBuIAowMDAwMDI0MDY2IDAwMDAwIG4gCjAwMDAwMTM5MDkgMDAwMDAgbiAKMDAwMDAxNDE0NiAwMDAwMCBuIAowMDAwMDE0Mzg5IDAwMDAwIG4gCjAwMDAwMTQ3NjkgMDAwMDAgbiAKMDAwMDAxNTA4NiAwMDAwMCBuIAowMDAwMDE1MzkxIDAwMDAwIG4gCjAwMDAwMTU2OTUgMDAwMDAgbiAKMDAwMDAxNjAxNyAwMDAwMCBuIAowMDAwMDE2NDg1IDAwMDAwIG4gCjAwMDAwMTY2OTQgMDAwMDAgbiAKMDAwMDAxNzAxNiAwMDAwMCBuIAowMDAwMDE3MTgyIDAwMDAwIG4gCjAwMDAwMTczMDggMDAwMDAgbiAKMDAwMDAxNzQ1MiAwMDAwMCBuIAowMDAwMDE3NTcxIDAwMDAwIG4gCjAwMDAwMTc4MDcgMDAwMDAgbiAKMDAwMDAxODIwMiAwMDAwMCBuIAowMDAwMDE4NDkzIDAwMDAwIG4gCjAwMDAwMTg2NDggMDAwMDAgbiAKMDAwMDAxODc3MSAwMDAwMCBuIAowMDAwMDE5MDg3IDAwMDAwIG4gCjAwMDAwMTkzMjAgMDAwMDAgbiAKMDAwMDAxOTcyNyAwMDAwMCBuIAowMDAwMDE5ODY5IDAwMDAwIG4gCjAwMDAwMjAyNjIgMDAwMDAgbiAKMDAwMDAyMDM1MiAwMDAwMCBuIAowMDAwMDIwNTU4IDAwMDAwIG4gCjAwMDAwMjA5NzEgMDAwMDAgbiAKMDAwMDAyMTI5NSAwMDAwMCBuIAowMDAwMDIxNTQyIDAwMDAwIG4gCjAwMDAwMjE2ODkgMDAwMDAgbiAKMDAwMDAyMTkwMyAwMDAwMCBuIAowMDAwMDIyMDUxIDAwMDAwIG4gCjAwMDAwMjQ4MDIgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyA1MCAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgNTEgPj4Kc3RhcnR4cmVmCjI0OTU5CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["visualize_dequantization(quants=8, prior=np.array([0.075, 0.2, 0.4, 0.2, 0.075, 0.025, 0.0125, 0.0125]))"]}, {"cell_type": "markdown", "id": "cbe2bafd", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.039099, "end_time": "2021-09-16T12:41:56.246361", "exception": false, "start_time": "2021-09-16T12:41:56.207262", "status": "completed"}, "tags": []}, "source": ["Transforming such a probability into a Gaussian is a difficult task, especially with such hard borders.\n", "Dequantization has therefore been extended to more sophisticated, learnable distributions beyond uniform in a variational framework.\n", "In particular, if we remember the learning objective\n", "$\\log p(x) = \\log \\mathbb{E}_{u}\\left[\\frac{p(x+u)}{q(u|x)} \\right]$,\n", "the uniform distribution can be replaced by a learned distribution $q_{\\theta}(u|x)$ with support over $u\\in[0,1)^D$.\n", "This approach is called Variational Dequantization and has been proposed by Ho et al.\n", "[3].\n", "How can we learn such a distribution?\n", "We can use a second normalizing flow that takes $x$ as external input and learns a flexible distribution over $u$.\n", "To ensure a support over $[0,1)^D$, we can apply a sigmoid activation function as final flow transformation.\n", "\n", "Inheriting the original dequantization class, we can implement variational dequantization as follows:"]}, {"cell_type": "code", "execution_count": 11, "id": "5b61f162", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:56.333681Z", "iopub.status.busy": "2021-09-16T12:41:56.333192Z", "iopub.status.idle": "2021-09-16T12:41:56.335214Z", "shell.execute_reply": "2021-09-16T12:41:56.334809Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.049683, "end_time": "2021-09-16T12:41:56.335320", "exception": false, "start_time": "2021-09-16T12:41:56.285637", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class VariationalDequantization(Dequantization):\n", " def __init__(self, var_flows, alpha=1e-5):\n", " \"\"\"\n", " Args:\n", " var_flows: A list of flow transformations to use for modeling q(u|x)\n", " alpha: Small constant, see Dequantization for details\n", " \"\"\"\n", " super().__init__(alpha=alpha)\n", " self.flows = nn.ModuleList(var_flows)\n", "\n", " def dequant(self, z, ldj):\n", " z = z.to(torch.float32)\n", " img = (z / 255.0) * 2 - 1 # We condition the flows on x, i.e. the original image\n", "\n", " # Prior of u is a uniform distribution as before\n", " # As most flow transformations are defined on [-infinity,+infinity], we apply an inverse sigmoid first.\n", " deq_noise = torch.rand_like(z).detach()\n", " deq_noise, ldj = self.sigmoid(deq_noise, ldj, reverse=True)\n", " for flow in self.flows:\n", " deq_noise, ldj = flow(deq_noise, ldj, reverse=False, orig_img=img)\n", " deq_noise, ldj = self.sigmoid(deq_noise, ldj, reverse=False)\n", "\n", " # After the flows, apply u as in standard dequantization\n", " z = (z + deq_noise) / 256.0\n", " ldj -= np.log(256.0) * np.prod(z.shape[1:])\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "80820d23", "metadata": {"papermill": {"duration": 0.03912, "end_time": "2021-09-16T12:41:56.414896", "exception": false, "start_time": "2021-09-16T12:41:56.375776", "status": "completed"}, "tags": []}, "source": ["Variational dequantization can be used as a substitute for dequantization.\n", "We will compare dequantization and variational dequantization in later experiments."]}, {"cell_type": "markdown", "id": "a46fc99c", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.038999, "end_time": "2021-09-16T12:41:56.493208", "exception": false, "start_time": "2021-09-16T12:41:56.454209", "status": "completed"}, "tags": []}, "source": ["### Coupling layers\n", "\n", "
\n", "\n", "Next, we look at possible transformations to apply inside the flow.\n", "A recent popular flow layer, which works well in combination with deep neural networks,\n", "is the coupling layer introduced by Dinh et al.\n", "[1].\n", "The input $z$ is arbitrarily split into two parts, $z_{1:j}$ and $z_{j+1:d}$, of which the first remains unchanged by the flow.\n", "Yet, $z_{1:j}$ is used to parameterize the transformation for the second part, $z_{j+1:d}$.\n", "Various transformations have been proposed in recent time [3,4], but here we will settle for the simplest and most efficient one: affine coupling.\n", "In this coupling layer, we apply an affine transformation by shifting the input by a bias $\\mu$ and scale it by $\\sigma$.\n", "In other words, our transformation looks as follows:\n", "\n", "$$z'_{j+1:d} = \\mu_{\\theta}(z_{1:j}) + \\sigma_{\\theta}(z_{1:j}) \\odot z_{j+1:d}$$\n", "\n", "The functions $\\mu$ and $\\sigma$ are implemented as a shared neural network,\n", "and the sum and multiplication are performed element-wise.\n", "The LDJ is thereby the sum of the logs of the scaling factors: $\\sum_i \\left[\\log \\sigma_{\\theta}(z_{1:j})\\right]_i$.\n", "Inverting the layer can as simply be done as subtracting the bias and dividing by the scale:\n", "\n", "$$z_{j+1:d} = \\left(z'_{j+1:d} - \\mu_{\\theta}(z_{1:j})\\right) / \\sigma_{\\theta}(z_{1:j})$$\n", "\n", "We can also visualize the coupling layer in form of a computation graph,\n", "where $z_1$ represents $z_{1:j}$, and $z_2$ represents $z_{j+1:d}$:\n", "\n", "
\n", "\n", "In our implementation, we will realize the splitting of variables as masking.\n", "The variables to be transformed, $z_{j+1:d}$, are masked when passing $z$ to the shared network to predict the transformation parameters.\n", "When applying the transformation, we mask the parameters for $z_{1:j}$\n", "so that we have an identity operation for those variables:"]}, {"cell_type": "code", "execution_count": 12, "id": "89c6ff25", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:56.579882Z", "iopub.status.busy": "2021-09-16T12:41:56.574501Z", "iopub.status.idle": "2021-09-16T12:41:56.581937Z", "shell.execute_reply": "2021-09-16T12:41:56.581514Z"}, "papermill": {"duration": 0.048905, "end_time": "2021-09-16T12:41:56.582037", "exception": false, "start_time": "2021-09-16T12:41:56.533132", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class CouplingLayer(nn.Module):\n", " def __init__(self, network, mask, c_in):\n", " \"\"\"Coupling layer inside a normalizing flow.\n", "\n", " Args:\n", " network: A PyTorch nn.Module constituting the deep neural network for mu and sigma.\n", " Output shape should be twice the channel size as the input.\n", " mask: Binary mask (0 or 1) where 0 denotes that the element should be transformed,\n", " while 1 means the latent will be used as input to the NN.\n", " c_in: Number of input channels\n", " \"\"\"\n", " super().__init__()\n", " self.network = network\n", " self.scaling_factor = nn.Parameter(torch.zeros(c_in))\n", " # Register mask as buffer as it is a tensor which is not a parameter,\n", " # but should be part of the modules state.\n", " self.register_buffer(\"mask\", mask)\n", "\n", " def forward(self, z, ldj, reverse=False, orig_img=None):\n", " \"\"\"\n", " Args:\n", " z: Latent input to the flow\n", " ldj: The current ldj of the previous flows.\n", " The ldj of this layer will be added to this tensor.\n", " reverse: If True, we apply the inverse of the layer.\n", " orig_img (optional): Only needed in VarDeq. Allows external\n", " input to condition the flow on (e.g. original image)\n", " \"\"\"\n", " # Apply network to masked input\n", " z_in = z * self.mask\n", " if orig_img is None:\n", " nn_out = self.network(z_in)\n", " else:\n", " nn_out = self.network(torch.cat([z_in, orig_img], dim=1))\n", " s, t = nn_out.chunk(2, dim=1)\n", "\n", " # Stabilize scaling output\n", " s_fac = self.scaling_factor.exp().view(1, -1, 1, 1)\n", " s = torch.tanh(s / s_fac) * s_fac\n", "\n", " # Mask outputs (only transform the second part)\n", " s = s * (1 - self.mask)\n", " t = t * (1 - self.mask)\n", "\n", " # Affine transformation\n", " if not reverse:\n", " # Whether we first shift and then scale, or the other way round,\n", " # is a design choice, and usually does not have a big impact\n", " z = (z + t) * torch.exp(s)\n", " ldj += s.sum(dim=[1, 2, 3])\n", " else:\n", " z = (z * torch.exp(-s)) - t\n", " ldj -= s.sum(dim=[1, 2, 3])\n", "\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "42bd2031", "metadata": {"papermill": {"duration": 0.039104, "end_time": "2021-09-16T12:41:56.660313", "exception": false, "start_time": "2021-09-16T12:41:56.621209", "status": "completed"}, "tags": []}, "source": ["For stabilization purposes, we apply a $\\tanh$ activation function on the scaling output.\n", "This prevents sudden large output values for the scaling that can destabilize training.\n", "To still allow scaling factors smaller or larger than -1 and 1 respectively,\n", "we have a learnable parameter per dimension, called `scaling_factor`.\n", "This scales the tanh to different limits.\n", "Below, we visualize the effect of the scaling factor on the output activation of the scaling terms:"]}, {"cell_type": "code", "execution_count": 13, "id": "c1bed75e", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:56.769037Z", "iopub.status.busy": "2021-09-16T12:41:56.743280Z", "iopub.status.idle": "2021-09-16T12:41:57.496549Z", "shell.execute_reply": "2021-09-16T12:41:57.496937Z"}, "papermill": {"duration": 0.797388, "end_time": "2021-09-16T12:41:57.497082", "exception": false, "start_time": "2021-09-16T12:41:56.699694", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDcxMC42MTk1MzYwMzUxIDIxMi43NDU2MjUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCniczVvBchy5Db33V/Rx92CaIACCzG23nLhqb951JYdUDi7F3thlecvrbPb384CWNN2jGVkz0zOyq2Srn1og8QiCwBuaxg/D8x9o/PXLiL/GPH7A15/4/qU/DxlP14NRTpW6csXjx/ljoZJMtBYFnpeP/xmGd0NOnayKZW1t3H6QnqnXbG383Yd+ee+Fu4dh6+1h4IIfN9OxWGLJPuT1UHJPVHqxOoM/zmHqOUm5we9sLNCY9+dxxwCYTiqUtVQmbSNVThmzeTv+Y/w0Pv+hTCz+hK8P+AoWh+cv3v7v/dXbn1/+OF59GQR+lMbUF9PeoIuJDL8Mr8bPt4ZzIsUK3dqOx5c36PB5IBD4LONHMKy1dREGUz1JIxi7uh5+fD0+/xuNROPrd0OHH9V6df/y+Prfwz/H7zTl78d/ja9/Gv76Gv7nlMmN5tl3V9du49mLtx/e/P2PX958+vLs+v2nP76ML34bXw2vYrZnoa0RpgBTy9XeoCvQVi1V8NYyIYgP4q0k/VZ5gy+JslCnBXEzeAXmCJueFdZyq6J7qct3hOV5oJ3T+wo3O5bTlt5v4DW815JEYK0iNdkjvF+Eyxm9L5lSaRnWtlLkHbyG9x3usFtjzf0R3uuF1v6Q4+E41y2BWE5Uc2d7XKbgbzVP3BlVSVrus7WBT2FLs8/40WyVb54tHECW77O1gU9hy8w5fzRb9M2zRbklnK/36JrhR6ehmuRmDHHud7I2O4Aum4KIa8p1h+Mb/HTHmZzErzhOF3Ycv1t2JN8Zfrrj2p3ErzheLuz45mS93tt9nOp4q07iVxzntR1POv6Jbqz0lhlFlBrmKFOThhoAD/W2TZPx55fjkqBZp9MVNm4Y6i1106oOcu/ZwA5lTvhp0+JoZdYaaE2K8rUF2pt2OOxwS6boS1GAgZ8mPVD0qZ5C4Ww3FLwwWQIuWA+Rzg73rGgyA2Y0TlYVc/OE1LpqwChZYN95yliO2rj1wDGQYraAke/Nap6MW2rdSD2vcTIyohZ4T60Qo0R0H6yI9ni/CJxDGQU7hFM2l6wcONYEVbXgfYykmGS4hFWr1iox4JaySotYQnzBJzK0fD5S4VxjkgwGqnTB6wWTL1qnUREF8ERg03GQbNOozKlzYX+dEqIjxvQ0RVjWCW3+7oSDdCmtOQ57iEULIgU1YccQGjgLi8ag4uVhBfGBwzWQ7bhxQq+KtZxwxFKt0bWjTqduWNQb/Lab340v91PxgE34wSRmPKI63K0GwOrOsvHBfX1g9Xlw3shw7tGNbcSpFyRqLjDUW0uzpFGWSeOXqzcf33/6dXz35uq/v/3+veccTCM39j/jd3/BKIse5kZAOkTrmWSmbc3nnly01HyK9YTQl7Zs6EQVUSkZm2JO7Axe8Dezcl/38Qi6ewE7pNCxiXLYFXilYf9id+pWKG3gVRQgSZhiRSuGvfIEGtAZCWQYQ57vvWzt3Tt4BQIZmZiVKpIeDp4nEIPOSWAFU73hFF4SuIHXINCLMoE1Q3Kux0lCZ+RAMg5p2K28zCMbeA0OOtI3uzWcePuV2AeFoXNyIKh9TKxvSdEbeAUOhLFHilujvn8fPagOnTMZH3ScHJmLlVEPivHN9C8vGF2GwLlwNCPwdOFoi8An0JAuQ+BcS5oReLqWtEXgE8hKlyFwIS/Na7/T5SXMDx3HnQNHikwXomEuNs1pOF1s2qbhOMnpQjTMpac5DadLT9s0HCdAXYiGeUO5ryFah4bj5Kg1aBhClBqPFqVm3VekW85oI1ASExJHJ3SjDrdSik2XBu5ajVaw8uKCBmDm5MoJGi7AUtAsW8CWpLva5HAn1uYaCZxMTXtjGRsjIatIC5iTCfoOzAQ9nLXaY0ipoJ9EMKQmKtxCDANridGh+Nt4Ad9awDgjxfclhgSdhZD7A0aBWVXR7LeWzJhLCVjwdjfvc3C2NgUbAbvOyF1l7CUVhE0JLxVDciMDjN8zz6sOVw8n9J0tlLaOwWMmtYC3louEdmaYiYQ/Vf2B/ZP7rAkO9B7vG7gwbCgKdQtLrxJTN0VRilK6hgJVFS+GHWtYAPAyuoxWWHSiqyFapWZ46jKaVMGxFjjWy/IkbzU/7DjHsOCaFJS5YNUT/tU+ve8aXOcWqY6yUaOAm0uMnGGewQ27Jug4+HPV0DU4BAMZ/g6YE3ZeceshryE8Y9ReEesEX4CLK3wSkYG2Acs+fT6Aehj8hVOCQ4d6w7oFjg2jUgPH+4RDbsLNBA46DgJJsB0nvCvW2ENMMAf0JM1J8yNBaomI9MK/gcpbXBuFDrsPP0Bt21Nn79NhYHlnCf5gAju4lj8iSR6iukXyCEVvDb2NnkJtQ+pNjPjD7p1Tig2UdEvc/DhDF6zNbOzW2u5eiPJ01WNQsauYzPoyiGbwCqcg9rXvLew2mDns3tA6Wts5CTSk+KxWlhLBDF6DwEqehFiRtf2wubjWdkYCcTTh8MfxtvwgYAavcXENJ6gJrIl6MjtKazsnBy4BNFrq3XfgGv4LEjbO4JxLp8dcQdqhs53Tf9RJqG/isuycgQ28BgcoC3vcyM2s+xX/B4W2cyaSQ06SI9MI6ku9kzmeQme7DH9znW3D3+ky2xZ/TyCzXYa/ucy24e90lW2LvydQ2S7D30JlmxV9p4ts6o3MZv5HimwXYmEuss1YOF1j22bhOI3tQizMNbYZC6dLbNssHCexXYiFefe4pw1ah4XjFLY1WDhVYZv1XGivi7RirrCpFBTCipzpsJXau0sUKt11GlIcbJi7i0+uaGiccoX8YyVDz8BFLOCemAvqX4dbv7mfpXBHM9qyPipyE1r9HLYr/NdcUCXh97r1STJSy2jX/VNPtCOFewsByDud3PB2HWtJFWk7BCBFdsfGNK89vcyY5B9tnq5BTxurusSiVAMuqWOyxv5fTFqpJhPsl9pQrZWxYh4CfDLSML3KTKMVdwuL43AnV4IYzlhcR7O4xqWdEw4SeG4tEfhr4UuvqYFDUIxzrZrrYgFjnEzVVU4n+4Y+VIwJxGOGrv+BDafG4Yq+NiQs16m4Ih87TDgpq/lAnqCLEZV4vcDL7tfO4k4WOuJcWuCYi2QXBl1ya66P+mTgeiq5kZe/HKJfiztl8DwV9DHkGpqFMBnqakVmJSH8LoVy2vzincOIFJcdsQ+EXX2juGpWhVxrbS7iSMXB3GoogFUEOyqHYih+6RCkxXTEkvNowJVc5AoRtKIMQp/b/KaZFgSIlLhBWJX9P8D5LUOKoJQyeYtwxQprwbAIaERzn6ZZsbk7x+uGPYQgDGcRooZiwC/EIUYFGzKkNcwVw+Lsmz5cEDA9rYnhMOjg1j/e8L2I341hsZGN4JV/vOHddG7do6Yi9BBBfvvO8dqI2oS7sI0CxG+TFqzDTQR7T4Kc4yqn45p9/R/AD5AA93QA++QhWN7VHDyUXw/tMQ7P4IfIf4oUoLXffui6jgq4POCG/wO+NiDHCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMjU5NAplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagoxNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM0MSA+PgpzdHJlYW0KeJw1UjvSm0EI679T6AKeWd7LeZzJpPhz/zYCOxUssEIC0gIHmXiJIapRrvglTzBeJ/B3vTyNn8e7kFrwVKQfuDZt4/1YsyYKlkYshdnHvh8l5Hhq/BsCPRdpwoxMRg4kA3G/1ufPepMph9+ANG1OHyVJD6IFu1vDji8LMkh6UsOSnfywrgVWF6EJc2NNJCOnVqbm+dgzXMYTYySomgUk6RP3qYIRacZj56wlDzIcT/Xixa+38VrmMfWyqkDGNsEcbCcz4RRFBOIXlCQ3cRdNHcXRzFhzu9BQUuS+u4eTk173l5OowCshnMVawjFDT1nmZKdBCVStnAAzrNe+ME7TRgl3arq9K/b188wkjNscdlZKpsE5Du5lkzmCZK87JmzC4xDz3j2CkZg3v4stgiuXOddk+rEfRRvpg+L6nKspsxUl/EOVPLHiGv+f3/v58/z+B4wofiMKZW5kc3RyZWFtCmVuZG9iagoxOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMwNyA+PgpzdHJlYW0KeJw9kktuAzEMQ/c+hS4QwPrZnvOkKLqY3n/bJyXpihzZFkVqlrpMWVMekDSThH/p8HCxnfI7bM9mZuBaopeJ5ZTn0BVi7qJ82cxGXVknxeqEZjq36FE5Fwc2Taqfqyyl3S54Dtcmnlv2ET+80KAe1DUuCTd0V6NlKTRjqvt/0nv8jDLgakxdbFKrex88XkRV6OgHR4kiY5cX5+NBCelKwmhaiJV3RQNB7vK0ynsJ7tveasiyB6mYzjspZrDrdFIubheHIR7I8qjw5aPYa0LP+LArJfRI2IYzcifuaMbm1MjikP7ejQRLj65oIfPgr27WLmC8UzpFYmROcqxpi1VO91AU07nDvQwQ9WxFQylzkdXqX8POC2uWbBZ4SvoFHqPdJksOVtnbqE7vrTzZ0PcfWtd0HwplbmRzdHJlYW0KZW5kb2JqCjE5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMyID4+CnN0cmVhbQp4nDVRSW7EMAy7+xX8wADW7rwnxaCH9v/XUsoUCEAltrglYmMjAi8x+DmI3PiSNaMmfmdyV/wsT4VHwq3gSRSBl+FedoLLG8ZlPw4zH7yXVs6kxpMMyEU2PTwRMtglEDowuwZ12Gbaib4h4bMjUs1GltPXEvTSKgTKU7bf6YISbav6c/usC2372hNOdnvqSeUTiOeWrMBl4xWTxVgGPVG5SzF9kOpsoSehvCifg2w+aohElyhn4InBwSjQDuy57WfiVSFoXd2nbWOoRkrH078NTU2SCPlECWe2NO4W/n/Pvb7X+w9OIVQRCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MyA+PgpzdHJlYW0KeJwzNjZXMFAwNASRRkYGCqZAVoohF0jA0MhEIZcLJAhi5YBZBkAaojgHriaHKwPMBmmFqAexIOqNLY2hKhEsiGwGVxoAp8gXrwplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTM2ID4+CnN0cmVhbQp4nE2PQQ4DMQgD73mFn0AgQHjPVlUP2/9fS9h20wseyYBsUQaBJYd4hxvh0dsP30U2FWfjnF9SKWIhmE9wnzBTHI0pd/Jjj4BxlGosp2h4XkvOTcMXLXcTLaWtl5MZb7jul/dHlW2RDUXPLQtC12yS+TKBB3wYmEd142mlx932bK/2/ADObDRJCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDkgPj4Kc3RyZWFtCnicTVFJigMwDLvnFfpAIV6TvKdDmUPn/9fKDoU5BAmvkpOWmFgLDzGEHyw9+JEhczf9G36i2btZepLJ2f+Y5yJTUfhSqC5iQl2IG8+hEfA9oWsSWbG98Tkso5lzvgcfhbgEM6EBY31JMrmo5pUhE04MdRwOWqTCuGtiw+Ja0TyN3G77RmZlJoQNj2RC3BiAiCDrArIYLJQ2NhMyWc4D7Q3JDVpg16kbUYuCK5TWCXSiVsSqzOCz5tZ2N0Mt8uCoffH6aFaXYIXRS/VYeF+FPpipmXbukkJ64U07IsweCqQyOy0rtXvE6m6B+j/LUvD9yff4Ha8PzfxcnAplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzQxID4+CnN0cmVhbQp4nEVSS25EMQjbv1NwgUjhl5DztKq6mN5/W5tM1c3gCWBseMtTpmTKsLklIyTXlE99IkOspvw0ciQipvhJCQV2lY/Ha0usjeyRqBSf2vHjsfRGptkVWvXu0aXNolHNysg5yBChnhW6snvUDtnwelxIuu+UzSEcy/9QgSxl3XIKJUFb0HfsEd8PHa6CK4JhsGsug+1lMtT/+ocWXO9992LHLoAWrOe+wQ4AqKcTtAXIGdruNiloAFW6i0nCo/J6bnaibKNV6fkcADMOMHLAiCVbHb7R3gCWfV3oRY2K/StAUVlA/MjVdsHeMclIcBbmBo69cDzFmXBLOMYCQIq94hh68CXY5i9Xroia8Al1umQvvMKe2ubnQpMId60ADl5kw62ro6iW7ek8gvZnRXJGjNSLODohklrSOYLi0qAeWuNcN7HibSOxuVff7h/hnC9c9usXS+yExAplbmRzdHJlYW0KZW5kb2JqCjI0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzIgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZcQL6piblCLhdIDMTKAbMMgLQlnIKIZ4CYIG0QxSAWRLGZiRlEHZwBkcvgSgMAJdsWyQplbmRzdHJlYW0KZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDcgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZclhBWLhdMLAfMAtGWcAoinsGVBgC5Zw0nCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0JCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzkKL1N1YnR5cGUgL0Zvcm0gL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnic4zI0MFMwNjVVyOUyNzYCs3LALCNzIyALJItgQWQzuNIAFfMKfAplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYzID4+CnN0cmVhbQp4nEWQOxIDIQxDe06hI/gjAz7PZjIpNvdvY9hsUsDTWCCDuxOC1NqCieiCh7Yl3QXvrQRnY/zpNm41EuQEdYBWpONolFJ9ucVplXTxaDZzKwutEx1mDnqUoxmgEDoV3u2i5HKm7s75Q3D1X/W/Yt05m4mBycodCM3qU9z5NjuiurrJ/qTH3KzXfivsVWFpWUvLCbedu2ZACdxTOdqrPT8fCjr2CmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTggPj4Kc3RyZWFtCnicPVC5jQQxDMtdhRpYwHrtqWcWi0um//RI+fYi0RZFUio1mZIpL3WUJVlT3jp8lsQOeYblbmQ2JSpFL5OwJffQCvF9ieYU993VlrNDNJdoOX4LMyqqGx3TSzaacCoTuqDcwzP6DW10A1aHHrFbINCkYNe2IHLHDxgMwZkTiyIMSk0G/65yj59eixs+w/FDFJGSDuY1/1j98nMNr1OPJ5Fub77iXpypDgMRHJKavCNdWLEuEhFpNUFNz8BaLYC7t17+G7QjugxA9onEcZpSjqG/a3Clzy/lJ1PYCmVuZHN0cmVhbQplbmRvYmoKMjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MyA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JvY+UZTC3r8NECVuuCfdPVwdCZkpbjPDQwaeDCyGXXGB9JYwC1xHUI6d7KNh1b7qBI31plLz7w+Unuys4obrAQJCGmYKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDUxID4+CnN0cmVhbQp4nDM2tFAwUDA0MAeSRoZAlpGJQoohF0gAxMzlggnmgFkGQBqiOAeuJocrgysNAOG0DZgKZW5kc3RyZWFtCmVuZG9iagozMSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTggPj4Kc3RyZWFtCnicMza0UDCAwxRDrjQAHeYDUgplbmRzdHJlYW0KZW5kb2JqCjMzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMzID4+CnN0cmVhbQp4nEWPSw4EIQhE95yijsDHH+dxMumFc//tgJ1uE2M9hVSBuYKhPS5rA50VHyEZtvG3qZaORVk+VHpSVg/J4Iesxssh3KAs8IJJKoYhUIuYGpEtZW63gNs2DbKylVOljrCLozCP9rRsFR5folsidZI/g8QqL9zjuh3Ipda73qKLvn+kATEJCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNDAgPj4Kc3RyZWFtCnicNVI5bgQxDOv9Cn0ggG7b79kgSJH8vw2p2RQDcXRSlDtaVHbLh4VUtex0+bSV2hI35HdlhcQJyasS7VKGSKi8ViHV75kyr7c1ZwTIUqXC5KTkccmCP8OlpwvH+baxr+XIHY8eWBUjoUTAMsXE6BqWzu6wZlt+lmnAj3iEnCvWLcdYBVIb3TjtiveheS2yBoi9mZaKCh1WiRZ+QfGgR4199hhUWCDR7RxJcIyJUJGAdoHaSAw5eyx2UR/0MygxE+jaG0XcQYElkpg5xbp09N/40LGg/tiMN786KulbWllj0j4b7ZTGLDLpelj0dPPWx4MLNO+i/OfVDBI0ZY2Sxget2jmGoplRVni3Q5MNzTHHIfMOnsMZCUr6PBS/jyUTHZTI3w4NoX9fHqOMnDbeAuaiP20VBw7is8NeuYEVShdrkvcBqUzogen/r/G1vtfXHx3tgMYKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI1MSA+PgpzdHJlYW0KeJwtUUlyA0EIu88r9IRmp99jlyuH5P/XCMoHBg2LQHRa4qCMnyAsV7zlkatow98zMYLfBYd+K9dtWORAVCBJY1A1oXbxevQe2HGYCcyT1rAMZqwP/Iwp3OjF4TEZZ7fXZdQQ7F2vPZlByaxcxCUTF0zVYSNnDj+ZMi60cz03IOdGWJdhkG5WGjMSjjSFSCGFqpukzgRBEoyuRo02chT7pS+PdIZVjagx7HMtbV/PTThr0OxYrPLklB5dcS4nFy+sHPT1NgMXUWms8kBIwP1uD/VzspPfeEvnzhbT43vNyfLCVGDFm9duQDbV4t+8iOP7jK/n5/n8A19gW4gKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxNSA+PgpzdHJlYW0KeJw1UTkOAyEM7PcV/kAkjC94T6Iozf6/zYzRVh7BXIa0lCGZ8lKTqCHlUz56mS6cutzXzGo055a0LXOAuLa8L62SwIlmiIPBaZi4AZo8AUPX0ahRQxce0NSlUyiw3AQ+irduD91jtYGXtiHniSBiKBksQc2pRRMWbc8npDW/Xosb3pft3chTpcaWGIEGAVY4HNfo1/CVPU8m0XQVMtSrNcsYCRNFIjz5jqbVE+taNNIyEtTGEaxqA7w7/TBOAAATccsCZJ9KlLPkxG+x9LMGV/r+AZ9HVJYKZW5kc3RyZWFtCmVuZG9iagoxNSAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAxNiAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0NiAvcGVyaW9kIDQ4IC96ZXJvIC9vbmUgL3R3byAvdGhyZWUgNTMgL2ZpdmUgNTggL2NvbG9uIDgzIC9TCjk3IC9hIDk5IC9jIDEwMiAvZiAvZyAxMDUgL2kgMTA4IC9sIDExMCAvbiAvbyAxMTQgL3IgMTE2IC90IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxNCAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxMyAwIFIgPj4KZW5kb2JqCjE0IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTMgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTYgMCBvYmoKPDwgL1MgMTcgMCBSIC9hIDE4IDAgUiAvYyAxOSAwIFIgL2NvbG9uIDIwIDAgUiAvZiAyMSAwIFIgL2ZpdmUgMjIgMCBSCi9nIDIzIDAgUiAvaSAyNCAwIFIgL2wgMjUgMCBSIC9uIDI3IDAgUiAvbyAyOCAwIFIgL29uZSAyOSAwIFIKL3BlcmlvZCAzMCAwIFIgL3IgMzEgMCBSIC9zcGFjZSAzMiAwIFIgL3QgMzMgMCBSIC90aHJlZSAzNCAwIFIgL3R3byAzNSAwIFIKL3plcm8gMzYgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAxNSAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0YxLURlamFWdVNhbnMtbWludXMgMjYgMCBSID4+CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagozNyAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MTU3KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDM4CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDEwMjM4IDAwMDAwIG4gCjAwMDAwMTAwMTYgMDAwMDAgbiAKMDAwMDAxMDA0OCAwMDAwMCBuIAowMDAwMDEwMTQ3IDAwMDAwIG4gCjAwMDAwMTAxNjggMDAwMDAgbiAKMDAwMDAxMDE4OSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDA0MDYgMDAwMDAgbiAKMDAwMDAwMzA5NiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDMwNzUgMDAwMDAgbiAKMDAwMDAwODcyNCAwMDAwMCBuIAowMDAwMDA4NTI0IDAwMDAwIG4gCjAwMDAwMDgwOTkgMDAwMDAgbiAKMDAwMDAwOTc3NyAwMDAwMCBuIAowMDAwMDAzMTE2IDAwMDAwIG4gCjAwMDAwMDM1MzAgMDAwMDAgbiAKMDAwMDAwMzkxMCAwMDAwMCBuIAowMDAwMDA0MjE1IDAwMDAwIG4gCjAwMDAwMDQzNjAgMDAwMDAgbiAKMDAwMDAwNDU2OSAwMDAwMCBuIAowMDAwMDA0ODkxIDAwMDAwIG4gCjAwMDAwMDUzMDUgMDAwMDAgbiAKMDAwMDAwNTQ0OSAwMDAwMCBuIAowMDAwMDA1NTY4IDAwMDAwIG4gCjAwMDAwMDU3NDAgMDAwMDAgbiAKMDAwMDAwNTk3NiAwMDAwMCBuIAowMDAwMDA2MjY3IDAwMDAwIG4gCjAwMDAwMDY0MjIgMDAwMDAgbiAKMDAwMDAwNjU0NSAwMDAwMCBuIAowMDAwMDA2Nzc4IDAwMDAwIG4gCjAwMDAwMDY4NjggMDAwMDAgbiAKMDAwMDAwNzA3NCAwMDAwMCBuIAowMDAwMDA3NDg3IDAwMDAwIG4gCjAwMDAwMDc4MTEgMDAwMDAgbiAKMDAwMDAxMDI5OCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDM3IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAzOCA+PgpzdGFydHhyZWYKMTA0NTUKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": ["
"]}, "metadata": {}, "output_type": "display_data"}], "source": ["with torch.no_grad():\n", " x = torch.arange(-5, 5, 0.01)\n", " scaling_factors = [0.5, 1, 2]\n", " sns.set()\n", " fig, ax = plt.subplots(1, 3, figsize=(12, 3))\n", " for i, scale in enumerate(scaling_factors):\n", " y = torch.tanh(x / scale) * scale\n", " ax[i].plot(x.numpy(), y.numpy())\n", " ax[i].set_title(\"Scaling factor: \" + str(scale))\n", " ax[i].set_ylim(-3, 3)\n", " plt.subplots_adjust(wspace=0.4)\n", " sns.reset_orig()\n", " plt.show()"]}, {"cell_type": "markdown", "id": "12dba007", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.041344, "end_time": "2021-09-16T12:41:57.581362", "exception": false, "start_time": "2021-09-16T12:41:57.540018", "status": "completed"}, "tags": []}, "source": ["Coupling layers generalize to any masking technique we could think of.\n", "However, the most common approach for images is to split the input $z$ in half, using a checkerboard mask or channel mask.\n", "A checkerboard mask splits the variables across the height and width dimensions and assigns each other pixel to $z_{j+1:d}$.\n", "Thereby, the mask is shared across channels.\n", "In contrast, the channel mask assigns half of the channels to $z_{j+1:d}$, and the other half to $z_{1:j+1}$.\n", "Note that when we apply multiple coupling layers, we invert the masking for each other layer so that each variable is transformed a similar amount of times.\n", "\n", "Let's implement a function that creates a checkerboard mask and a channel mask for us:"]}, {"cell_type": "code", "execution_count": 14, "id": "deef0879", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:57.685569Z", "iopub.status.busy": "2021-09-16T12:41:57.685029Z", "iopub.status.idle": "2021-09-16T12:41:57.686454Z", "shell.execute_reply": "2021-09-16T12:41:57.686034Z"}, "papermill": {"duration": 0.063936, "end_time": "2021-09-16T12:41:57.686559", "exception": false, "start_time": "2021-09-16T12:41:57.622623", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def create_checkerboard_mask(h, w, invert=False):\n", " x, y = torch.arange(h, dtype=torch.int32), torch.arange(w, dtype=torch.int32)\n", " xx, yy = torch.meshgrid(x, y)\n", " mask = torch.fmod(xx + yy, 2)\n", " mask = mask.to(torch.float32).view(1, 1, h, w)\n", " if invert:\n", " mask = 1 - mask\n", " return mask\n", "\n", "\n", "def create_channel_mask(c_in, invert=False):\n", " mask = torch.cat([torch.ones(c_in // 2, dtype=torch.float32), torch.zeros(c_in - c_in // 2, dtype=torch.float32)])\n", " mask = mask.view(1, c_in, 1, 1)\n", " if invert:\n", " mask = 1 - mask\n", " return mask"]}, {"cell_type": "markdown", "id": "613e51d7", "metadata": {"papermill": {"duration": 0.041153, "end_time": "2021-09-16T12:41:57.769008", "exception": false, "start_time": "2021-09-16T12:41:57.727855", "status": "completed"}, "tags": []}, "source": ["We can also visualize the corresponding masks for an image of size $8\\times 8\\times 2$ (2 channels):"]}, {"cell_type": "code", "execution_count": 15, "id": "6b1bfc65", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:57.857339Z", "iopub.status.busy": "2021-09-16T12:41:57.856865Z", "iopub.status.idle": "2021-09-16T12:41:57.988833Z", "shell.execute_reply": "2021-09-16T12:41:57.988350Z"}, "papermill": {"duration": 0.177754, "end_time": "2021-09-16T12:41:57.988944", "exception": false, "start_time": "2021-09-16T12:41:57.811190", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDE2My44OSAxMTEuMDY1IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSIC9UeXBlIC9QYWdlCj4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTIgMCBSID4+CnN0cmVhbQp4nE2OwU7DMAyG736K/9gdlsRp0jbHjW0V3IYicUAcRhc2xrqprQSvj4M0IJIVO/Hnz4wT6QXjMMHgJPEFRgu9Sp/vXXpsl+gmmhv56ImrUjVB0vMtZWZlKi8P0vJXHIkuNKBW9ifYBeUCGlbeYUx4wgV6YbOTxcniNGgFYJ9FBo39Zbse+p6xumJLWww3zuDwn801DcRy50VgrWLvQ1l7BKdqJ2NoGaE3DLaIb5QtcU/PKO6OqfuYofRCVNajSOPrdTfOwJUKtjZNmQ+KPfrdJI0viA+0jiTr0Dd7GUUPCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMjIzCmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjM1ID4+CnN0cmVhbQp4nDVRSW4AMQi75xX+QKWwJ++Zquqh/f+1hlEvAwPY2CTvwUYkPsSQ7ihXfMrqNMvwO1nkxc9K4eS9iAqkKsIKaQfPclYzDJ4bmQKXM/FZZj6ZFjsWUE3EcXbkNINBiGlcR8vpMNM86Am5PhhxY6dZrmJI691Svb7X8p8qykfW3Sy3TtnUSt2iZ+xJXHZeT21pXxh1FDcFkQ4fO7wH+SLmLC46kW72mymHlaQhOC2AH4mhVM8OrxEmfmYkeMqeTu+jNLz2QdP1vXtBR24mZCq3UEYqnqw0xoyh+o1oJqnv/4Ge9b2+/gBDTVS5CmVuZHN0cmVhbQplbmRvYmoKMTkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMDcgPj4Kc3RyZWFtCnicPZJLbgMxDEP3PoUuEMD62Z7zpCi6mN5/2ycl6Yoc2RZFapa6TFlTHpA0k4R/6fBwsZ3yO2zPZmbgWqKXieWU59AVYu6ifNnMRl1ZJ8XqhGY6t+hRORcHNk2qn6sspd0ueA7XJp5b9hE/vNCgHtQ1Lgk3dFejZSk0Y6r7f9J7/Iwy4GpMXWxSq3sfPF5EVejoB0eJImOXF+fjQQnpSsJoWoiVd0UDQe7ytMp7Ce7b3mrIsgepmM47KWaw63RSLm4XhyEeyPKo8OWj2GtCz/iwKyX0SNiGM3In7mjG5tTI4pD+3o0ES4+uaCHz4K9u1i5gvFM6RWJkTnKsaYtVTvdQFNO5w70MEPVsRUMpc5HV6l/DzgtrlmwWeEr6BR6j3SZLDlbZ26hO76082dD3H1rXdB8KZW5kc3RyZWFtCmVuZG9iagoyMCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NCA+PgpzdHJlYW0KeJxFkU1yBSEIhPeeoi/wquRXPc+kUllM7r8NzbwkK1qF5gPTAhNH8BJD7ImVEx8yfC/oMny3MjvwOtmZcE+4blzDZcMzYVvgOyrLO15Dd7ZSP52hqu8aOd4uUjV0ZWSfeqGaC8yQiK4RWXQrl3VA05TuUuEabFuCFPVKrCedoDToEcrwd5RrfHUTT6+x5FTNIVrNrRMairBseEHUySQRtQ2LJ5ZzIVH5qhurOi5gkyXi9IDcoJVmfHpSSREwg3ysyWjMAjbQk7tnF8aaSx5Fjlc0mLA7STXwgPfitr73NnGP8xf4hXff/ysOfdcCPn8AS/5dBgplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjMyID4+CnN0cmVhbQp4nDVRSW7EMAy7+xX8wADW7rwnxaCH9v/XUsoUCEAltrglYmMjAi8x+DmI3PiSNaMmfmdyV/wsT4VHwq3gSRSBl+FedoLLG8ZlPw4zH7yXVs6kxpMMyEU2PTwRMtglEDowuwZ12Gbaib4h4bMjUs1GltPXEvTSKgTKU7bf6YISbav6c/usC2372hNOdnvqSeUTiOeWrMBl4xWTxVgGPVG5SzF9kOpsoSehvCifg2w+aohElyhn4InBwSjQDuy57WfiVSFoXd2nbWOoRkrH078NTU2SCPlECWe2NO4W/n/Pvb7X+w9OIVQRCmVuZHN0cmVhbQplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMzEgPj4Kc3RyZWFtCnicNU85kgQhDMt5hT4wVRjbQL+np7Y22Pl/upKZTpDwIcnTEx2ZeJkjI7Bmx9taZCBm4FNMxb/2tA8TqvfgHiKUiwthhpFw1qzjbp6OF/92lc9YB+82+IpZXhDYwkzWVxZnLtsFY2mcxDnJboxdE7GNda2nU1hHMKEMhHS2w5Qgc1Sk9MmOMuboOJEnnovv9tssdjl+DusLNo0hFef4KnqCNoOi7HnvAhpyQf9d3fgeRbvoJSAbCRbWUWLunOWEX712dB61KBJzQppBLhMhzekqphCaUKyzo6BSUXCpPqforJ9/5V9cLQplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ5ID4+CnN0cmVhbQp4nD1QO45EIQzrOYUv8CTyI3AeRqstZu/frgOaKVBMfrYzJNARgUcMMZSv4yWtoK6Bv4tC8W7i64PCIKtDUiDOeg+IdOymNpETOh2cMz9hN2OOwEUxBpzpdKY9ByY5+8IKhHMbZexWSCeJqiKO6jOOKZ4qe594FiztyDZbJ5I95CDhUlKJyaWflMo/bcqUCjpm0QQsErngZBNNOMu7SVKMGZQy6h6mdiJ9rDzIozroZE3OrCOZ2dNP25n4HHC3X9pkTpXHdB7M+Jy0zoM5Fbr344k2B02N2ujs9xNpKi9Sux1anX51EpXdGOcYEpdnfxnfZP/5B/6HWiIKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2NCA+PgpzdHJlYW0KeJxFkMdxBTEMQ++qAiUwgAr1rMfzD+v+r4b000F6GEIMYk/CsFxXcWF0w4+3LTMNf0cZ7sb6MmO81VggJ+gDDJGJq9Gk+nbFGar05NVirqOiXC86IhLMkuOrQCN8OrLHk7a2M/10Xh/sIe8T/yoq525hAS6q7kD5Uh/x1I/ZUeqaoY8qK2seatpXhF0RSts+LqcyTt29A1rhvZWrPdrvPx52OvIKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgzID4+CnN0cmVhbQp4nD3MORKAMAgF0J5T/COEyCL3cRyLeP9WMNEGHqt6oCE4g7rBreFgyrp0E+9T49XGnBIJqHhKTZa6C3rUtL7Uvmjgu+vmS9WJP83PF50Pux0Z3QplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjU4ID4+CnN0cmVhbQp4nEWRS3IEIAhE956CI4D85DyTSmUxuf82Dc5kNnaXqP2ESiOmEiznFHkwfcnyzWS26Xc5VjsbBRRFKJjJVeixAqs7U8SZa4lq62Nl5LjTOwbFG85dOalkcaOMdVR1KnBMz5X1Ud35dlmUfUcOZQrYrHMcbODKbcMYJ0abre4O94kgTydTR8XtINnwByeNfZWrK3CdbPbRSzAOBP1CE5jki0DrDIHGzVP05BLs4+N254Fgb3kRSNkQyJEhGB2Cdp1c/+LW+b3/cYY7z7UZrhzv4neY1nbHX2KSFXMBi9wpqOdrLlrXGTrekzPH5Kb7hs65YJe7g0zv+T/Wz/r+Ax4pZvoKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxOCA+PgpzdHJlYW0KeJw9ULmNBDEMy12FGljAeu2pZxaLS6b/9Ej59iLRFkVSKjWZkikvdZQlWVPeOnyWxA55huVuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rnKPn16LGz7D8UMUkZIO5jX/WP3ycw2vU48nkW5vvuJenKkOAxEckpq8I11YsS4SEWk1QU3PwFotgLu3Xv4btCO6DED2icRxmlKOob9rcKXPL+UnU9gKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJxFkDkSAzEIBHO9gidIXIL3rMu1wfr/qQfWR6LpAjQcuhZNynoUaD7psUahutBr6CxKkkTBFpIdUKdjiDsoSExIY5JIth6DI5pYs12YmVQqs1LhtGnFwr/ZWtXIRI1wjfyJ6QZU/E/qXJTwTYOvkjH6GFS8O4OMSfheRdxaMe3+RDCxGfYJb0UmBYSJsanZvs9ghsz3Ctc4x/MNTII36wplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM0ID4+CnN0cmVhbQp4nC1SS3LFIAzbcwpdoDP4B+Q86XS6eL3/tpKTRUYOYPQx5YaJSnxZILej1sS3jcxAheGvq8yFz0jbyDqIy5CLuJIthXtELOQxxDzEgu+r8R4e+azMybMHxi/Zdw8r9tSEZSHjxRnaYRXHYRXkWLB1Iap7eFOkw6kk2OOL/z7Fcy0ELXxG0IBf5J+vjuD5khZp95ht0656sEw7qqSwHGxPc14mX1pnuToezwfJ9q7YEVK7AhSFuTPOc+Eo01ZGtBZ2NkhqXGxvjv1YStCFblxGiiOQn6kiPKCkycwmCuKPnB5yKgNh6pqudHIbVXGnnsw1m4u3M0lm675IsZnCeV04s/4MU2a1eSfPcqLUqQjvsWdL0NA5rp69lllodJsTvKSEz8ZOT06+VzPrITkVCaliWlfBaRSZYgnbEl9TUVOaehn++/Lu8Tt+/gEsc3xzCmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxOCA+PgpzdHJlYW0KeJwzNrRQMIDDFEOuNAAd5gNSCmVuZHN0cmVhbQplbmRvYmoKMTYgMCBvYmoKPDwgL0Jhc2VGb250IC9EZWphVnVTYW5zIC9DaGFyUHJvY3MgMTcgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNjcgL0MgOTcgL2EgL2IgL2MgL2QgL2UgMTA0IC9oIDEwNyAvayAxMDkgL20gMTExIC9vIDExNCAvciAvcwpdCi9UeXBlIC9FbmNvZGluZyA+PgovRmlyc3RDaGFyIDAgL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udERlc2NyaXB0b3IgMTUgMCBSCi9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdIC9MYXN0Q2hhciAyNTUgL05hbWUgL0RlamFWdVNhbnMKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTQgMCBSID4+CmVuZG9iagoxNSAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE5hbWUgL0RlamFWdVNhbnMgL0l0YWxpY0FuZ2xlIDAKL01heFdpZHRoIDEzNDIgL1N0ZW1WIDAgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9YSGVpZ2h0IDAgPj4KZW5kb2JqCjE0IDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjE3IDAgb2JqCjw8IC9DIDE4IDAgUiAvYSAxOSAwIFIgL2IgMjAgMCBSIC9jIDIxIDAgUiAvZCAyMiAwIFIgL2UgMjMgMCBSIC9oIDI0IDAgUgovayAyNSAwIFIgL20gMjYgMCBSIC9vIDI3IDAgUiAvciAyOCAwIFIgL3MgMjkgMCBSIC9zcGFjZSAzMCAwIFIgPj4KZW5kb2JqCjMgMCBvYmoKPDwgL0YxIDE2IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL0NBIDAgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PgovQTIgPDwgL0NBIDEgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvSTEgMTMgMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvQml0c1BlckNvbXBvbmVudCA4IC9Db2xvclNwYWNlIFsvSW5kZXhlZCAvRGV2aWNlUkdCIDIgKP///39/fwAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgMTUwIC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCA4MiAvTGVuZ3RoIDMxIDAgUiAvU3VidHlwZSAvSW1hZ2UKL1R5cGUgL1hPYmplY3QgL1dpZHRoIDE1MCA+PgpzdHJlYW0KeJzt1DEKA0EQA8H1/v/RhvNERnDXgfGCWtkkTUWzXkdu/RuQJ4tMFpksMllksshkkX2x9rX12b695kiN/bSxU0OWrG4WbM2RGutpY8WGLFmyzmPB1hyp0fBOZckqY8HWHKnR8E5lySpjwdYcqdHwTmXJKmPB1hyp0fBOZckqY8HWHKnR8E5lyfoVC7bmSI2GvyVLVhnrlMkik0Umi0wWmSwyWWSHst5k4DANCmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKMTcxCmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagozMiAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MTU3KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDMzCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA2ODM2IDAwMDAwIG4gCjAwMDAwMDYxODcgMDAwMDAgbiAKMDAwMDAwNjIxOSAwMDAwMCBuIAowMDAwMDA2MzE4IDAwMDAwIG4gCjAwMDAwMDYzMzkgMDAwMDAgbiAKMDAwMDAwNjM2MCAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzOTUgMDAwMDAgbiAKMDAwMDAwMDcxMyAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA2OTMgMDAwMDAgbiAKMDAwMDAwNjM5MiAwMDAwMCBuIAowMDAwMDA0OTc4IDAwMDAwIG4gCjAwMDAwMDQ3NzggMDAwMDAgbiAKMDAwMDAwNDQxMyAwMDAwMCBuIAowMDAwMDA2MDMxIDAwMDAwIG4gCjAwMDAwMDA3MzMgMDAwMDAgbiAKMDAwMDAwMTA0MSAwMDAwMCBuIAowMDAwMDAxNDIxIDAwMDAwIG4gCjAwMDAwMDE3MzggMDAwMDAgbiAKMDAwMDAwMjA0MyAwMDAwMCBuIAowMDAwMDAyMzQ3IDAwMDAwIG4gCjAwMDAwMDI2NjkgMDAwMDAgbiAKMDAwMDAwMjkwNiAwMDAwMCBuIAowMDAwMDAzMDYxIDAwMDAwIG4gCjAwMDAwMDMzOTIgMDAwMDAgbiAKMDAwMDAwMzY4MyAwMDAwMCBuIAowMDAwMDAzOTE2IDAwMDAwIG4gCjAwMDAwMDQzMjMgMDAwMDAgbiAKMDAwMDAwNjgxNiAwMDAwMCBuIAowMDAwMDA2ODk2IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMzIgMCBSIC9Sb290IDEgMCBSIC9TaXplIDMzID4+CnN0YXJ0eHJlZgo3MDUzCiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": ["
"]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["checkerboard_mask = create_checkerboard_mask(h=8, w=8).expand(-1, 2, -1, -1)\n", "channel_mask = create_channel_mask(c_in=2).expand(-1, -1, 8, 8)\n", "\n", "show_imgs(checkerboard_mask.transpose(0, 1), \"Checkerboard mask\")\n", "show_imgs(channel_mask.transpose(0, 1), \"Channel mask\")"]}, {"cell_type": "markdown", "id": "b52dfa44", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.042944, "end_time": "2021-09-16T12:41:58.075778", "exception": false, "start_time": "2021-09-16T12:41:58.032834", "status": "completed"}, "tags": []}, "source": ["As a last aspect of coupling layers, we need to decide for the deep neural network we want to apply in the coupling layers.\n", "The input to the layers is an image, and hence we stick with a CNN.\n", "Because the input to a transformation depends on all transformations before,\n", "it is crucial to ensure a good gradient flow through the CNN back to the input,\n", "which can be optimally achieved by a ResNet-like architecture.\n", "Specifically, we use a Gated ResNet that adds a $\\sigma$-gate to the skip connection,\n", "similarly to the input gate in LSTMs.\n", "The details are not necessarily important here, and the network is\n", "strongly inspired from Flow++ [3] in case you are interested in building\n", "even stronger models."]}, {"cell_type": "code", "execution_count": 16, "id": "310d88c0", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:58.171587Z", "iopub.status.busy": "2021-09-16T12:41:58.169353Z", "iopub.status.idle": "2021-09-16T12:41:58.173708Z", "shell.execute_reply": "2021-09-16T12:41:58.173294Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.055251, "end_time": "2021-09-16T12:41:58.173850", "exception": false, "start_time": "2021-09-16T12:41:58.118599", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class ConcatELU(nn.Module):\n", " \"\"\"Activation function that applies ELU in both direction (inverted and plain).\n", "\n", " Allows non-linearity while providing strong gradients for any input (important for final convolution)\n", " \"\"\"\n", "\n", " def forward(self, x):\n", " return torch.cat([F.elu(x), F.elu(-x)], dim=1)\n", "\n", "\n", "class LayerNormChannels(nn.Module):\n", " def __init__(self, c_in):\n", " \"\"\"This module applies layer norm across channels in an image.\n", "\n", " Has been shown to work well with ResNet connections.\n", " Args:\n", " c_in: Number of channels of the input\n", " \"\"\"\n", " super().__init__()\n", " self.layer_norm = nn.LayerNorm(c_in)\n", "\n", " def forward(self, x):\n", " x = x.permute(0, 2, 3, 1)\n", " x = self.layer_norm(x)\n", " x = x.permute(0, 3, 1, 2)\n", " return x\n", "\n", "\n", "class GatedConv(nn.Module):\n", " def __init__(self, c_in, c_hidden):\n", " \"\"\"\n", " This module applies a two-layer convolutional ResNet block with input gate\n", " Args:\n", " c_in: Number of channels of the input\n", " c_hidden: Number of hidden dimensions we want to model (usually similar to c_in)\n", " \"\"\"\n", " super().__init__()\n", " self.net = nn.Sequential(\n", " nn.Conv2d(c_in, c_hidden, kernel_size=3, padding=1),\n", " ConcatELU(),\n", " nn.Conv2d(2 * c_hidden, 2 * c_in, kernel_size=1),\n", " )\n", "\n", " def forward(self, x):\n", " out = self.net(x)\n", " val, gate = out.chunk(2, dim=1)\n", " return x + val * torch.sigmoid(gate)\n", "\n", "\n", "class GatedConvNet(nn.Module):\n", " def __init__(self, c_in, c_hidden=32, c_out=-1, num_layers=3):\n", " \"\"\"Module that summarizes the previous blocks to a full convolutional neural network.\n", "\n", " Args:\n", " c_in: Number of input channels\n", " c_hidden: Number of hidden dimensions to use within the network\n", " c_out: Number of output channels. If -1, 2 times the input channels are used (affine coupling)\n", " num_layers: Number of gated ResNet blocks to apply\n", " \"\"\"\n", " super().__init__()\n", " c_out = c_out if c_out > 0 else 2 * c_in\n", " layers = []\n", " layers += [nn.Conv2d(c_in, c_hidden, kernel_size=3, padding=1)]\n", " for layer_index in range(num_layers):\n", " layers += [GatedConv(c_hidden, c_hidden), LayerNormChannels(c_hidden)]\n", " layers += [ConcatELU(), nn.Conv2d(2 * c_hidden, c_out, kernel_size=3, padding=1)]\n", " self.nn = nn.Sequential(*layers)\n", "\n", " self.nn[-1].weight.data.zero_()\n", " self.nn[-1].bias.data.zero_()\n", "\n", " def forward(self, x):\n", " return self.nn(x)"]}, {"cell_type": "markdown", "id": "8dd3b2c4", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.042657, "end_time": "2021-09-16T12:41:58.260796", "exception": false, "start_time": "2021-09-16T12:41:58.218139", "status": "completed"}, "tags": []}, "source": ["### Training loop\n", "\n", "Finally, we can add Dequantization, Variational Dequantization and Coupling Layers together to build our full normalizing flow on MNIST images.\n", "We apply 8 coupling layers in the main flow, and 4 for variational dequantization if applied.\n", "We apply a checkerboard mask throughout the network as with a single channel (black-white images),\n", "we cannot apply channel mask.\n", "The overall architecture is visualized below.\n", "\n", "\n", "
"]}, {"cell_type": "code", "execution_count": 17, "id": "27d5089b", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:58.351932Z", "iopub.status.busy": "2021-09-16T12:41:58.351458Z", "iopub.status.idle": "2021-09-16T12:41:58.353625Z", "shell.execute_reply": "2021-09-16T12:41:58.353155Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.050041, "end_time": "2021-09-16T12:41:58.353742", "exception": false, "start_time": "2021-09-16T12:41:58.303701", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def create_simple_flow(use_vardeq=True):\n", " flow_layers = []\n", " if use_vardeq:\n", " vardeq_layers = [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=2, c_out=2, c_hidden=16),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " for i in range(4)\n", " ]\n", " flow_layers += [VariationalDequantization(var_flows=vardeq_layers)]\n", " else:\n", " flow_layers += [Dequantization()]\n", "\n", " for i in range(8):\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=1, c_hidden=32),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " ]\n", "\n", " flow_model = ImageFlow(flow_layers).to(device)\n", " return flow_model"]}, {"cell_type": "markdown", "id": "5b491b9b", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.042728, "end_time": "2021-09-16T12:41:58.439262", "exception": false, "start_time": "2021-09-16T12:41:58.396534", "status": "completed"}, "tags": []}, "source": ["For implementing the training loop, we use the framework of PyTorch Lightning and reduce the code overhead.\n", "If interested, you can take a look at the generated tensorboard file,\n", "in particularly the graph to see an overview of flow transformations that are applied.\n", "Note that we again provide pre-trained models (see later on in the notebook)\n", "as normalizing flows are particularly expensive to train.\n", "We have also run validation and testing as this can take some time as well with the added importance sampling."]}, {"cell_type": "code", "execution_count": 18, "id": "0447d535", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:58.532547Z", "iopub.status.busy": "2021-09-16T12:41:58.532063Z", "iopub.status.idle": "2021-09-16T12:41:58.534199Z", "shell.execute_reply": "2021-09-16T12:41:58.533718Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.052256, "end_time": "2021-09-16T12:41:58.534297", "exception": false, "start_time": "2021-09-16T12:41:58.482041", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def train_flow(flow, model_name=\"MNISTFlow\"):\n", " # Create a PyTorch Lightning trainer\n", " trainer = pl.Trainer(\n", " default_root_dir=os.path.join(CHECKPOINT_PATH, model_name),\n", " gpus=1 if torch.cuda.is_available() else 0,\n", " max_epochs=200,\n", " gradient_clip_val=1.0,\n", " callbacks=[\n", " ModelCheckpoint(save_weights_only=True, mode=\"min\", monitor=\"val_bpd\"),\n", " LearningRateMonitor(\"epoch\"),\n", " ],\n", " )\n", " trainer.logger._log_graph = True\n", " trainer.logger._default_hp_metric = None # Optional logging argument that we don't need\n", "\n", " train_data_loader = data.DataLoader(\n", " train_set, batch_size=128, shuffle=True, drop_last=True, pin_memory=True, num_workers=8\n", " )\n", " result = None\n", "\n", " # Check whether pretrained model exists. If yes, load it and skip training\n", " pretrained_filename = os.path.join(CHECKPOINT_PATH, model_name + \".ckpt\")\n", " if os.path.isfile(pretrained_filename):\n", " print(\"Found pretrained model, loading...\")\n", " ckpt = torch.load(pretrained_filename, map_location=device)\n", " flow.load_state_dict(ckpt[\"state_dict\"])\n", " result = ckpt.get(\"result\", None)\n", " else:\n", " print(\"Start training\", model_name)\n", " trainer.fit(flow, train_data_loader, val_loader)\n", "\n", " # Test best model on validation and test set if no result has been found\n", " # Testing can be expensive due to the importance sampling.\n", " if result is None:\n", " val_result = trainer.test(flow, test_dataloaders=val_loader, verbose=False)\n", " start_time = time.time()\n", " test_result = trainer.test(flow, test_dataloaders=test_loader, verbose=False)\n", " duration = time.time() - start_time\n", " result = {\"test\": test_result, \"val\": val_result, \"time\": duration / len(test_loader) / flow.import_samples}\n", "\n", " return flow, result"]}, {"cell_type": "markdown", "id": "27f6d057", "metadata": {"papermill": {"duration": 0.050519, "end_time": "2021-09-16T12:41:58.627997", "exception": false, "start_time": "2021-09-16T12:41:58.577478", "status": "completed"}, "tags": []}, "source": ["## Multi-scale architecture\n", "\n", "
\n", "\n", "One disadvantage of normalizing flows is that they operate on the exact same dimensions as the input.\n", "If the input is high-dimensional, so is the latent space, which requires larger computational cost to learn suitable transformations.\n", "However, particularly in the image domain, many pixels contain less information in the sense\n", "that we could remove them without loosing the semantical information of the image.\n", "\n", "Based on this intuition, deep normalizing flows on images commonly apply a multi-scale architecture [1].\n", "After the first $N$ flow transformations, we split off half of the latent dimensions and directly evaluate them on the prior.\n", "The other half is run through $N$ more flow transformations, and depending on the size of the input,\n", "we split it again in half or stop overall at this position.\n", "The two operations involved in this setup is `Squeeze` and `Split` which\n", "we will review more closely and implement below."]}, {"cell_type": "markdown", "id": "207a4bd1", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.043555, "end_time": "2021-09-16T12:41:58.714567", "exception": false, "start_time": "2021-09-16T12:41:58.671012", "status": "completed"}, "tags": []}, "source": ["### Squeeze and Split\n", "\n", "When we want to remove half of the pixels in an image, we have the problem of deciding which variables to cut,\n", "and how to rearrange the image.\n", "Thus, the squeezing operation is commonly used before split, which divides the image into subsquares\n", "of shape $2\\times 2\\times C$, and reshapes them into $1\\times 1\\times 4C$ blocks.\n", "Effectively, we reduce the height and width of the image by a factor of 2 while scaling the number of channels by 4.\n", "Afterwards, we can perform the split operation over channels without the need of rearranging the pixels.\n", "The smaller scale also makes the overall architecture more efficient.\n", "Visually, the squeeze operation should transform the input as follows:\n", "\n", "
\n", "\n", "The input of $4\\times 4\\times 1$ is scaled to $2\\times 2\\times 4$ following\n", "the idea of grouping the pixels in $2\\times 2\\times 1$ subsquares.\n", "Next, let's try to implement this layer:"]}, {"cell_type": "code", "execution_count": 19, "id": "3435b3ac", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:58.805945Z", "iopub.status.busy": "2021-09-16T12:41:58.805454Z", "iopub.status.idle": "2021-09-16T12:41:58.807515Z", "shell.execute_reply": "2021-09-16T12:41:58.807040Z"}, "papermill": {"duration": 0.050124, "end_time": "2021-09-16T12:41:58.807614", "exception": false, "start_time": "2021-09-16T12:41:58.757490", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class SqueezeFlow(nn.Module):\n", " def forward(self, z, ldj, reverse=False):\n", " B, C, H, W = z.shape\n", " if not reverse:\n", " # Forward direction: H x W x C => H/2 x W/2 x 4C\n", " z = z.reshape(B, C, H // 2, 2, W // 2, 2)\n", " z = z.permute(0, 1, 3, 5, 2, 4)\n", " z = z.reshape(B, 4 * C, H // 2, W // 2)\n", " else:\n", " # Reverse direction: H/2 x W/2 x 4C => H x W x C\n", " z = z.reshape(B, C // 4, 2, 2, H, W)\n", " z = z.permute(0, 1, 4, 2, 5, 3)\n", " z = z.reshape(B, C // 4, H * 2, W * 2)\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "99659504", "metadata": {"papermill": {"duration": 0.04349, "end_time": "2021-09-16T12:41:58.894090", "exception": false, "start_time": "2021-09-16T12:41:58.850600", "status": "completed"}, "tags": []}, "source": ["Before moving on, we can verify our implementation by comparing our output with the example figure above:"]}, {"cell_type": "code", "execution_count": 20, "id": "f0ffca0f", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:58.984627Z", "iopub.status.busy": "2021-09-16T12:41:58.984154Z", "iopub.status.idle": "2021-09-16T12:41:58.988910Z", "shell.execute_reply": "2021-09-16T12:41:58.988509Z"}, "papermill": {"duration": 0.051435, "end_time": "2021-09-16T12:41:58.989009", "exception": false, "start_time": "2021-09-16T12:41:58.937574", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Image (before)\n", " tensor([[[[ 1, 2, 3, 4],\n", " [ 5, 6, 7, 8],\n", " [ 9, 10, 11, 12],\n", " [13, 14, 15, 16]]]])\n", "\n", "Image (forward)\n", " tensor([[[[ 1, 2, 5, 6],\n", " [ 3, 4, 7, 8]],\n", "\n", " [[ 9, 10, 13, 14],\n", " [11, 12, 15, 16]]]])\n", "\n", "Image (reverse)\n", " tensor([[[[ 1, 2, 3, 4],\n", " [ 5, 6, 7, 8],\n", " [ 9, 10, 11, 12],\n", " [13, 14, 15, 16]]]])\n"]}], "source": ["sq_flow = SqueezeFlow()\n", "rand_img = torch.arange(1, 17).view(1, 1, 4, 4)\n", "print(\"Image (before)\\n\", rand_img)\n", "forward_img, _ = sq_flow(rand_img, ldj=None, reverse=False)\n", "print(\"\\nImage (forward)\\n\", forward_img.permute(0, 2, 3, 1)) # Permute for readability\n", "reconst_img, _ = sq_flow(forward_img, ldj=None, reverse=True)\n", "print(\"\\nImage (reverse)\\n\", reconst_img)"]}, {"cell_type": "markdown", "id": "912fb9c0", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.04387, "end_time": "2021-09-16T12:41:59.077196", "exception": false, "start_time": "2021-09-16T12:41:59.033326", "status": "completed"}, "tags": []}, "source": ["The split operation divides the input into two parts, and evaluates one part directly on the prior.\n", "So that our flow operation fits to the implementation of the previous layers,\n", "we will return the prior probability of the first part as the log determinant jacobian of the layer.\n", "It has the same effect as if we would combine all variable splits at the\n", "end of the flow, and evaluate them together on the prior."]}, {"cell_type": "code", "execution_count": 21, "id": "420c7e21", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:59.170054Z", "iopub.status.busy": "2021-09-16T12:41:59.169563Z", "iopub.status.idle": "2021-09-16T12:41:59.171697Z", "shell.execute_reply": "2021-09-16T12:41:59.171229Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.05122, "end_time": "2021-09-16T12:41:59.171798", "exception": false, "start_time": "2021-09-16T12:41:59.120578", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class SplitFlow(nn.Module):\n", " def __init__(self):\n", " super().__init__()\n", " self.prior = torch.distributions.normal.Normal(loc=0.0, scale=1.0)\n", "\n", " def forward(self, z, ldj, reverse=False):\n", " if not reverse:\n", " z, z_split = z.chunk(2, dim=1)\n", " ldj += self.prior.log_prob(z_split).sum(dim=[1, 2, 3])\n", " else:\n", " z_split = self.prior.sample(sample_shape=z.shape).to(device)\n", " z = torch.cat([z, z_split], dim=1)\n", " ldj -= self.prior.log_prob(z_split).sum(dim=[1, 2, 3])\n", " return z, ldj"]}, {"cell_type": "markdown", "id": "5d191c06", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.043739, "end_time": "2021-09-16T12:41:59.259120", "exception": false, "start_time": "2021-09-16T12:41:59.215381", "status": "completed"}, "tags": []}, "source": ["### Building a multi-scale flow\n", "\n", "After defining the squeeze and split operation, we are finally able to build our own multi-scale flow.\n", "Deep normalizing flows such as Glow and Flow++ [2,3] often apply a split operation directly after squeezing.\n", "However, with shallow flows, we need to be more thoughtful about where to place the split operation as we need at least a minimum amount of transformations on each variable.\n", "Our setup is inspired by the original RealNVP architecture [1] which is shallower than other,\n", "more recent state-of-the-art architectures.\n", "\n", "Hence, for the MNIST dataset, we will apply the first squeeze operation after two coupling layers, but don't apply a split operation yet.\n", "Because we have only used two coupling layers and each the variable has been only transformed once, a split operation would be too early.\n", "We apply two more coupling layers before finally applying a split flow and squeeze again.\n", "The last four coupling layers operate on a scale of $7\\times 7\\times 8$.\n", "The full flow architecture is shown below.\n", "\n", "
\n", "\n", "Note that while the feature maps inside the coupling layers reduce with the height and width of the input,\n", "the increased number of channels is not directly considered.\n", "To counteract this, we increase the hidden dimensions for the coupling layers on the squeezed input.\n", "The dimensions are often scaled by 2 as this approximately increases the computation cost by 4 canceling with the squeezing operation.\n", "However, we will choose the hidden dimensionalities $32, 48, 64$ for the\n", "three scales respectively to keep the number of parameters reasonable\n", "and show the efficiency of multi-scale architectures."]}, {"cell_type": "code", "execution_count": 22, "id": "f1aa3acb", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:59.353192Z", "iopub.status.busy": "2021-09-16T12:41:59.352715Z", "iopub.status.idle": "2021-09-16T12:41:59.354799Z", "shell.execute_reply": "2021-09-16T12:41:59.354387Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.052212, "end_time": "2021-09-16T12:41:59.354899", "exception": false, "start_time": "2021-09-16T12:41:59.302687", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def create_multiscale_flow():\n", " flow_layers = []\n", "\n", " vardeq_layers = [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=2, c_out=2, c_hidden=16),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " for i in range(4)\n", " ]\n", " flow_layers += [VariationalDequantization(vardeq_layers)]\n", "\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=1, c_hidden=32),\n", " mask=create_checkerboard_mask(h=28, w=28, invert=(i % 2 == 1)),\n", " c_in=1,\n", " )\n", " for i in range(2)\n", " ]\n", " flow_layers += [SqueezeFlow()]\n", " for i in range(2):\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=4, c_hidden=48), mask=create_channel_mask(c_in=4, invert=(i % 2 == 1)), c_in=4\n", " )\n", " ]\n", " flow_layers += [SplitFlow(), SqueezeFlow()]\n", " for i in range(4):\n", " flow_layers += [\n", " CouplingLayer(\n", " network=GatedConvNet(c_in=8, c_hidden=64), mask=create_channel_mask(c_in=8, invert=(i % 2 == 1)), c_in=8\n", " )\n", " ]\n", "\n", " flow_model = ImageFlow(flow_layers).to(device)\n", " return flow_model"]}, {"cell_type": "markdown", "id": "ca021fca", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.043766, "end_time": "2021-09-16T12:41:59.442691", "exception": false, "start_time": "2021-09-16T12:41:59.398925", "status": "completed"}, "tags": []}, "source": ["We can show the difference in number of parameters below:"]}, {"cell_type": "code", "execution_count": 23, "id": "c13d7d7c", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:41:59.534720Z", "iopub.status.busy": "2021-09-16T12:41:59.534247Z", "iopub.status.idle": "2021-09-16T12:42:02.273185Z", "shell.execute_reply": "2021-09-16T12:42:02.272760Z"}, "papermill": {"duration": 2.786959, "end_time": "2021-09-16T12:42:02.273302", "exception": false, "start_time": "2021-09-16T12:41:59.486343", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["Number of parameters: 335,128\n", "Number of parameters: 379,556\n", "Number of parameters: 1,062,090\n"]}], "source": ["def print_num_params(model):\n", " num_params = sum(np.prod(p.shape) for p in model.parameters())\n", " print(f\"Number of parameters: {num_params:,}\")\n", "\n", "\n", "print_num_params(create_simple_flow(use_vardeq=False))\n", "print_num_params(create_simple_flow(use_vardeq=True))\n", "print_num_params(create_multiscale_flow())"]}, {"cell_type": "markdown", "id": "949772c8", "metadata": {"papermill": {"duration": 0.04404, "end_time": "2021-09-16T12:42:02.361991", "exception": false, "start_time": "2021-09-16T12:42:02.317951", "status": "completed"}, "tags": []}, "source": ["Although the multi-scale flow has almost 3 times the parameters of the single scale flow,\n", "it is not necessarily more computationally expensive than its counterpart.\n", "We will compare the runtime in the following experiments as well."]}, {"cell_type": "markdown", "id": "a40ce763", "metadata": {"papermill": {"duration": 0.043937, "end_time": "2021-09-16T12:42:02.450029", "exception": false, "start_time": "2021-09-16T12:42:02.406092", "status": "completed"}, "tags": []}, "source": ["## Analysing the flows\n", "\n", "In the last part of the notebook, we will train all the models we have implemented above,\n", "and try to analyze the effect of the multi-scale architecture and variational dequantization.\n", "\n", "### Training flow variants\n", "\n", "Before we can analyse the flow models, we need to train them first.\n", "We provide pre-trained models that contain the validation and test performance, and run-time information.\n", "As flow models are computationally expensive, we advice you to rely on\n", "those pretrained models for a first run through the notebook."]}, {"cell_type": "code", "execution_count": 24, "id": "8b210a38", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:02.543059Z", "iopub.status.busy": "2021-09-16T12:42:02.542580Z", "iopub.status.idle": "2021-09-16T12:42:02.794090Z", "shell.execute_reply": "2021-09-16T12:42:02.793670Z"}, "papermill": {"duration": 0.300432, "end_time": "2021-09-16T12:42:02.794206", "exception": false, "start_time": "2021-09-16T12:42:02.493774", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True, used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Found pretrained model, loading...\n", "Found pretrained model, loading...\n", "Found pretrained model, loading...\n"]}], "source": ["flow_dict = {\"simple\": {}, \"vardeq\": {}, \"multiscale\": {}}\n", "flow_dict[\"simple\"][\"model\"], flow_dict[\"simple\"][\"result\"] = train_flow(\n", " create_simple_flow(use_vardeq=False), model_name=\"MNISTFlow_simple\"\n", ")\n", "flow_dict[\"vardeq\"][\"model\"], flow_dict[\"vardeq\"][\"result\"] = train_flow(\n", " create_simple_flow(use_vardeq=True), model_name=\"MNISTFlow_vardeq\"\n", ")\n", "flow_dict[\"multiscale\"][\"model\"], flow_dict[\"multiscale\"][\"result\"] = train_flow(\n", " create_multiscale_flow(), model_name=\"MNISTFlow_multiscale\"\n", ")"]}, {"cell_type": "markdown", "id": "0f554c5e", "metadata": {"papermill": {"duration": 0.046682, "end_time": "2021-09-16T12:42:02.888035", "exception": false, "start_time": "2021-09-16T12:42:02.841353", "status": "completed"}, "tags": []}, "source": ["### Density modeling and sampling\n", "\n", "Firstly, we can compare the models on their quantitative results.\n", "The following table shows all important statistics.\n", "The inference time specifies the time needed to determine the\n", "probability for a batch of 64 images for each model, and the sampling\n", "time the duration it took to sample a batch of 64 images."]}, {"cell_type": "code", "execution_count": 25, "id": "1a7fa75f", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:02.984361Z", "iopub.status.busy": "2021-09-16T12:42:02.983766Z", "iopub.status.idle": "2021-09-16T12:42:02.986662Z", "shell.execute_reply": "2021-09-16T12:42:02.987047Z"}, "papermill": {"duration": 0.052662, "end_time": "2021-09-16T12:42:02.987161", "exception": false, "start_time": "2021-09-16T12:42:02.934499", "status": "completed"}, "tags": []}, "outputs": [{"data": {"text/html": ["\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["%%html\n", "\n", ""]}, {"cell_type": "code", "execution_count": 26, "id": "30428798", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:03.098132Z", "iopub.status.busy": "2021-09-16T12:42:03.092877Z", "iopub.status.idle": "2021-09-16T12:42:03.101398Z", "shell.execute_reply": "2021-09-16T12:42:03.100990Z"}, "papermill": {"duration": 0.067197, "end_time": "2021-09-16T12:42:03.101499", "exception": false, "start_time": "2021-09-16T12:42:03.034302", "status": "completed"}, "tags": []}, "outputs": [{"data": {"text/html": ["
\n", "\n", "
Model
Validation Bpd
Test Bpd
Inference time
Sampling time
Num Parameters
\n", "\n", "\n", "
simple
1.109 bpd
1.107 bpd
51 ms
50 ms
335,128
\n", "
vardeq
1.068 bpd
1.066 bpd
69 ms
50 ms
379,556
\n", "
multiscale
1.029 bpd
1.026 bpd
40 ms
22 ms
1,062,090
\n", "\n", "
"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["\n", "table = [\n", " [\n", " key,\n", " \"%4.3f bpd\" % flow_dict[key][\"result\"][\"val\"][0][\"test_bpd\"],\n", " \"%4.3f bpd\" % flow_dict[key][\"result\"][\"test\"][0][\"test_bpd\"],\n", " \"%2.0f ms\" % (1000 * flow_dict[key][\"result\"][\"time\"]),\n", " \"%2.0f ms\" % (1000 * flow_dict[key][\"result\"].get(\"samp_time\", 0)),\n", " \"{:,}\".format(sum(np.prod(p.shape) for p in flow_dict[key][\"model\"].parameters())),\n", " ]\n", " for key in flow_dict\n", "]\n", "display(\n", " HTML(\n", " tabulate.tabulate(\n", " table,\n", " tablefmt=\"html\",\n", " headers=[\"Model\", \"Validation Bpd\", \"Test Bpd\", \"Inference time\", \"Sampling time\", \"Num Parameters\"],\n", " )\n", " )\n", ")"]}, {"cell_type": "markdown", "id": "0c8899de", "metadata": {"papermill": {"duration": 0.047332, "end_time": "2021-09-16T12:42:03.196088", "exception": false, "start_time": "2021-09-16T12:42:03.148756", "status": "completed"}, "tags": []}, "source": ["As we have intially expected, using variational dequantization improves upon standard dequantization in terms of bits per dimension.\n", "Although the difference with 0.04bpd doesn't seem impressive first, it is a considerably step for generative models\n", "(most state-of-the-art models improve upon previous models in a range of 0.02-0.1bpd on CIFAR with three times as high bpd).\n", "While it takes longer to evaluate the probability of an image due to the variational dequantization,\n", "which also leads to a longer training time, it does not have an effect on the sampling time.\n", "This is because inverting variational dequantization is the same as dequantization: finding the next lower integer.\n", "\n", "When we compare the two models to multi-scale architecture, we can see that the bits per dimension score again dropped by about 0.04bpd.\n", "Additionally, the inference time and sampling time improved notably despite having more parameters.\n", "Thus, we see that the multi-scale flow is not only stronger for density modeling, but also more efficient.\n", "\n", "Next, we can test the sampling quality of the models.\n", "We should note that the samples for variational dequantization and standard dequantization are very similar,\n", "and hence we visualize here only the ones for variational dequantization and the multi-scale model.\n", "However, feel free to also test out the `\"simple\"` model.\n", "The seeds are set to obtain reproducable generations and are not cherry picked."]}, {"cell_type": "code", "execution_count": 27, "id": "ab20301b", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:03.294176Z", "iopub.status.busy": "2021-09-16T12:42:03.293684Z", "iopub.status.idle": "2021-09-16T12:42:03.408730Z", "shell.execute_reply": "2021-09-16T12:42:03.408245Z"}, "papermill": {"duration": 0.165353, "end_time": "2021-09-16T12:42:03.408844", "exception": false, "start_time": "2021-09-16T12:42:03.243491", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 44\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDM0MC41NiAzNDAuNTYgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UKPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicPY1NCoMwEIX3c4p3giSTtHGtCMGl3XiAEFrFH6xQr99JaVw83scwj48xka4ZzwMGk+QEI0C36TPG9AgN4kFG7gu5m1F3LzgX/NcsDxe+iFbaUSn7i7NesS/1ThiwQtc2C1mELEKDIAtnK2RR7rKOC3THaDf01NMXrxEktQplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjEyNgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjUzICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDMyNyAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMzI3IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMzI3ID4+CnN0cmVhbQp4nO2deVgURxrG5+K+A4iKRhQXBY0Ka4ga4hVFlF2jiKLx2KiEeC0RV6MQTRTxiATdeC8mxogHEuNqTIhnEIQoasQrBxKPCCKHCIqGc55nP3qcMNP0UT0UMLrf+4cyfbz99m9mqqumq6tkH6BoSNbSAZ4TIUc6Qo50hBzpSMNRTU8fNIkhbT/qhsiRjiFypGOIHOkYIkc6hsiRjiFypGOIHOkYIketzp0717dvX5lM8tGQo56QowF+zwvHoqIimUyhUERERJSXExjSSqfmC1hTEx4erlQqVaqgoCCOvQICAvbuFTBEjhohR6l6fjguWLDg1VdfVSoVjDw8Ll26JGbY5AHz8jp16gQczcwmTJjAXrl+/XonJ6cPP6ytrc3Ozq6p4TBEjho9Yxxzc3MXL7a0tITETk4+Pj62trZyeceOHU+cOKHdpKBg9uzZ1dXVeoZNHjAlxd7eHt7VgACOXfz8/CBjba2aKZCY/9mGyFEj5EgnYLNzrKioqGU7iccE1dauWLHC1dW17pqo6tGjR04OLLx69Wp4uEwmCwkJefgwPz//008/nTVr8+bNbEMJAcXEFzA0NFQuly9cWKNX/v3xx4YNGxwcHIKDgwUNkaNWyFGajIJjTU3//v3hvMeNI415586dL7/8ctiwYRUVUPwo6+Tu7n7z5k29vRITE5lVYK1Urlq1im1IHFBcfBynTJni6+s7enRKSorO0sGDAa7gBwc56qtZOarV8OVbtGiRoyNpTIjBEFLK5fB9BpQ9ely7dk1vlzlz5lhbWzOQYesPPqisrGQbSggoJuEKxZ49JiYm3t7e+/ZBFaKsrKxVKzMzMwJD5Kgv5EiqluAIhdumTZuKi4sNjwmxlE8FHOHfbt30dygsNDU1Vda1tgFifHw8pyH58SUHZOv06dNRUVFyuZWV1fTp019+GYpsAkPkyBZyNCwgW/Q53rgxYcIEOMOZM2caHrN79+6a38eUSqaN2qAGsXKlZr2JSUxMDMf65uaoZtoazI9o8qf6+eefS0tLU1MjIyOhARESkpeXxzZEjhxCjgYE5BBljhkZ5ubmYLdmzRrDY65fv37fvn0WFhbW1q1btz569Kje1j/88IO5OUCEonPePAFD8uNLDsits2czMjI8PT1NTCBdQUEBvMFz52oaZArF9evX2YbIkVvUON64cWPaNM1FNjk5ubExIUZ5eVxc3Lp16/SW9+vXj8kWHR0taEh+fAMD8qmoCNK5urr+5z//sbTUFlDLli1jGyJHYSFHaQH51DiOcMHq2rUrU20GjvDnnDljxowhKicFYlZVVV24cEH76vbtefPmMU3u9957T8xQ/LjEksaxrGz48OEvvfTS+++/X3crSQ6t7a+/1q9VIEcCIUcDAnKoMRzPnBk5ciRTEPTq1Ss0NBQu//b2tra2Li4uK1ZAk5vZ6NixY9AYZjeWxWJCDaK8/PDhw1CtPXFCwTSp27X79ddfxc6b9LQJJI2jWl1eXp6Wlvbaa6+1agX/X7x4Ue9XPTVyJJSBHB88ePD3v5uamgK0YcOys7N1VpWUlMhkgYGB+/fvnzULuNrZ2c2ebUBMgAjfjDNnNI2ur74SPo+W5lhdXe3t7Q1BP/tMwBA5igk5Cvg1Pcfdu3erVLDXoEGDOPZJSQkPD4e1lpbwD1zDGhGzoqJ9+/bgkpb25MkTaIKPGTNx4sQvv/yS01DAJy8v75dffv/9d4JDSgvIqLwcrgKdO3eGugXfzVfkSCDkyKdm4ghydISad1FREZ93YmJiSgrUywFCo2LCUcAiMnLcuHHwv4kJ8yKS05DboLh4zpw5NjY2/v5QqhMdUlpAaEePHGltbb1u3bqHDx8KGiJHATWS49atXl5eYseAyjOnt4SYZ8+eNTExUanatGmzZMmSx4/hIs7uIqAW5BgXBxD79OlDdDSpAa9fvw6fFaZtbGFhUcPuFc4yRI58Qo7Caj6OBw785S9/kZTNgJhqpq+1s7Ozvb1ARjU/x8WLF9vYWFpaQkO/SQJWVFS8+OKLqjpBwS3QFwI5Cgo50gnYeI4PH2ZlZXHvw7Q1R44cWVXV2Jig5cuXM/34hLfi9Lty5YqTk5O5+fjx43Nzc4mOZkBAsE5OhqoYlJNihshRQI3kyCe4nvr4wJkPHDiQQsyqKmgHwsVw7lzh7Tj8YmKgcgxfNicnxof3PW1cQEmGyLHRQo50JJkjyNLSwcHhwIEDIq4khrduafqVzppFEFPndXp6upWVpqdNWFhlZaX+U0viQo4aIUd9/V9z3LtX+Ka9hJh37mg4fv45QUzti4SEBOapYzs7u5iYmICA7t27m5qa3r5Nkl1qQEmGyLHRMoQjoSvZfYXY2NgRI0acP09gqH0BVQULC4vlyzXPJ1ZWQpMA3g3kiBwFXZEjHVfqhrT9kCMdP+RIxw850vFDjnT8mogjqrFCjnSEHOkIOdIRcqQj5EhHWO+hY4gc6Ri2HMfiYk9PT9FBhpCjmJCjuCtypONKYpiRYWFhcV7sV1wRjsXFxUOHDlUoPvnkE+oBJRkix0YLOdJRC3N8800vLy/2eBV8htyrzp/38/OTy+WtW6enp1MPKMkQOTZazczxwIEhQ4ZYWVmNHx8XF1dUVNSly5kzZxQKxerVf/zxh5ghx/Ls7OwRIwDiwIED79+nEdBQIUc6Qo50JIVjbe2FCxe6dOnSrx+RawPDnJwcW1vNaJoKRZs2bZj5EyorK48dO1ZQQGDIXvjkyRNTU1O5PCkpiW8/vk7dyFFHyPH/mOPx48cHDYKWsKWlpfhISNwxL126xHTSg1VJScHBwcIjxDc0ZC+E/WNjY9PS+HbauHFj//79JQTUupKmYhsiR31X0lRsQ+So70qaim0ozDEvLw8SQeWiZ8/Jkyd///33hK4NDBMTE1WqlStXGhBXanFWUVGxaJG1tfXdu3clBGQGToW9tm/PyMjgNq6sbNu2bVrD9w456qnpONbWtmrVasCAAXD+5M/6cMeEk8vOlvrEkI6h2Eb379+PjIwcPbpfv35QLzc3FxgyjY/jF198AR8YZqheaDD8/HODDWpqZDLZjRs3OA2Ro1bIUW3kHJnpswTGnyGNSTY/Q04Od82ZiOMnn3yiHTSdeUSpkj24oGBARlCejmAa7Izmz2+wwePHHTt25DBGjnpCjmoj55iYOG/evL180xWTx8xhpthTq7OysqBy3HAEM43S04GAdlO2odhR4+LimNFNX3zxRY6ZbMUCalVSUvLLL6tWrbK1tbWwKCwsFDuqjiFy1FETcdy1C7Jt2LCBzE4s5rZtbm5uYOjiwj1Pc0lJ7969W7du/eQJh6HYUcEyOjr6+HGCcc4EOGr19ttvm5omJCRcuXJFzE2NHPmFHI2So1ptbm4O1dpz584R2AnGhKKxfXvNc8JmZsePH+fcFUpj4Mx+SJ6IowQRcNy+fTvzkzM0Q06fJjNEjhxCjgYH1BN1josXL4aTh5Zno2JCZcbT01OpZAYpVJmZ8XWiuHfvnoWFRWAghyH58Q0JqKdbt25BQa6dK/DpT32ihsiRrabg6OfnB9+0iIiI/PxGxGRuKqiYqUMA5YcfCux97do19i8tzc0xOTlZWa8dO8gMkSNbyNHggHpqCo7MdMp12rixETFTUlIcHByYkYt8fX1JjfQMpe4k5idgmJSUxPRWCAoK+vjjjwnG00aOnEKOEvyamaOmjJDL5ebm77zzjqExy8vLfXx8zM3DwsIkjDuvZyh1JzE/AcMjR47AB8fXV3t3GFohTG8kYUPkyFZTcFy6dOnmzZsnT57s7AzeXl5eDYdJJ4tZWlqamdmIbh8G7CfoJ2DYt29fOFftr83V1S4uLtOmTRMzRI5sIUcJfs3MUetVvWaNTCaDSnTDu+BSY3JLbFoSsYASDiUc8OZNgOjo6KidKbymxsbGpmvXrmKGyFFfyJFULcfx0aNHM2dC5Wd2w1n2SGPeuXNn376tW7fGxsY23OWtt96C2lX37gKGOq+hlH3lFZ3Xhw8fdnGZMGHCnj17+G/+iwZkVFZW1rEjcNTpkHDsGHyApk+fLmaIHHXUxBzT09NNTe3s7IgmZeOImZqayrS0NJ1GZLJJkybBye/aBeXFwYMHIyOZlYqdOwUM9RfNmKFz6S8pKWGanGBB1P9FgCNYMZN7vP7669q7lu7uHh4eFRUVYobIUUfI8ZngePXq1Rde6NWrF0FIzphQ7Hh6ejI/HTETk2ta7TY2gwcPZtDCC2tr69RUAUP9Re3bBwYGnjp1qqrq0KFDbdq0YTp/Qm3i0KHLly8bEFCrysrK4mLwk9W918eOHWMmBFy4cCGBIXLUEXJ8JjhKEndMKGKzshTM+TIcFU/F/Ons7Lx//35BQ/1FPj7A38zMbOpUKGf9/f21b41C0adPn4SEhG+/hbfut99+kxCwXl9//TW4P/VTREaK/iqAHDll1BwZQflw/PjxtWujo6NnzJgxfvzatWuhfn716g8//CBmqL/o7Fmoum/btu3MGe3rzMxM+EovWODg4MB0S96yZQtfZzExjlVVVTt37ty4ceLEiUePHhXskqpjiBzZQo7PFEfhSRsN/b1HzJB049LS0kePHuXnG/C8gsFCjnSEHOmo2cvHRhjS9kOOdPyQIx0/5EjHDznS8UOOdPyQIx0/5EjHr4k4ohor5EhHyJGOkCMdIUc6wus1HUPkSMcQOdIxRI50DJEjHUPkSMcQOdIxRI50DIU5VjOS7trCHPPzQ0NDvby8OAfTRI7EQo58htJ2MTqOGRkZERERW7ZsCQnZvn0731YFulMktCzHnJyczMxMba/Ajz7i80OOwkKOwjFJNzZSjra2ttpMJiYmnEXlmDFj3njjDf6Yt2/fzs3NLS0tNeihYTU5x9ra2r59VSoVHK1Xr6lTp0LmJUv4/JAjr4yXY3S0BmGPHj127ODrwf3vf/8bwtfPSaAfMz1d8ySBhYUF37i4evr9998HDNDv2S3htE+f3vn0aab8/HxTU9MZMzg2Qo6iQo5aPX8cobixt9eMSS44jnpYWBigPnqUO2ZWFrwJmkHSly8XC/Tjjz+6u7srlVCJZsckO6N6wUVbqVT6+3OsksaxuPjgwYOaP0+dCgoK0rsW6BgiR2EhRyPhCHrlFWaYapdbt26p1VeuXDlw4MDWreyNevbsCYVf/dxD7Jg1NTXZ2dnt27dPSRF+MurBg6FDhzKPUUVHR7NjCp9sQ61bt04mk40ezbFKGsfMzEmTJmn+bNcOyvrg4GBOQ+QoLBocd+5kHk9V2dnZ2dg4OTm1bdt2wICSkhKd73lWlrm5+XLdryxHzKqqKngr6iECWvjenTwZEhJy+PDh0lJYlJaWFhCgqR24uemP024Qx7Fjx4IVZ1EijeO0aX8OTuHoCBDLyso4DZGjsJCj8XBUqz09PaE6C63s1q0dHR1HjhxpYQHxAGldian+5ptvvLz8/Pykxty8ebOzs/OYMVC0dujQQan09vb28PBgIMLRaI2Pu2/fvpISjuXSOI4bl6cdEonv0SbkSCDkaEwc/1RFRW5uLpRr33wzfvx4ONmTJ2Fhly5dFIr33ntPWszycihywYJ5rLe6unrsWHgrmHEqoKg9deoUZ0zRgOSSxjEmRni2eDVyJBJljvW6efOmv7+/XA6VZoChVEZFRUmLWVysqQXUz8QVHx/PDF0RERHBF1NCQDFJ40hqiBzpGCJHOoYG9u9JSFAw45UFBJSwLoqiMWNjgSO8BfWV+e3btzMjvbz++ut8MSUH5BdypCPkSEdGxbGsLDs7OzAw8MIFTldew8LCwqFD4S1488036xdCvZyphDMjRE2dP//Ro0dsQ8kB+YUc6cjIOP71r3/ld+UyrKrKysqCJiYzdthXX31Vv2b69Omurq6nTh06dOj9999XKMLDw9mGkgPyCznSEXKkI6PieO9e586d+V0bGJ44cWLmTO3ggv/4xz/0btWcPXtW24itqKjo2hWKSrah5ID8Qo50hBzpyKg4FhYOHz48Ly+vYU8+jph3795lBrHVVN0/+ihfcBa/ggL9cQxbiiO81aGhoYIjXusYIkc+UeZYVFSUlJTUcHlVVRUcydX1V9ZDAeyYUF9/6aWXmG90v379rKysOCcRF4spbRdxP1HDYcPKyspWrFhB8LQGchQQciTya36OBw8ehHKN814ZcDQ3j4qK0rv4smNGRERo7u+PGvX48eNJkyZpb7+RqmU4duiQnJxMNqEYchQQciTya36OixYtiouL41tbXHz48GG9IXPZMaGSDZUdb2/v69cjIyPNzMzefZcgGzumtF2Yqd1HjRrFuR8Zx4gIU1PTzLq5AskCIkdu0eUIq+AbaWtrO2XKwoULoT3n7Z2QkCDuqmNYUlIC35CLFy9evers7AxX7Vdeyam/VUgggziuX79epVLFxBAE5NatWyYmJkOGDLl/nywgcuQWcmRkVBwvX7a0tLSwsAC7du1kMhk0N999F0oecVcuw48+0syTolR+9913ouFYhhK2ZwQcoVQeNUpKQJbg0wNZCwvJAiJHPiFHtZFx3LYNypiJEyeq66bUBH5PtJPRibpyGe7fDzUJLy+vS5c4nj+urKw8eZJ76gaDOELrH94tprcrcUCWVq5cCRyLikQ3RI6CQo5qI+NYV0JeFphRmd+V2zAvL+/OnTucq3bs2HHxYkpKCp8h6dFrGRkaUE9QxEKBrjv0hpAhcuQTTY4GiDSmvmJjY3v2dHFx4TOkEEzPz1juF4q4Ikc6rsiRjqthhtwlmxEFFDBEjnQMkSMdQ2PiKGBI2w850vFrIo6oxgo50hFypCPkSEfIkY6QIx1hvYeOIXKkY4gc6RgiRzqGyJGOIXKkY4gc6RgiRzqGyJGOIXKkY4gc6RgiRzqGyJGO4TPJsba2Y8eOsbGxevMwqOt6rWVmZv7tb3+LikpOTuYYybYJAyJHOgGRI52ALcSR7Mm9ekP2Qqbf6UmVaurUqeq6UXBCQ0Pj4uJmzdIMOcA8z/jCCy/ojSsgLSATUb8v/Lffck80gRzFIhopR/impaam3r1bU1MD5/nyyxIMOZZ36tRJJoMvt5+fX+vWSqUSuMXEANcRI0b4+9szc5VYWgYHB5eXl5MG1FNYWJirq6v2FSD09XV3d+cLiBz5hByfb45HjqhUKjMzsx07fvrpJzhJmWzLli2khhzLly9fLpNpCkNPz/nz5584caJ+5bVr1zw9PZly0sbG5pbugwtkHJOTrays+vbtq30NtQBHx08//ZQvIHLkFnJk9Fxy/O9//wvn6eSkeb6QGcxeVodg48aNpIYGBMnPz1+1SjM0ht7YOGQcx4yBt8jJyWnZsrfffjsoKAg+AwEBAgGRI7eMkaOvr6+yXvb2cBHdu3evQjFgwIAVK1YEBW3atAkqvY8eVVdXc9TPGxHwu+80s5PcvHlTMCCHwsKY52VHHT+umeO2T58+N25wb4ocBYQcdfSccVy4cKGqTpp5FbTzMchkUVFRq1evHjBgypQpcGVs2/a1117r1atXbi6HoWFZmCm/1qxZIxaQQ6dPOzs7M3/BdR+QJicn822KHAWEHHVkvByLiqBZGxERwZ5ESywmU+7du3fvs88+060u/6m7d+/Gx4eGhnp4eMTHcxgSB9TTv/7l5ua2efNmgoANA589ezYnJ6dVK4AYGRkpsClyFFATc5wxA74pJiYma9cKuhrN/cLKykr2KA6kAaFW0aFDB4XCzs7ukuDYn8hRUMhRR0bJ8cGDB25uAPHzzz/nv71iXBz5/EQN09NtbGyYxnlYWBiBIXLkFnIk8mtRjjU1NVOmuLu7p6Wl1S88deqUg4NDfXlJEDM7O7tbt26cgx7xxSTblNxP2LCoqMjRUfMrn4fHz/x1PB1D5MghShxra7lHyAI9ecIenGvixIlwxF69iGPW1rZp0wYqtx9/rDnKvXtHjx4dO3asTBYdHU339x5OEXD85z//qZ39+OjRP5fyMUGOfEKOBGo+jhkZMpls7969v/xCEGv69OlwRHt7uJiTxdy5U3MTb+PG3377bdmyZb17a34Kksuhca11YcckCKKn8PBwZ2fnkBCOVWIBz507J5drOwkpnv7oU6eBA7knfEGOnEKOOjIKjt9/HxISAmf6zjsEibt3784csPDpSLLCMQ8dOuTsDFvDJrt3L1261NzcvG7GrrqSaPbs3bt388UkCKInX19faDBw/vYqxnH48OE6HM3Mzp8/P3r0aAgql7/66qt8hsiRLeSoI6PgyGj16tXr1zdYyq6WXLyo6QErl2vvbQrEvH//fteuXU1Np02bBn8OHgxn2rZt2y5dRowYMXfuXL7z4vOD8920qbLhhC5QiDE9e+bMmcPvxx3w8ePHH34or5fJU2luGJuY7Nq1i88QOeqoKThyq54jnAF8D5lOO4q6eZghg3DMx4+HDRsG4VSq5OTkzp07y2RQH+cYupkzJsfyhISEdu0GDRrk7+8fHCxj5OMDC3fu3Nmtm5ubG7v/qJ4fh2FZWVlgYGA9ROZLbcacnZ2d3YYNG44cEQiIHLVCjs8gx3rFxcVpe+wwHMViqlNTFU8fytD+QvrWW28RHEgwYLt27QDdTz9BFX7Pnj11b1Jdld7Scv/+/YJ+HIaJiYlyXSmeqn///qdPnxYLiBy1Qo7PMMcffxw7diyDEP5ZsmRJaalYTHV2NpwkkwxCQtG4YMEtzvkPOGNyr4qPh/p7ampqba2mnJ0xw8fHZ8KECQcPaueB5vPjMIyPj9ciZHoLQ1CofPv78z2nqG+IHOvNWoCjtbWCQSiXd+vW7cyZM+Ix1TU1Q4YM6dSp09ChSUlJ169fJz2U1ICGzq9QUFAArckuXSZNmgR1kcmTgeeFCxdIAyJHrZCjvp4RjpqZSXr3Xrp06eXLl8ljGqwm8aNuiBzpGCJHOobP5PgUjfdDjnT8kCMdP+RIxw850vFDjnT8kCMdP+RIx6+JOKIaK+RIR8iRjpAjHSFHOvofJ7UksAplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjY3MjgKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTA5MTYxNDQyMDMrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My40LjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My40LjMpID4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDg2MDIgMDAwMDAgbiAKMDAwMDAwMDYzNSAwMDAwMCBuIAowMDAwMDAwNjU2IDAwMDAwIG4gCjAwMDAwMDA3NTUgMDAwMDAgbiAKMDAwMDAwMDc3NiAwMDAwMCBuIAowMDAwMDAwNzk3IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM5NCAwMDAwMCBuIAowMDAwMDAwNjE1IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU5NSAwMDAwMCBuIAowMDAwMDAwODI5IDAwMDAwIG4gCjAwMDAwMDg1ODEgMDAwMDAgbiAKMDAwMDAwODY2MiAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDE1IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAxNiA+PgpzdGFydHhyZWYKODgxOQolJUVPRgo=\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["pl.seed_everything(44)\n", "samples = flow_dict[\"vardeq\"][\"model\"].sample(img_shape=[16, 1, 28, 28])\n", "show_imgs(samples.cpu())"]}, {"cell_type": "code", "execution_count": 28, "id": "4ff785f9", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:03.509095Z", "iopub.status.busy": "2021-09-16T12:42:03.508627Z", "iopub.status.idle": "2021-09-16T12:42:03.616157Z", "shell.execute_reply": "2021-09-16T12:42:03.615677Z"}, "papermill": {"duration": 0.158469, "end_time": "2021-09-16T12:42:03.616267", "exception": false, "start_time": "2021-09-16T12:42:03.457798", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 44\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDM0MC41NiAzNDAuNTYgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIgL1R5cGUgL1BhZ2UKPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMiAwIFIgPj4Kc3RyZWFtCnicPY1NCoMwEIX3c4p3giSTtHGtCMGl3XiAEFrFH6xQr99JaVw83scwj48xka4ZzwMGk+QEI0C36TPG9AgN4kFG7gu5m1F3LzgX/NcsDxe+iFbaUSn7i7NesS/1ThiwQtc2C1mELEKDIAtnK2RR7rKOC3THaDf01NMXrxEktQplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjEyNgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjU0ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCvr6+urq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh2dnZ1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDMyNyAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMzI3IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMzI3ID4+CnN0cmVhbQp4nO2deVwU9f/H2QOWW1hABeRGETFJESSOUsQkNStvvDAvEvEg+qJZmlR4H+WVmn69vvJFzStM8UJU1BJTDMQQFTEFDBHkUhD38XvvZ5gvu8vMZ2ZhqLXf5/UHOrPDa1/zHPYzn8/s59D7gkgI6f3dAf4hIhyFEeEojAhHYURxVAinL1rFUGg/wQ0JR2EMCUdhDAlHYQwJR2EMCUdhDAlHYQwJR2EM/2aODx48SE/nYdjSVJp+hGOLRTgKo38ex5cvX5qYmHh5FRQUlJSUYA0Fi0c4CiXCURj9/Ryrq6vT06dPn37z5s2cHKwr75g2NjYSiUgk+vrrr7GGfP1AJ0+ePHy4vr6+uQFzc3OTk5M3bz579qxUKhWLIZ2enl6/fv22bYMLz2ZIOGqq9ThCqD17LCwsZDLZ+PEjR45MTEysqamrq6usrNQ+prp8fHxQUltbW7ZDtPLLyMiwtLScNw/SYf1YDSsq4NIaGxsbGIg1ZWgYHx/PZkg4qotwRNJpjhs3btTTCwgISE1NbdxZVFTUo0eP27e1i6mie/cGDBhgYGBgbg4crays2I7TguPp023atIEzlkqXLl2K9WM1vHvXzc0NLCQS+OHq6jpjRnR09O7duxFJKCfZDAlHdRGOCh3mCDf6mJgYKHGGDbvdFFn37t3Pn9cuJq28vLxx4+bOnVtbW7t9O3CEEontUF5+UB5GRkZaWFAlmVS6evVqrB/G8O7duzt37kQ+y5YtQ7suXbqEtqHVwGZIOGqqNTj+8ssvABFuqk1f+v777yUSSVaWtjEpHTp0yNu7vLwcLtR33wFHqAqwHcrDr7o6LCwMfRjBJzY21sjI2dkZ64c3jIiIQNygYoK2Q0NDJUpt2rSJzZBwZBDhqKMc58yZA7+ekpKi+UJ5ubW1dUhIyJMnzYiZn58vl8sfPqS24uMh8VtvvcV2NA+OX35pamo6c+bMY8cePHhw+PBhmQzu248fP2b3wxs+efKkUyc48QMHDqBtBwcH2JJKoVRnMyQcGUQ46ihHuNHDSRYUFKjtLS4uNjSEZijUEJoRc9UqOOnx48ejjYqKCgcHeIvFixezHc/l9/Dhw3btpk6dWlpairZPnz4tkUD9Hv5l98NzrK+v/+wzKLWPHDmiUCQlJZmZmaHS9/PPP2d4dEY4solwVOgmR4gDdtnZ2Y27Hj16NHToUAODcePGaRvzypUrUIh16ACWYBwYCBChPJPJoNmanJzMloHrtPv37y8SRUVF0dtQ+kqlUFvD+nFWzFavBnKdO3e+edPR0RFBlKCqEMPfDuGIkbAcFyxYAL+r1liFo8HTygqq6NrGPHjwICATiz09PaEF6+t77tw5OFQmg7eAKjlbBmxAuIMaGhq2aXPz5k1617Vr10SioKAgrB8nx6gouNoQTCajCKKf8PmGvwVGQ8KRWYSjLnKEM4ffBccVKwoLC2tqaj77DEo2V1dXxi8guWIC+n79+oWGvnjxQqHs2vP06VN7e3uRyMjICIpKrCHbi3CzhjN0cLhz5w7arqysjI6OlsmgCG9GQBVNm0bBo9WxY8f4+OvXr7MZEo7MIhx1kSPo7bffBm4SSc+ePbt166anZ2FhkZuby348JiZUlqH+TV+Cmpply5aJlIL9zD0VGg0xAaGtIBK5uLigxu/s2bOh4X/x4rNnz5oRUKHsTeHj4wP1HdQAQXJycoLWP1TQsIaEo7pajSOlnBypVAqnbGoKn2t03lrHPH5cX1+/U6dONTXU9qVLcrkcLF1d2Z7MqBhiXk9ISIiJuXDhAjQt8/Lq6urgVi2Xc/VfYzaEz62j4//uz0hubjV0YKwh4agiwvFV4Th9OnjGxcUVFcF7ubm5jRqldUwfH/qxKOizzz5r356q6e7YwSMmZ0CFAorEFy+g4QCu8+Zx+TEY1tbWwk2ZLhVpjgEBjL2YmhgSjrQIR0qvBMfQUJUnUfALRkbnm3alwMSEes2SJVSily9HjBghRr31EEeRKCIigismZ8CGdwkODraxsWF5gosLiL5cRf1vqfLRygpKc0S0a9eu+HclHNX0l3D880+VO2BVVZWlJdy0tYh56NAhPb1evXpBOD094Geo/GLC3d197ty51tbt2rW7f/8+NiZnQIUC7vrr1slksnfffRd7HHPAbdvQoyjq6kJrs6oqKioKIYV6PkPfJk1DwpES4UjrleCoJrgzBgWxjdRgjpmRkeHvTz066tixb9++Ko+Gz5+HnWPHjsXGZNg/bNiw6Oj8/Hx6++DBg8rny1Y7d+7EpmcIWFpairr8wVV2cvrxxx/Rzvr6evQYu02bNlg/wvF/IhxfaY5eXl4GBirfiHDFpAVFbFpaWtP90EL29PTExmTYD0WuWBweHo42fv/9dyhtxWI4c670DAGPHz+OSkao92zcqDIUbOnSpSLR66+/zsOQcFT8PRx//fVXqVK//fYb35hcunHjxoULF7AxmV/KzR06dCic961bPXr0gJvr8uWY58GYgEVFRW+9RX8p6OHhERMTc+jQypUr0egHqNrzMCQcFYTjq8uxvt7e3l5PWZW+evUq35gtE9bP19c3NTU1LAwuLLTSHz3i69fEsKwM6u+2trbK7qJSujMPyMursLCQhyHhSIlwfKU4lpdDCZSUlPT222I0zM7Xt7a2VouYLRDWD64metwKV7e6upq/H7MhVKUWLaLa2UjQYGCfV0DdkHBUUetznDJFD0lfXy6Xx8XF3brVjJjNE5dfXV1dVRXnl3oafoIHJBxbLMJRGDXnfs3T9a/k2Cw/wlEYP8JRGD/CURg/wlEYv1biSNRSEY7CiHAURoSjMCIchRHhKIxIvUcYQ8JRGEPCURhDwlEYQ8JRGEPCURhDwlEYQ8JRGMO/j6NE0r9/f76G/N57+3Y/Pz8HB4dduzIyMrp37z5oENu0RYQjTv//OKIh7GZmZnwN+b33xYuOjo5vvvmmoSHVs8TcvNkBtRDhKIwIR2GkHceampodO3Z4eVlbW0dERJw+nZ6enpSU9PChZgnEI2Z9fb2Li4uDw7179/jG5HEcUmZmJgSKjASOISEhjENpCUceIhzxMXkch6TrHNPTvb29xY1CI35Rj+rQ0NC1a9dqFfP+/ftwnp07axGT36G0jhwZPnz4jRs3GBdP4RFwyZIlsbFUB8UPPpg0aRJ2oh3CkVWEIyUd5Th//vzGaRtUOKLx/EDEzs5Oq5gff/yxSFnrwb+rmiG/QxWK58+fx8bGurtbWFjQ8+Uy+rEa/v77gAED0LwAr7/++vLly93dZTKZsbHx5MmT2YaUEI4MajWO58+fd3d3R9349+/ff/bsvHnz4E7o7w8k4M7o7Q0cf/31V74xkUaPHg2//cEH1FZJSVlZ2cWLF9mO1oJjTU3fvn3RvBLwidy4cSO7H7NhdXX1yJEQrVevXtnZVVVVaCf8C2UX7J01CxOQcFQR4UhJlzlCSxiViB/Q54305Mn333+vbUxKiYnUjC5ffllbWztw4MA2bdC4JpGzMzW/JqMhxo/W2bNnfXyoyYuUxZt4zZo12gacNm2aMoez5mwZwBcMP/mE+Y0JR00Rjgrd5piWlmaAVpYcNaqoqIjHyXBxvHXrloMDtfLsmjXAzcnJSdQguZy5psKDY23tzp07odRunM7W3t5euwmvFAoo/dECgNBE13wJgvr5+fXqxfzuhKOaWpOjP7ovT506lX26Wb4x/6e2bcESKsuo0RYVFYXq89bW1rGxGEOMH9QaDh1CNWdqRY5OnTpJpdHR0VBq7NkDpdHly5fVnkkxB6yqgno3XIDDhxneIiMjA9xHjMAEJBwpEY66z/H+fblc3rlz5z/++ANzHkyurOe9dOlSVKU/rMy7a9cuc7S6q0h08OBBrCHbi1evXu3fvz/d+p89G+CdOXNGKgWYc+fONTCgvqcxNTW9cQMfsKwMOK5cubLpW0ArwcPDAzKyTWlDOKqIcNR9jqhwhDhoA8oYqJ6yTEqh6cpsCBaBgYEiUUBAAGxt2UI9elu9+vnz51yGzC8VFg4aNIiGCFWpqCh4i82bN6OnevS4fqoi5O6OD/jsWUVFRT3T6P9Lly6hWpm86YopKoaEI6VW4wi2VlZgABVQ+HNvmCtWLJXa2dnBh5N92DiGI3wIrZSeERERY8aM0dcHdx8fH5b5lNQNmQN++CHFKTgYPrlGRkZoPui+ffvSM5689tq1a9fCwsJgq3HFZc4KhYb27t0LQSMjI7EBCUcuEY46wRGE6rZiNL0kWkRLRH9LKBavWMH8cAYbc+HChcgCilzgKRZbWlqmpKRwzUjG5jdnzhzE67333svJuXv3LhRu5eVQOELTGu0HffDBhx9+KENLbA0ezCMggwoKoMICqRknXFQxJBzxIhx1h+PUqUDM0dFx69atqB8PNF4lEn1UrolEsJPdldkQ9iOOa9asQY9u4+LieJwJo9+hQ4egPJTJTExMJk6cqFB4enrGxMQMGqSPJJPBD0SSKj/Hjh3buDSLdhyPHoXIHTp04FpgiXDESwiO9+6tW7du0qRJKrtOnVq/fj3iCP+yuzIbxsfHwy8aG8N/USmh+SQGE1N9V3k5XEVjY+ONG6HRNnny5K+/pp42059nfX1q0jypFHh6e3urrUanHcePPhLRDQdsQMIRL8JRdzgyCs4A3bE3bNjA7spseOPGDXd3d5GoZ8+eqOiCk5w2bRr7CpAqhirbgP6HH6jGs5PTjh074L90ryN9/fbt24Orh0enTp0ArY8P/CFoNpq14FhcXIweGWGeRikIR24RjrrPMTc3F0GAZie7K6vhsmXLLCzo+jz1r7Hx7Nmz2Wr1iqYcMzMzlf2NqHlsqXIQlYeghQuhPX3ixAnsCWjBcfHixSjiw4cPuQwJR4xak2NOTg66Xzd7vmZUGZfSbc4GLVmy5JtvMDFVtgEVKhdojuhfaLxhuqxpF7BBWVlZZmZmKN23337LZUg4solwxOuv5/jkyX/+8x/1Xfn548aNQ+UaW5nBGbOysrCwEErZrVsjIyNtbGzatqWWjo2KqmMfX6Cyfe/evQ4dJDRCpIED2a5qcwI2CFoNCKK1tfWZM2e4DAlHNhGOeP31HCsqnJ2d6U4A+fn5U6dOtbQUoUXz1q9naxlrUa2gBTWZlJSUsjJMTPVdw4dT/Pz8vvnmmx9++AFbLWlewMuXO3TogP5gjhw5wsOQcGSWkBwViuDgYGhm2dvbv/EGtLTQfRr+0qEy/fRpy2JqoVbx4zS0t6cqEu+8U8Gr4Uo4Motw5OX3F3O8detWeHi4t7e3iQkUR9Dc/Pbb7Oxs7MPXfwjHR4/kcjn82YSFcT5qJhwxIhx5+f3FHJuhfwjHEycA4sCBA9PT+RkSjswiHHn5CW5IOApjSDgKY0g4CmNIOApjSDgKY0jWVxBGhKMwIhyFEeEojAhHYUTu18IYEo7CGBKOwhgSjsIYEo7CGBKOwhgSjsIYEo7CGBKOwhgSjsIYEo7CGP59HJ89i4yM9Pf3/+ILHsMqtAhQWlpaVVVbW3vu3Lnk5CtXrvzrX/9qXkC+IhyFEeEojPhx3LVrV0JCwsSJbm5uNjY2ISHt2rXz8vJKSHjcOMK+eTEPHHBwcEDj9lesWMEVk4cfUmZmpp2dXZcuKSkpo0aNun1b64CFhampqRCou1Jnz57lHtdMODJJcI7l5eWpqe+++65EIkG9R+nBgBYWFmiw6ZAhQ9hdWWPW1NTs3Zubm1tXV7d2rZOTk6FyKa0RbDNUNhpyngwaB3viBFxlCHz5MuNMZZwB4+Pjvb2NjIzQAEpqSPyKFTcap+rDGBKOKiIcFTrLcdCgQYgcnCmUFKGh/fr1y8nJKSsrKCjIyspKTGzesiTFxcWo23VeXp6/P8RE888EBQVxxcSfCYKIBkiB9Zw5c7CHYgIOGzZMJII/lM6dO48ZA6U3IjlhwgQeEysSjioiHHWXI5qBzMXF5dSpU1wnwTemAo0KE4v37NlTWVmZny+Xy+G8fX3hunAZ4t+ztDQ4OBhNMBAYGFjGNjCMO+DAgQPF4m8aBjAfP358wIABEolMJoNLtGrVKqwh4aii1uCIPnH79+/Hn4FWMUF//vnnyZMK9Dm8cQPFFn/1FQ9D/CEXL1KDhgYPPnDgANf4AkxAqDisWAER6W2oVYwdC1UAQOnt7Z2UhDEkHFVEOOo0R83Fe3gJz5HWixcvTp40NzeHEoh9xhkVQ/wh334LEH18fNCsvYmJibNmcc1jo8Xzntra2j/++MPW1lZPb/fu3WyGhCOXCMeWBGxUCzlu3boVTT46fPhwe3v7xv1QJ01LS2NfcYFfzJ9++umNN6RSaZMJMFgM8YcomwyiH3/8EW1UVVUNGQIlXTMm8FUoC253d/eUlJTNm6urqwsLC8ePj4qKop4tSCSurq5btmxhNCQc1SU4R6iGSiQ2NjYBAQHQXPr557i4OEtLSzc3uH2JlTOc//LLL82IiQRnCBxRC+wp+4B4NUPM60eOHBGLoYigJ9K4efOmvr6BgQHbJCeYgNnZ2TNnAjSV5z1iMTXfMz3bbps2bfLyGAwJRxURjrrL8e7du2imy+7du6OywdjYuFu3bv37Qwu2Y8eOEgnwhOJDq5gqAksrqy5dukAr3tYWbtoff/zx0KFAAN71u+/UH8Jy+X3yyScikcqj4OvXr6NKeTMmVoRza1jni55nV3/ixMrKyuLiYrhacjnsYvhVwlFThKPucoRCbNEiauVVuN1/9RW9jLFCOXfDo2XLqOHyWsWklZSUBGEnTIC3CAoK6tAB6rjh4eE7dkAhh+KHhYVpGrJZbdu2DX2BpDILNyovIficOXMYW9qYgE5OTjRCqRSKWJVmCFT3+vSB/fn5+YyGhKOKWoMjUl5eHnzSGF6oq0NfHhprFZNSWZmdnR1cGrWTLCgoQIuLo09kdHS0piGbmb+/P/ocqnB8/PixgQG1tCRjawETcMmSJQYGs2bNioiIWLr0zp07Ki9B2wNNxcuAg3DUFOGo4xxZ9ewZNclt02o0J8f334c4UA9X2zl27Fh0e5w/f77m41KM386dVOU4OFgt2jN0x4X9P/3E8FAXE7C2tpZxKk+Qr6+vWOzg4FBeXs5oSDiqiHBUi/YKcXz5cujQoYCy6fp7eI6JiYli8fTp09V2jhkzBhryVlYnTpxgeNiF8UPf24IWL87Kytq3b19lJXBLTk4WNeiNNzIyMrQLyKiHDx9CmS4Wf/rppwyvEo581Xoci4vd3Nygssviymy4evVqOOmwMJV+zPBJmjsXdpqYmEyezLwGFiYgvTplQAA1YzpalAtN8Q4QO3fu/OQJm58WZwz3f09PTzA2MqIfFTMYEo5cIhxfAY4zZkBsZ2dnLWImJkrRellqM6ZDzVf55McK8z0NJqCRkVhNDV1eQdA4Zut1rh3HU6dcXFyoJnfPnpiAhCNehKPuc4TyzdcXoKxbt06LmFeuQIm1cOFC+iFtbq63tzecsqtrZmYm5t0wAWfNogkaGRmNGDECPYRF5WMSW2cc9oAJCQnXr19X21VUVNS3L/VcNyQkBDvhPuFIq/U4du1KnbnaOhy//fZbu3btRKL3339fq5gKtMK3tbV1QUFlZeWZM2e6dAkNDT2Alv7GChOwru7Ro0dpaWlHj8IluXDhAnoiY29vP2UKptMZc8D6eqhN2NrabtqUkpKiUA7XiIiIsLS0RJ/onTt3YsbuEY4qIhx1n+OECXD7MzU17dGjh5+fh4dHQEBAjx7GxsawVy7/g6WLD4ZjXl4etKNRIcbwdTo2JtdBUFl+8ODBnj1wc7127Rq2JylbQLgkvXv3RqtIwN/Ia69RN2lb2wULFmBGNSkIRw0Rjgod51hdPW/ePGrxLCQ9paDyPX369JISbWNSgnJr06bu3bszf+vDbsj3YL5+zIbh4eENnXugqtO2bdtBgwZdusTPkHBUEeGohf56jkLHbIGh0H7MhiUlJcnJUPlBDXS2BeeZDQlHFRGO2voJbkg4CmNIOApjSDgKY0g4CmNIOApjSDgKY0g4CmNI1lcQRoSjMCIchRHhKIwIR2FEOAojUu8RxpBwFMaQcBTGkHAUxpBwFMaQcBTGkHAUxpBwFMaQcBTGkHAUxvCfx/H+/fsHDjxhHILU6MfP8OXLl2VlYFVRUcEVkHDEiHDUfY6ZmZn+yinUm7e+gpqysrISEhJiYgyQbGyYu2tqFfDx48fOzs5isYeHR8+ePZnmaeIdsKio6NNPP0Wdz2QymZWVVYcOP//8M5sh4cgmwlHXOUJJsWiRjY0NGkEF3snJyU0LDn4x161bZ2kJ8KhOm8oZqweLRJ9//jlbTH4B0aoQtCUoJobNj9Owvv7NN98EC0tLOT3rAJqIn82QcGRWK3GsrFy7dm3Xrl0bx5ZSXZ0/+qgZMYGXVCpFU6eBUdeu8+fP379/Pxp5zrDqlxYc16wxNDRsGPkqMTc3R1O+NyMgUklJyb///e+iotLS0i5duqCBi+np6WyGhCObCEdd5Xjs2LGGQrGJAgO3bt2qXcwzZ+ghEHCy8fHxz5SKjY3V0zM2NlafVKzRkOuka2pqIiMj0Tj2CRMmlJSYmpr26tWL8VAtKhSVlZVnz0KRSy3Z9cknzEcRjlwiHHWP48GDB01MTJgpKsshzcmh8DGhcuvmRg1p8vc/efIkGt8Lb4G42tvbMwzP5XHaL15AvQnleeedd1AR26dPHzs7O8ZltfhyfP78eVRUFLoRQKn7FfvyLoQjVkJy3Lt3r0q1FgkIxMTEmJtTN1u5XLuY4eHhaHTdggULGneOHDkSoe3WrRtbTDY/SsuXU9n696fLhffRrH6//87mhzd88eLFlCmjRo1C52hmZrZ8+XLM0YQjmwhHneT488/o2QlVDnbq1GnNmjXHjxcXF58/f97QkOLYdAVHTMwZM2ZAk9rCAspDtf1btmxBFygiIoItJuaMV6xYYWAAv79x48bG5zshISGwa+JENj9Ww5cvJ02aFBQURC9oK5Xu2rWL7WAVQ8JRXYSjjnLMycmhpwUUK1dpVnlp5cqVqJ7i6el5+TLvmAcOGKD5qHv3Vt8/bRqUi6ipzvw1COa0i4v9/f3R/FFwidVmPQwLCwPLffvY/JgNr1y5MmIEdcL6+h4eHtu3b3/2jGsdP8KxiVqDY25uLr12jaura2qqykuOjo7ovWbNmqXFdMjKtXxFDg4OjXuys7PhE6SnByQwnzQ2P6hv+/mhKoM8MVHzZL/88kt4SaPJig/43//+193dXSzu169famoqj/kU1AwJR1qEo05zfPr0KXrK4eXldeHChcaYS5Ys0VcWnL17965pOus1W8y8vDwLC+Boa2uLrKDxGhdniO76enqak7IzGGrurK6uHjJkCJqv8BLTKc+cOVOsXIKZza+J4fr1YnSLfu+9Orap2LEBCUdKhKOOc1Qovxzt0aOH+syC+fkIotjSMo9lfjLmmFlZWUZG1KPbjh2hdk/NktYwu5Kfnx+UZ9iYmjs3bNgAv2pism3bNsZfgso0pNy+nXdAtPgWWM6cuWrVKqjzMf2J4AISjpRak6OmKisrlQ0mpebOxboyGXp4iOlvGRv4mZubQ5GBZvWDWzbWUH1Xaip1NTdsYP6VwkIjNJWz2tTQ+IADBkAEa2tr9C0mNDD69MFMsctgSDhSIhwZpMscofFrZWWFIIaEhHC4MhlmZ8+bN2/atGkTJsDrV69epRvT48dTT1awhirbt27devttCwsLOE+26S2PHhWjdcUZ77zMAceOXbdu3fXr1zdv7tOnD+qlAEKzaV++fBnfxCYcVUQ4Mkh3OdbXL168GNVbunbteu3aNQ5XngUu0qhRKDPDEiwqhirb0AAwMmLu4kcLfd/zUdO+R3wDnjt37vz5O3fuBAcHi0QAE8rNwYMHs8EkHNlEOFLSUY7QGE5IoAoxe/vLTR+BaxtTRfX19YGBVP8KrKHK9tGjR52dGdZDQ8rNzV29erWpqZmZGXPnWW0DFhePHj2aqus9eoQJSDjiJRjHjIyMyMhIBBH+whm/WW9BzLKyMltbqgcN1lBlu7a21t8fYJ0+fXrt2pLGiaPham/f7uLiAm4ymeZXks0OqJyNXtlQaN++fWkpxpBw5BLhKETAFnOsqakpKNi9ezc17AG1iI8dO8b9ttrFnDx5MrJetGgR1lB9V3397du3g4KCRKK2bduOGTMmPNzd3R2KWFSMBQYGMnbj5hsQrlOxUtRWeblMJgPXkSNHNu9+TTgiEY7NDSgQR0gYG7tlyxZvb2/UjZu65/v7832MxI9jYWHh9OkzZszQ19cXibp168a13JDmzvz8fOBFr8+MfkDW0aP37dvH9f0KV8BTp06hInbixIknT550cICNXr16YVwJR0YJxNHS0lKsqi5duqxcuZJ9WLi2MRWo8h0XF0c/z7Wzg+Rchgz7y8vL79xJS0ubMmXKuHHQkrt58ybj+rDaBkxPT3d0dKS7xItE0BDGr05CODKKcNQpjr6+vspea/rt2rUzMIiOjoaSiEc4/jEViCPUu/X04Ie/v/+9ezwMtYrQ4oAVFRV37yYkJIwfPz4hgXPJIcKRTYSjMAGF4Ag3+LQ0zj7RLYoJWrZs2eDBjD1zmA2bm4bNT3BDwlEYw3/ePEi8/AhHYfwIR2H8CEdh/AhHYfwIR2H8WokjUUtFOAojwlEYEY7CiHAURv8HK67ZpwplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjc5OTcKZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMSAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTA5MTYxNDQyMDMrMDInMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My40LjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My40LjMpID4+CmVuZG9iagp4cmVmCjAgMTYKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMDk4NzQgMDAwMDAgbiAKMDAwMDAwMDYzNSAwMDAwMCBuIAowMDAwMDAwNjU2IDAwMDAwIG4gCjAwMDAwMDA3NTUgMDAwMDAgbiAKMDAwMDAwMDc3NiAwMDAwMCBuIAowMDAwMDAwNzk3IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM5NCAwMDAwMCBuIAowMDAwMDAwNjE1IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAwMDU5NSAwMDAwMCBuIAowMDAwMDAwODI5IDAwMDAwIG4gCjAwMDAwMDk4NTMgMDAwMDAgbiAKMDAwMDAwOTkzNCAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDE1IDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSAxNiA+PgpzdGFydHhyZWYKMTAwOTEKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["pl.seed_everything(44)\n", "samples = flow_dict[\"multiscale\"][\"model\"].sample(img_shape=[16, 8, 7, 7])\n", "show_imgs(samples.cpu())"]}, {"cell_type": "markdown", "id": "d070c015", "metadata": {"papermill": {"duration": 0.049106, "end_time": "2021-09-16T12:42:03.715575", "exception": false, "start_time": "2021-09-16T12:42:03.666469", "status": "completed"}, "tags": []}, "source": ["From the few samples, we can see a clear difference between the simple and the multi-scale model.\n", "The single-scale model has only learned local, small correlations while the multi-scale model was able to learn full,\n", "global relations that form digits.\n", "This show-cases another benefit of the multi-scale model.\n", "In contrast to VAEs, the outputs are sharp as normalizing flows can naturally model complex,\n", "multi-modal distributions while VAEs have the independent decoder output noise.\n", "Nevertheless, the samples from this flow are far from perfect as not all samples show true digits."]}, {"cell_type": "markdown", "id": "e16d33ec", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.049321, "end_time": "2021-09-16T12:42:03.814088", "exception": false, "start_time": "2021-09-16T12:42:03.764767", "status": "completed"}, "tags": []}, "source": ["### Interpolation in latent space\n", "\n", "Another popular test for the smoothness of the latent space of generative models is to interpolate between two training examples.\n", "As normalizing flows are strictly invertible, we can guarantee that any image is represented in the latent space.\n", "We again compare the variational dequantization model with the multi-scale model below."]}, {"cell_type": "code", "execution_count": 29, "id": "c066c621", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:03.919045Z", "iopub.status.busy": "2021-09-16T12:42:03.918562Z", "iopub.status.idle": "2021-09-16T12:42:03.949438Z", "shell.execute_reply": "2021-09-16T12:42:03.948964Z"}, "papermill": {"duration": 0.085776, "end_time": "2021-09-16T12:42:03.949542", "exception": false, "start_time": "2021-09-16T12:42:03.863766", "status": "completed"}, "tags": []}, "outputs": [], "source": ["@torch.no_grad()\n", "def interpolate(model, img1, img2, num_steps=8):\n", " \"\"\"\n", " Args:\n", " model: object of ImageFlow class that represents the (trained) flow model\n", " img1, img2: Image tensors of shape [1, 28, 28]. Images between which should be interpolated.\n", " num_steps: Number of interpolation steps. 8 interpolation steps mean 6 intermediate pictures besides img1 and img2\n", " \"\"\"\n", " imgs = torch.stack([img1, img2], dim=0).to(model.device)\n", " z, _ = model.encode(imgs)\n", " alpha = torch.linspace(0, 1, steps=num_steps, device=z.device).view(-1, 1, 1, 1)\n", " interpolations = z[0:1] * alpha + z[1:2] * (1 - alpha)\n", " interp_imgs = model.sample(interpolations.shape[:1] + imgs.shape[1:], z_init=interpolations)\n", " show_imgs(interp_imgs, row_size=8)\n", "\n", "\n", "exmp_imgs, _ = next(iter(train_loader))"]}, {"cell_type": "code", "execution_count": 30, "id": "dc209bbb", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:04.053707Z", "iopub.status.busy": "2021-09-16T12:42:04.053237Z", "iopub.status.idle": "2021-09-16T12:42:04.365802Z", "shell.execute_reply": "2021-09-16T12:42:04.366194Z"}, "papermill": {"duration": 0.36694, "end_time": "2021-09-16T12:42:04.366333", "exception": false, "start_time": "2021-09-16T12:42:03.999393", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDYzMS4wNDYyNSA5NS45NCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjUEKgzAURPf/FHOCJD8mX11ahNCl3fQAIbSVamkFe/1+XUgXAzPDDI8xku0YtwUOo+oLRoLty/rI5ZJOyAs57SeSio0L4qOm519qo2mDNu5wd6KZ3qiN3yUsRvZpwyYGfAqumGE7v2FZsaxYh6Qf4RobrvHHPU+wZ0b/wkAD/QBzWSXbCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTM1CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyNDAgKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/b29vX19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Obm5uXl5eTk5OPj4+Li4uHh4eDg4N/f397e3t3d3dzc3Nvb29ra2tnZ2djY2NfX19bW1tXV1dTU1NPT09HR0dDQ0M/Pz87Ozs3NzczMzMvLy8rKysnJycjIyMfHx8bGxsXFxcTExMPDw8LCwsHBwb+/v76+vry8vLq6urm5ubi4uLa2trW1tbS0tLOzs7KysrCwsK+vr66urq2traysrKurq6qqqqmpqaioqKenp6ampqWlpaSkpKOjo6KioqCgoJ+fn56enp2dnZubm5qampmZmZiYmJeXl5aWlpWVlZSUlJOTk5KSkpGRkY+Pj46Ojo2NjYyMjIuLi4qKioiIiIeHh4aGhoWFhYSEhIODg4KCgoGBgYCAgH9/f35+fn19fXx8fHt7e3l5eXh4eHd3d3Z2dnV1dXR0dHNzc3JycnFxcXBwcG9vb25ubm1tbWxsbGtra2pqamlpaWhoaGdnZ2ZmZmVlZWRkZGNjY2JiYmFhYWBgYF9fX11dXVxcXFxcXFtbW1paWlhYWFdXV1ZWVlVVVVRUVFNTU1JSUlFRUVBQUE9PT05OTk1NTUxMTEtLS0pKSklJSUhISEdHR0ZGRkVFRURERENDQ0JCQkFBQUBAQD8/Pz4+Pj09PTw8PDs7Ozo6Ojk5OTg4ODc3NzY2NjU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLi0tLSwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx4eHh0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExISEhAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNjE3IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCA4MiAvTGVuZ3RoIDE0IDAgUiAvU3VidHlwZSAvSW1hZ2UKL1R5cGUgL1hPYmplY3QgL1dpZHRoIDYxNyA+PgpzdHJlYW0KeJztnflDFOUfx2UPQFFBBCExwKQDEQSlFMW4kcwK8MrSklADRRPSIsAIUCGUPDArSQQUzJVTCyUhQinQCNNA0Lg0OeXYP+H7zLM7zH3sspHf9nn/sOvMzrz3vc/zWubzzD4zTtiNhDQemvBvB0DSEyHSkMZHiDSk8REiDWl8pCJNqRvt1rmZLr1QME29dGqGSNPOTJde+hEMkaadmS699CMYIk07M1166UcwRJp2Zrr00o9giDTtzHTppR/BEGnamenSSz+CIdK0M9Oll34EQ6RpZ6bZLpWVMTExXl5eGRkjIyNjCvboUW1trZubm4GBVCo9e3ZMwe7du6dQTJs2zcDAYMKEsLCwMQVrbi4uLp4yZQrmNWHxYmYwRJp2Zprtgkj7L5F2+/bt0NDQRYvi4uLGGKypqWnPnokTJ0okkhkzbt68OaZg33yzfPlyuVwCNXPm5cuXtQ+2a9cuV1eJWoC0U6e0DQaA37vXxcVFMqrw8HDtgyUkJPj7E14SDw9mMEQaU4g0RBqfEGmIND6BWmHfvn2HD3OYaeC0evVqKSZDQ8PsbE2CDQ01NDQoFLGxsdnZBQUFUVFz584lGu3ll18eGBAdrL+/v7x8586dlpbe3t4+Pk5OTkZGErLWrl0rOphSeefOnQ8+sLa2DgoKCAhwcgKfjmI2f/6tW7fEBYNqbGzcsCExMXHz5tdff11CFajXfvpJdDCogYGBnp7c3NxFi4yNjWlmR48epZkh0hBp/zZpmZmZYBRB/Zs6+g6Dg2y7cMcDGEyY8N5779FWl5YuXboUJPP35zBje/OtW7eCYBJJTk6OUjk8PAxXp6VNnjxZqhYDXI5gQ0NDCoXC0VF1LJJKDQzgvyiNJpfLr14VE0yprKqqevHFF9X7GUAzhubPny8imBI7vp08eXLevHlqMzYvyezZd+/eFRFMpepqDw8PVh9M4DXyN0qYtJYWf8oRk2rGbDFEGiINkYZI0wfSHj50dnYG+2Rl0dJGRUWZm5u7uIDjM2Mnjnig+IEVxsaNG4mVra3p6elmZqpkzz3HTMARbN8+/OOcPn26qSk5ObmsLCwszN6e9EHT08UFA18mzpYnaeVKyvlWjmBXr1pZWQl6gSoyNlYwGFBcnKoqo4FPkVT6xRdfCAaDSklJmTiRXudhUn0jDDBpUKd9+umngYHcwb76irI1Ig2RhkhTItL0hbTWVtU+x48rMew6OjoqK0GBRLTl9u3bGTtxxDt16hTcZcuWLY8fFxUVlZUdPHhw2TJSsmXLmAnYvECPb96Mt/K6devMzVk/6OefiwnW1uZKnAflk6HhiRMn+IP9/fffXl5ivEBnm5sLtthnn302aZJqaw4T9bM/ucLl6Mq6uri4uIkYZ0wnmQz0R0RERGDgjBkzRJAGOqCj4+OPPzYxMWF4mZqalpeXX8ekLp4pZog0YSHSyB2ga9KyslS7W1iAQ5OdHd11yZKenh7GTqzxhocDiT+0cKTJbLfbt5kJODqU3Eoyjm4gc8EdLC2Nshebl2q1XJ6YmMgfrKSkhJKAIxh8MDISCKZQ2NnZ4cNNzmBQLi4u/MGA0tPxcTAlmFTq7u4eEaHa5uJFR0fHDz8UCKZUgl5nSxMUFBQcXFFRwXxzwgyRJtChEkQaSYg0qh0ijawnnLSenhUrVkyZMoVwmzTJ29s7PFy1wFKkccdTKBS0xgeLvr5PP/00XADH/NpaDjPm6ogI1pan6MsvxQRrabFjfoXYujYvb5hceLB5NTY28g0SCW/w9WD8VEk3Gxw8duwY82QtfAOZTErIxeUUeToHR4uRvwS4nnnmmcOH+/v7mVvzBWMjzcPD49Ch7u5uTiclIg2R9kSQBgWGEklJz2ECw42iIrBm3ToJHGS0tGgQD+j8+fObNnl6epqa2tjYBAYmJSX19eE/ZuzYsYPTjLk6OJjU7nDmhoHB1KlT8X6GPZCZKSZYXZ0RbaIFkzS4EBoqGGz9+vXsZNHxA2vu3xcIlpLC+lMWQRq+8NFHAsEGBiIjIynDRLjf5MmlpaU3bgwNDbW0pKamFhQUpKYOUGesMIP19fXt3u3g4EB4TZ8+fcGC++SPc/fu3a4uJUOINEQaIk2JSNMr0mgC9RY8VZqXl8e+ATdpNI2MjCQny7E5zyD6bbaRJ3uHgsrC1hbA5ebm9u67lZWVsHzauxegr25+leLjh6nnDlmD1deTemDduuPHjycmEnxICUdbW8FgISEh5BOqUvIC8aBa197OF6y2tpblrOgoprgJfJ40SWAuh0LBsT9oexub0NBQeOoarluzZg3jJzeKWXFxMcVs9uyysrLu7uNkgcLezw88nzvHbDFEGiLtySPtpZdegn8ym5qa2DcQTdrg4KCvL8z71ltvtba27tnT2dnJZkZbl5+fL5UeOHCAsvLoUXqPyGQNDQ2CwWJjVa28ejU4mD96pMRGkKwdPGeOQLDmZjjNjnSs5DsLdv06X7AffviBgzM2r+TkZL5geXnMXTjswKA4OpovGIU0UK9cupSZmcmYJK6SlVU2eYY9Ik2CSCO1GSKNJEQazeu/TNq2bds4NxBN2p9//gk/sofH48ePwbK7O5NeNq+NGzdOm9bR0UFZqR72gUbcv39/bi5sNdBf/MHq6+ttbcGmfn5+MAHUjRtEX+AVDXhwcBAIFh6Ob0oRpXIjtHUrMc5jBlu0aBEFDXbC8BcsLS1LS7mDwSmGxI4Szq8AlLn5udECixnMwsKCwuWsWZQT+zSB0pm4shWRhkhDpCkRafpLWk6OmZnZ0qWPYEXDLtGkgeaUSoEdnHt3584dM7O2tjY2M8qaigrw2WhnK3/++WcbG9jyGzZsAOVfQgL4mM8/LxgMLGKzKoyuXLmiVHZ3d5eWpqSk5ORQIMHl5SUQbPFiEldk0ihdS7xwfbRUE+hQCiTMQhA+EPO7mcEMDdm8uGEjJoszg3Fc7MUt2lcAkYZIe6JIq6mpgVeXrFzJt5VI0nbsgKmdnZ3hYmNjo4EB7HGmGbF4//59OCH80CHSNvHxcvXNVaysHj58qMROoUqw2XMCwYqLYYcaGxvHx8cvXvzCCy/Qmt/JydvbG6djzpxr165xBsvKyoJHERwuMmZ419Jw+/777zmCpaXhH4hNrJ1NXNxK86qqquIDgc0rIyODIxgiDZGGSOMWIg2RxqkjR47AvfPz+bYSQxoYds6cCb3S0tLgmrq6OolEcOwJyiiJxN7enphFcvDgQRMTvPRReyn9/IDzmjUCwYKC8P1k6qmFMBFRSV24kJubSyza2dlxBsNvQcCoz+jgEa+VlZVxBOO8iBLuSgmLi7juluZVXl7OYWZismTJkqam4OBgipexcdbo5b0iSTM2dnV1PXmytbW1pGQZ+XI3a2vrykpKMEQaIu0JIq2zE94+0NPTk+UyFZ4OZdNR9Y9HNjZ//PEHXLNt2zaptIUx443mFY5NLF9CPi4uxoZ8eA/Cu7Q0NDRYW4M1e/cKBJsxg4KGlHE27PTphQsX4tvIZI6OjpzBnnrqKWL6BpNZMn2q1SYm3MFWr+YjjQ4ZNqFiNjEDg+ZVXV3NNvaUYHO7Q0JCFiygH6jnzeMOplQyb7kmwU7BJWHauXMn7YWYmBiaGSINkYZIQ6TpHWmjRRpvlSaONDc3N2i2axdc7O3tdXd3l0pXrVrFZkYsgqpMKl2xYgW+3NPzyiuvqPsO1AojI4ODg8uXL4fFBPnqTNZgx47BkkcGxaACrLOwMFT3Elh2dKRcykjzyiHO9zJmPbKSRp5HSg9WX2+G3xWHBTbmujNnznAGA1JXxGIEyrCSEu5gzJmQfJo797fffqOZIdIQaVBPBmkBARLsDueNjY382wmTVlEBe9DIyEihwNe98847MTGXLl1iMyMWb926JZWCQ1p/f3NzMxjrvPkmcaQKCwvr6/v666/BGrn8jTfeYD8Sk8y6uxl4sYwYwfOzz0ZHR5OvyWAGe/DggVxO2o+BBv4aGI5t3QoO8PyTqJ0pv4ozhPtPn/7+++/fvDlIvjkns/Vfe00MF1ZWVgkJ4GPwB6uqqpo1S9jMxubixYts96BGpCHSEGmINP0irb29Hd7X18eHbyuOeHRlZ8M4YLwmzoxYvHLliqEhGC0FB1taWhJgSLDrfK5du5afD4Zhe/bsqa4WE6y3F3QoG2RSrEqDTxYWFps3b25uFgwGOjs6Gnx/2Moo3HPqVH9//2qWbMwWq6mp4WBNHRW8lZeXV3GxYDCgrq4F2BCTj4xNmzado13KxB4MqLBwFj9s27dvP3+eIxgiDZH2pJAGGgj7m2hTU8O3FXc8ik6cgE0WS77XK58ZsXjv3r2CAh8fH/wgBY9Y8F8y2auvvmpqun79+qGhIQ2CJSWRjpS0qWXLlqWmpooLBpWSkgL7gMmuoSFIR7vvq0Cw9vaQkBC2337AuDQycv/+/RoEA4qOhjfVpzALHmxtv/322wsXaFfG8gZTKi9fvsxGmK2tbSmmx8T8ZYYZIg2RJjoYIg2RxhlM+X9E2qpVq7BbHrPc81hsPFydnZ2+vhASynwvPjPaut7e3pqaAwcO+EKf0cLK3NycMm1NXLDh4d9//33Lli1mZmQv2dSpb7/9dl+fZsGU2N1PQJUC78iGfwUWLgRFEPV/WRQVDNZ+hYUZGRl4rKioTz755LvvmD/ZiQkGb05x4cKFs2cLCwvPnCkqKqL+v8maBNNOiDREmqbBtBMiDZGmaTDtJIq0ujr4iz/ogbAwyo1ttIgH6gJ1y3l6erLfYIhuxvFaW1ubgwPeDXL52rVrHzzQOlh1taurKyxdDA2dnJz4mRUIBlg4cgS2WEBAwF9/CTj9Ex2qSy9EGiJN42BaeY0rab/8AgcX4KhAvhGwdvHAwMXYGKBhaVlJzPzlN+N++ddf7e3tZTIA2Y8/jjVYV1dXUlJkZCRjDog2wTQSIm1UiDRNg2kkRNqoEGmaBtNIiLQxOOpHu+nSSz+CIdK0M9Oll34EQ6RpZ6ZLL/0IhkjTzkyXXvoRDJGmnZkuvfQjmIo0JKR/Wog0pPERIg1pfIRIQxofIdKQxkf/A70XNa8KZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iagozNzg4CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagoxNSAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MjA0KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDE2CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA1NjMzIDAwMDAwIG4gCjAwMDAwMDA2NDYgMDAwMDAgbiAKMDAwMDAwMDY2NyAwMDAwMCBuIAowMDAwMDAwNzY2IDAwMDAwIG4gCjAwMDAwMDA3ODcgMDAwMDAgbiAKMDAwMDAwMDgwOCAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzOTYgMDAwMDAgbiAKMDAwMDAwMDYyNiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA2MDYgMDAwMDAgbiAKMDAwMDAwMDg0MCAwMDAwMCBuIAowMDAwMDA1NjEyIDAwMDAwIG4gCjAwMDAwMDU2OTMgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAxNSAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMTYgPj4Kc3RhcnR4cmVmCjU4NTAKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDYzMS4wNDYyNSA5NS45NCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjUEKgzAURPf/FHOCJD8mX11ahNCl3fQAIbSVamkFe/1+XUgXAzPDDI8xku0YtwUOo+oLRoLty/rI5ZJOyAs57SeSio0L4qOm519qo2mDNu5wd6KZ3qiN3yUsRvZpwyYGfAqumGE7v2FZsaxYh6Qf4RobrvHHPU+wZ0b/wkAD/QBzWSXbCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTM1CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyNDQgKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/b29vX19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Ofn5+bm5uXl5eTk5OLi4uHh4eDg4N/f397e3t3d3dzc3Nvb29ra2tnZ2djY2NfX19bW1tXV1dTU1NPT09LS0tHR0dDQ0M/Pz87Ozs3NzczMzMvLy8rKysnJycjIyMfHx8bGxsXFxcTExMPDw8LCwsHBwcDAwL+/v76+vr29vby8vLu7u7q6urm5ubi4uLe3t7a2trW1tbS0tLOzs7KysrGxsbCwsK+vr66urq2traysrKurq6qqqqmpqaioqKenp6ampqWlpaSkpKOjo6KioqGhoaCgoJ+fn56enp2dnZycnJubm5iYmJeXl5aWlpWVlZSUlJOTk5GRkY+Pj46Ojo2NjYyMjIuLi4qKiomJiYiIiIeHh4aGhoWFhYSEhIKCgoGBgYCAgH9/f35+fn19fXx8fHt7e3p6enl5eXh4eHd3d3Z2dnV1dXR0dHNzc3JycnFxcXBwcG9vb25ubm1tbWxsbGtra2pqamlpaWhoaGdnZ2ZmZmVlZWRkZGJiYmFhYWBgYF5eXl1dXVxcXFxcXFtbW1paWllZWVhYWFdXV1ZWVlVVVVRUVFNTU1JSUlFRUVBQUE9PT05OTk1NTUxMTEtLS0pKSklJSUhISEdHR0ZGRkVFRURERENDQ0JCQkFBQUBAQD8/Pz4+Pjw8PDs7Ozo6Ojk5OTg4ODc3NzU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLi0tLSwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExISEhERERAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNjE3IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCA4MiAvTGVuZ3RoIDE0IDAgUiAvU3VidHlwZSAvSW1hZ2UKL1R5cGUgL1hPYmplY3QgL1dpZHRoIDYxNyA+PgpzdHJlYW0KeJztnftfFNUfxtkdWBa5ygokCkJeuHgFK9HyAoiE4i0NkdJKSuNiQhokZEkiEamYCigmkXxDy0pJBCQM0wgSBDMRUZGiMBIQhPkDvp85O8PC7szszLJhtuf5wZfM7Dw8c857ds45c85gtBULayhk9LADYBmIMGlYQyNMGtbQCJOGNTRSkkbqR1v1bqZPLxxMrJdezTBpupnp08swgmHSdDPTp5dhBMOk6WamTy/DCIZJ081Mn16GEQyTppuZPr0e4WC3b8+ePfvMGSFmmDTdzPTp9QgHw6SxeeFgYr0wabp54WBIBw8eHDt23759gw927969+fMlEslzzwkJhknTzUyfXpg00Xr0K/TcuXNffinETMtnurvz8/PDw3fs2NHTM9hgGRkZ1tZGRkYffDDYYD09PcnJcrlcKo2JiRHgxWvW3Ozn5yehpI1aTBqLMGmCg2HSOL0waZg0bjO+D9y9e7ey8sKFC7m5b7zxxoYNBQUFgwh240Z8fLyPj6mpqYnJ+PHj164tKytraWnRKVhxcfH8+QQlExOTuLhTp06dP9/d3a1bsP37ZTIZQUil0pMn+T4nJBiAL6UkkZSWlgrw4jULDkaY2dnZ3b8vJBgmTSlMmthg+ictOzt73DgUb9myZQcPQv0OIt4nnwQEBEgkcC8YMyY0NPToUS3x+LzefvttVANSKapVhUKRl5fX0aFbsJAQor8ZgmThwoU6BQsODh7oRZiZRUdHt7e36xBs8WLkZGxsXF198uTJ4OCkpKSzZ8/qFGwxZQZ0BAb29vbyfVBIMDc3RFpmZia/E4lJGyBMmthgmDTdgmHSxAbTK2mtra1hYdBXQaVFl9vEiRMLCwt1iNfW9sILL5ia0l5MJaSnp/PF49gHBZWUpBZMqfJy0cHQYOawYWrBQE5OTs3N4oKBLl2CpotaMGNj+GfLli1ig4GWLEGZEGxAvzEluKhqakQHI8lFixYhbIOC+D4lMBgizdLy+PHjwswwaSQmjUWYNKUwaf8x0qAGCMLe3j4jA/WPmXIzNzfnaShwme3ahYocaUCFHjlyhNOMw6uqqsrKijle5QW2UVEdLE013mC3bk2fPp2GTBVM+b+KCnHBQO++y2RCtermZm1tjRw3btwoLhhJ1tbWenmhJFBoOTmdnZ03b1pYWIDZd9+JDtbbu2DBAqlUJpPl53N/SlgwkibNz0+rE4lJY4RJY9FQktbY2Dhu3DiC+P7770ny6tWreZToEUnCxeXPP/8UEa+ystLVFY6bMOGVV14JDISv8bCwkSNHIi9LS8u6Oo54bF7Nzc1z5sxBvUO5HA4ODIROokLBAMLWMeMtt5QUBi1z8507d0ZGTpo0idny+ecigoFKSkosLdGhw4cPT0+HzmJNzWuvvYa82BoKWip09erVCFcXl2vXrnV1wZaOjhEjRsCWCxfEBQPFxCCvJ554gvszQoOlpqYaGwNpkZECvDBptDBpLMKkkZi0/xxpUDCQKzZ2wBO77m74Hehc169fLyLe7t27qYo0P3BAta2+vt7XV1mh69ZxxGPzKioqQk2XWbNmqTbeujVq1Chk5u/vLyIYXE+oGSWXQ13SLby///77ySeV4E6fLiIYaOzYsei4tWt/+eUXetvrr7+OtrHNn+Cr0IyMDNSkdXR0LClhNl68qGz2sVznvMHq6upcXQENqRTOmOMzQoPBpe7t7Y3GbW/fFuCFSaOESWPX0JFWV6dQKOD+phELqgGKUyIJCQkRGu+nn36ytbWVyT7++GO1PdXV6DGXhG2AhyNYWxvcK6Gop0xpampSbe7qeumll9DdwcPDQ2gwkrp1Ku+dnp499Eyy3t6KigrUPQaZmAgORpItLS1wESI7ehFHYWHhrl3GtJmnp6eIYCQZGhqKjps5c2Z9Pfz8xRdwb3dxUXaKt20TEYxElzqFGei3334jyc8++ywr67333ktNhcqBKhUVLDc3V0ILkZaf/z9KX331Fccvx6SRmDRMGibNUEiLiiLQkyeNHcXFxag5lJOTIzReYmIiHGJjo7a5t7c3JkbZTouI4Iinufn8eYKaG2GGHjvt3g2d4t5e+Cc8nBkqZetbcZdbfHw8QQwbNmzOHLh+EhKgcRoSwgzfoidAgoOR5A8//EAQTBcYgA8IsLCwkPbJwcFBRDCSBAd0HFSoi4uvr6+JicpLOnq0iGCgyZMnIyeJBMpn6VKIiH5AXl5eXqKC+fj4MKQFBS1fvpz2ksvl8ANbbxSTRmLSHj5pzz9PsD3+7ulZtWoV7Jk2rQsN7wiKt2nTJjhE7W7b07Nt2zZUK1DPxcUc8TQ3l5XBIVOnTu3pKSgomDABSKmuhtOkMQPt2LFDaDDQli1bCALux+7uKkgIlZmdneBgqH+H+p7SgY+0VF52164JDkbDwSUbm6+//lpoMBDwRcOh8mBwgaZNUZGIYP1I05SjY4XGIzxMGolJw6Rh0gyCtNbWVicnKCjNX/L++8oiPHaMLQJHvHnz5slkssZG6N1FRNymB/t+/XX06NHIi2NRE8eppqbCIbGxsadOSal5Ce7u7osWESqtWXP58mWhwUCnT5+2sWH40tSqVYKDIcXFxXHCAdWQkiI4GElOmzYNHcVh9tRTT4kINnv2bA3SZJSUwfz9Ozs7hQbrR5qDw8j+srKyMjJydXVVG9DFpJGYNEwaJs0gSLtz5w4q5wEjGS0tGzdupBa5EqwM8sQbhqZO+/hcv349Pj4xMbGrKzMz09NTStUuZLvN/vCM41S3bjVWydv7o48+cnbuh0ZtrYhgoPLycjT1mp6kOQCzmTNnsoxp8k4yQa8iQAfDidnYgMOcOS+++OLatVANAQEBGnPPuYN1dzs4OCAgoBm1fTtcEaWlhYWF587BPytXrpRKnZ2dhQYDQSMRoeHtffXq1fb2nr6XOBw4cAANLd+7d09YMBVpc+fO/euvAXuKi4vt7WEPGmlWM8OkYdIeImkk1feEc2SeODU05ObmGhkx37oRbAOtPPGio6PhKFPTvLy80tKJEyc+8wz6EkcV4ujoyO7FVW7l5UTfJHFjDw+FQoGcqLccmEDtXrkiIhhJLxpVkYZyjhgxYu9esONYjMHptXfvXqaIYmI+/fTTzZtV+6qQNFZZcgfLzGS8vvnmG7V9RUVFpqaBgYFCgzU1NQ0fPhzRoTF57+bNm9bW1s7Ofw2ERghpKRqNgd9//93d3d7eXu0ZJiaNxKRh0jBphkLaW29B4VtYWMyj5OBAUM8tATw0Q1Asad9++y0AZWwMdtbWUhqxFSuSk5MJAhoh7F7cpCGvfi0qakFUVFoa8AGNIbFvS4Bz6u81ZsyY6uq2trbDhwlq/RHLAiSeCp01axZBD9QWFICJiqvk5ISEhB6WV6lxB0NropCys7P77ygpKRk1apSLy8WLF4UGg0scXdvwz6ZNA/Y0NHh5eUFp7t+v9v4EIaT5+/v3xxNY9fPzs7fXLDZMGolJe/ik3b//6quvKuh1IHI5mKLvXcTcoUOH2CNwx4MvWrrbikhDU8OnTp1KEM8++yy7F3e5oeP6aeRI2IjG/9zc3MQGY06RFpw0GntCP1RWVooKZmpqKu17/AR9ukuXuru79+1bsmSJmRnAkZWVJSKYkxMz9gV90Fu3EAh1dXUREY899hhkmzRJRLBjx44xZjIZ9D3RxtbW9PR0NF0EfoHGRcBH2po1a5jxtBMn0CxBuIjKyhYsWABbWGYHYtIoYdIwaZg0wyAN6e7du2iVJ700586dO7a2UDXXr19n/zx3vF5qumJsbCyqwfXrb1ICLqTStLQ0jl/O6VVTUwPVqJp8oVAsX76cGjFlndKnJdjOnTvRZHFacrnczo5pBRZpzHDgDwbNRTs7O1ShCDe5HJ0io01qbST+YMnJ0GZUHbxs2bLQUOjVMT/PmCEiWG1tbXh4uJkZg9vmzZGRkfRMEdi2YsUKEcEQFsH0u0clEl9f3w0b4AJl2EtO5giGScOk/QtIU9PRo0cJIpJn2TJfPJJ+oS/cRv74g6QmZkupRRxsS+H7mXGH2bwZVai1tbW5uQu1jgPQ+JLrDe+8wTo733zzzf5P2Jnu8enTp8UGu3Llir+/VEPIU+xbrRobG9FjJ03B7Y4tmpYSO3SI6YD2f8weFhbWzPLuLi1V2dHRMX68ZKDgKoX78YMHHMEwaZg0TBomzVBJmzt3rlQKzTbOD2iJl5CQQBBo6RI0HNDcAq4h4D4z7t3UrGzQjBkzqNWnEvTOYY7OotZg2dnZfUgoe46opaZ64YGYYIWFJgNWMSmpffzxx9nePawlWENDAxrrpiWhZrFPnjy5qqpKh2APHixdupTxoeTn58ffSubl4sSJhQsX0k4ymezIkXzO17Jh0hhh0sQGI/9J0urrbW1tCWIQpKHVIc9Rf6MPelKo/Ps/g2Y1494dGUlQT8WhVul7HnTMatje/iogWEFBgdqNkyACAgLus//xEK0l9uGHHzJmNGnOzrViJ84xOnNm9erVNjbQ5ffwgKs0KOgM5x9u1erV3t5+/HhKSkpiYk5Oztmzf6BmDI+XwG8grcKkMcKk6RBMhDBpjDBpOgQTIZGkXbqEim8QpKWlpclkVlZWQUHALKqBw4e1xOPenZXlQD0aM+6bmx0XF6drsJaWlu3bZczMdYrZxYtv3LihWzA0TL1nD/Rno6Kg6VJRcfnyZbVJ0IKDqRy1fkRIMBHCpPUJkyY+mAhh0vqESRMfTIQebdJAP/+8cuXKKVNQI8bLy0vj7QHqZnxeTU1NL7/MkBYdrbZgUWQwkkxKSnrnHXB6+mmO552Cg4mQ/itUn16YNKUwaWxm+vR6SKR1dq5bt87c/McffxxsvM7O3NzcrVv37NnD90fKRZwq3K4G/+fOBevfXaH69MKkqQuTpjLTpxcmTV2YNJWZPr0eEmlCHQ2j3PTpZRjBMGm6menTyzCCYdJ0M9Onl2EEw6TpZqZPL8MIhknTzUyfXoYRTEkaFtY/LUwa1tAIk4Y1NMKkYQ2NMGlYQ6P/A1PrMEwKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iagozNzU1CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagoxNSAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MjA0KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDE2CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA1NjEyIDAwMDAwIG4gCjAwMDAwMDA2NDYgMDAwMDAgbiAKMDAwMDAwMDY2NyAwMDAwMCBuIAowMDAwMDAwNzY2IDAwMDAwIG4gCjAwMDAwMDA3ODcgMDAwMDAgbiAKMDAwMDAwMDgwOCAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzOTYgMDAwMDAgbiAKMDAwMDAwMDYyNiAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA2MDYgMDAwMDAgbiAKMDAwMDAwMDg0MCAwMDAwMCBuIAowMDAwMDA1NTkxIDAwMDAwIG4gCjAwMDAwMDU2NzIgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAxNSAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMTYgPj4Kc3RhcnR4cmVmCjU4MjkKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["pl.seed_everything(42)\n", "for i in range(2):\n", " interpolate(flow_dict[\"vardeq\"][\"model\"], exmp_imgs[2 * i], exmp_imgs[2 * i + 1])"]}, {"cell_type": "code", "execution_count": 31, "id": "6a103456", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:04.481539Z", "iopub.status.busy": "2021-09-16T12:42:04.481065Z", "iopub.status.idle": "2021-09-16T12:42:04.697014Z", "shell.execute_reply": "2021-09-16T12:42:04.696379Z"}, "papermill": {"duration": 0.278454, "end_time": "2021-09-16T12:42:04.697129", "exception": false, "start_time": "2021-09-16T12:42:04.418675", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDYzMS4wNDYyNSA5NS45NCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjUEKgzAURPf/FHOCJD8mX11ahNCl3fQAIbSVamkFe/1+XUgXAzPDDI8xku0YtwUOo+oLRoLty/rI5ZJOyAs57SeSio0L4qOm519qo2mDNu5wd6KZ3qiN3yUsRvZpwyYGfAqumGE7v2FZsaxYh6Qf4RobrvHHPU+wZ0b/wkAD/QBzWSXbCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTM1CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyNDMgKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/b29vX19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Obm5uXl5eTk5OPj4+Li4uHh4eDg4N/f397e3t3d3dzc3Nvb29ra2tnZ2djY2NfX19bW1tXV1dTU1NPT09LS0tHR0dDQ0M/Pz87Ozs3NzczMzMvLy8rKysnJycjIyMbGxsXFxcTExMPDw8LCwsHBwcDAwL+/v7y8vLu7u7q6urm5ubi4uLe3t7a2trW1tbS0tLOzs7KysrGxsbCwsK+vr66urq2traysrKurq6qqqqmpqaioqKenp6ampqWlpaOjo6KioqGhoaCgoJ+fn56enp2dnZycnJubm5qampmZmZiYmJeXl5aWlpWVlZSUlJOTk5KSkpGRkZCQkI+Pj46Ojo2NjYyMjIuLi4qKiomJiYiIiIeHh4aGhoWFhYSEhIODg4KCgoGBgYCAgH9/f35+fnx8fHt7e3p6enl5eXh4eHd3d3Z2dnV1dXR0dHNzc3JycnFxcXBwcG9vb25ubm1tbWxsbGtra2pqamlpaWhoaGdnZ2ZmZmVlZWRkZGNjY2JiYmFhYWBgYF9fX15eXl1dXVtbW1paWlhYWFZWVlVVVVRUVFNTU1JSUlFRUVBQUE5OTk1NTUxMTEtLS0pKSklJSUhISEdHR0ZGRkVFRURERENDQ0JCQkFBQUBAQD8/Pz4+Pjw8PDs7Ozo6Ojk5OTg4ODc3NzY2NjU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLi0tLSwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx4eHhwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExISEhERERAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNjE3IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCA4MiAvTGVuZ3RoIDE0IDAgUiAvU3VidHlwZSAvSW1hZ2UKL1R5cGUgL1hPYmplY3QgL1dpZHRoIDYxNyA+PgpzdHJlYW0KeJztnftDFEUAx7k9EA8QxAMULRFSHgmcYFBJoWla9kYjClNSICQDxAeaJZKmGEKBWVIYBYmQEEk+4qUCKRYEYiQoIBwImoYI3D/Q3LDb3j5vd7nIuvn+gNw+vnx35rO3M7uzo8k2JKSxkMm/HQDJSIRIQxobIdKQxkaINKSx0QhpGsNom8HNDOmFgon1MqgZIk2amSG9jCMYIk2amSG9jCMYIk2amSG9jCMYIk2amSG9jCMYIk2amSG9jCMYIk2amSG9jCMYIk2amSG9jCMYIk2amSG9xJkNDQ0dPnw4KemPP/4YZbDBwcHU1OjoaLlcHho6PDw8qmCtrampqSYmJhg2bdq0n39mBkOkSTMzpBciTbT+TxWqx0zE9levXm1t7ejoaG5uHmWw27dv19eHh4djGCaX79y5c3BwFMF27QoJCZHLMVxffPGF9GBbtmxZsAAjtWoVMxgiTZqZiO0RaYi0UZiJ2B6RhkjjVkiIvb19bGzs6dOjCNbT09nZ+f77CoUCw2QymaenZ2+vtGDd3fHx8enprq6umK5++010sKKisrKyzMyamhoLC4oXwE5ssLa2toqKoKAgLy9zc3OKWUAAMxgijVWItDEk7fPPPwfF89xzzDW0boqQeGfPnvX1Bd/3jBWDtCsAzYxjHfi2bmujLbty5UpmZubbb7M56ik3AMSRI+D6VliYnp6+adOzzz5rZTVSZK+9Ji6YRntEXV1dn30WFxc3eTK8zOnUwJ494oLBwwoMDHRzA/1DmQyj6cwZMcEgGkuWEDtTgmHYunXrxAVrb3/88cfpiXAFBzODIdI0iDREGiLNCEjr7PT29gb7bN5MWXz5MsBlsrYIKyoqGDtxxLt586avry+GTZ8+naTg1q38/HwfHysrK3CRP3eOmYAj2MWLERERpqamKlV7eztcMjx8/PjxxETQYIBHuXXrVqHBgNRqdWDgnDlzLCxiYmKYdenmJjgYVGVlpb8/WZeyvx3lUKGhgoNBpaamKpWEHTNdSYngYECHDk2YMIEDDZmsoKBARLDS0lJ7e6aNHMeXVp+INA0iDZGGSNPKGEhraRkppZwc8KG396effiooCAgIsLUlzD788EPGThzxqqur4S5z5869fh18iIpavnw5bHoQCQ8cYCbgCFZePrKfpSVorCUkbNiw4bHH4H0EQk5OTkKDAe3duxcyQLBAa724uwsOptH8+uuvDg4O0AH/QZCGf5Tv3y842PBwXV3dfffdR6tD+NukSZNcXfPy8oaGhAVraWn56quvbGyYaJiagm+O3Nzc8vIhqhlHMNAIvnIFnNPjxo1jeE2bNq2pqen330+fPk1ryCPSNIi0f5+0xkYzMzNo4OLi4uRka2sLC434Bjcz27FjB2MnjnglJSX4YZlaWxPlTrkWnDzJTMARDPcC+zO5gAKVLTQYkIWFBfQxM4P/gIgUTw8PwcHgCmgi02YDP8aPz8rKKirCcIrt7e07OwUHq66mgE+cDLNmffLJJwMDAxx/n83r2rWwsDDKdRz+O3t2SkoKrXGkN1hfXx+zxDHMx8fnnXd6GfelKWaINEQapxciDZH2PyJNowF9T1DuGFGvMheX+fPnb9gw8jkyUkS8u3fvJiYm6iZzdXUNCgItPfghMDDw9m0OM+biO3ceffRRog7gDzMzlUoVEDDibG6enJwsNBgQOIXw8ocHC23Jw8aiogQH02hAQ4UoLJkM9LYrKmprax95hPBau3atiGDEveO/W3og2IMPPvjKKxyYcQfr6tI5I+FvSmVaWlpVFYcNX7AbN26QtQhtPT09d+++fv26XjNEGiKN02uMSNPAG2FRUS+99FJc3KVLl+CtsMJCYA/QuHlTRDwNhK25OSYmJiwMXHUbG69evarRHDx4UC6fOHHioUOHOM3YVoA+WXBwsFxuZ2fn7++rfcjV39+/YsXI0UdHiwu2detWskIhItoByiMCXTwRN/o0mqVLlxKkkcThlWxlZXWG9vCIJ1hkZCQRgmxlyGSgFnp7QWvmN9ogDp5goNu5fLnOOT5u3JQpU9zd2bLoCQaqsKJi7ty5pJe1tXVa2sWLF/mdNIg0KEQauxBpGkTa/5E0mr799lv4OOPIkSNC43GroAB0+7RjRVgGi5Bm3PufOgVxhSKq1t/f/9IlccFAhykkBNO2OUC7VKEYP3483lMEjbX4+HhRwXbt2kVUgqmpTLdz7eeXnp4uOFhTE2CcaFpNnz5v3jw7O/j7/fffv2wZKLYXXnhBaDCi7Qjl7u6end3Y2KhUenl5rVwJ+p7r1z/88MM7d+7s6dEb7MKFC2T7Fcjb+9y5c8PDQUFB67SKjAwPD1+0aJGPz6efflpZyTRDpCHS7j3S1q9fj2EPPPAA55gyMaTFx4O0KtXly5c5NhDhlZ2dDW/+ZWVlSQh2925DQ8PJk6AGNJqMjAy80wfqoru7W1Swvr6+J554ApL6wQeZmZnkRXThQo6/zRrs7FkM7yQC3L/5RqO9JUa5mILFbEfK9Gpvt4Z3MKHeeKNHCxQoJNqdSPBh3rympib+YDNnzsR0depUWVlZVpYVXl6U25A2Nlu2bKGZIdIQaYg0RJqxkgZ6oTIZaIxwbiCGNC8vkOipp7g3EOzV1+fh4YFhoMEB+qCjDFZaWgoHhMnlhYWFEoIdO3YsJQXsP3XqkiVL8GYNIG3lSj4vull/P1Fza9asaW0FPc2jR0EDkvIQz84OnF56g+Xm6tyFdnVdvXr1zJn29vZsD/FAT173RTBmMNB2pOygVJppH+ExnXDWbIqLKWaINIoQaZzBEGmINO5g/03S0tMtLS3l8qKiIs5NBFco6FdbWICg1BHFLGZ6nYaHh9esgceWAwdtSg82MDBQWztjxgxo9vTTkoOp1W5ubpSHpxiWkHDnzh3BwchWGZSJiQn58JOs2rVr1+odCXnwIAcIrPL35wsGbwCJEdlSQ6RRhEi7J0g7f/68g4OeGhBMWn4+HPumUqn0mun1AhcEeFienn/++ecogg0MREdHk6X08ceSgy1bRtqA2sHHVMfGxrIFZA3W3k6vNMiXs3NwcHBOzltvvUUsVqv1BAsLYwJAwOrouGrVqo0bddboDsZjBmMdDEjzpIh8yIhI0xEiDZHGL0QaX7D/IGn79u2De/M00rjKjSltMwZ0on755Re9Znqc0tIs8ClzMjJGEQw09vbudYCnEq6ICGnBjh49CkdwaAdNjIuIaGtr6+kpKCiAHdBNmzYJC9bcTB+4rlDk5+c3NMC1oHtLLMZfeOUOpnvy6MjJ6bvvvoNjML7/Xuf5rG5nlhmMGDNKkVK5ePHiGTOioqLCw4l7uFDOzs5dXRQzRJoGkXbvkNbX5+fnB4dhs49L4y03mn744QcIh3YyMa43HEgzPcHw6REnTJjAMt+i8GDl5eVKJShya+spU6bANxk5b4DxBRscHHz11VdxRBISEuDCjo7nn38eLklKShIWrLaWHL0+YpaZCVeAEqupmT17NlGdtFlwmF4ZGZQuMJS3t/eZM6A1dPhwWFgY2Z80Nzc/dYovmLu7O5M0R8cDBw68/HJAQIDuK7fgvALeNDNEmgaRhkhDpBkbacXFcO+vv/6afzv9pHV3h4aGwpYBbfYSDjO+DSorK+FhyeVwGMYognl4eMBMmZkNDQ2QtPfekxAMLMbwfuK2bXCSuerqajiJGqzI1tZWYcEKChQKBdlUA//m5anV6s2bp06dSow9l8tffPFFvcEGBkLgOE9dTZ48eeFCOoB2dmVlZfzBqvE3UGl2zGXgc0oK5cEgIo0QIu1eIW3RIkzbYeEeS8ZXbhSVl8MwoMhaWvT8Ub1e8IqOaUfM0cZWiQx24gSoWJkMvlPy+uuvw35iXJyEYEQgsH9ODuhZr17t6OgIl0yatHv3bhHBiHdfSJFojFSs0BdRs7OzCRA474cBZhnvazKD1dTUkDNsccvBAZz3bI/JEGmINEQaIs24SLt16xZ8NefJJ/m24ohHV10djMM2+RqrGd8GoJ8IJ1phmZxSVLDkZJgpMTExJcXS0hLO5cA+rltPMLAf0WaB89PgZa9SqdjfaOUOlp+fr0MVlThnZ+cM9vvUbF719fVs86YR7ra28+fPZ3t9lDXYyZOzZs3iwwycqHV1HMEQaYi0e4W0L7/8Eo7IvHCBbyvueBTt3w/yPPQQoFeYGffq48fhyxjbt2+nfVWLC9bR0QHntcKwiRMn4vN4geuEpGAjpBE3wnBIPDz47htyBBsaGoJv9BBzWeF2K1asaG6+ceOGuGAdHS4uLtjfE1MSP9zdwWUOf8IlONiPP/5I3oIj0mEYALCq6sSJE3yj8BBpiDQRwRBpkoMh0kQF+0dJi42Nlcuf4nu3RF88Qnl5ebAHZWcnwEsfafhUJLm5uaMK1t/fv2ABwYVMplQq9+wZ4meX00utVhPtGDMz0ICcMyc4OJi/j80d7Nq1qqqqmJiY7dtB7ZaUFBcXp6RwT53GH0yC9H9piDVDpCHSxAWTaoZIQ6SJCybVTB9px47BqecVCsXSpT2MGWnExSOmILWxERyPe7WPD2wpbNy4UagXl9lHH0HOnnnmmaQktv9XWGSw8+dLS0vr6y+xz3okJphYIdIIIdLEBROr/zZpycn4RIkmvr4cM6IIjvfmm2/CIROOjoLjca/28wN8vPsuR29HXDBxurcr1JBeiDStEGnsZob0QqRphUhjNzOk1xj3PcU6Gke5GdLLOIIh0qSZGdLLOIIh0qSZGdLLOIIh0qSZGdLLOIIh0qSZGdLLOIKNkIaE9E8LkYY0NkKkIY2NEGlIYyNEGtLY6C/m5CTmCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMzYwNgplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDkxNjE0NDIwNCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjQuMykgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNTQ1NyAwMDAwMCBuIAowMDAwMDAwNjQ2IDAwMDAwIG4gCjAwMDAwMDA2NjcgMDAwMDAgbiAKMDAwMDAwMDc2NiAwMDAwMCBuIAowMDAwMDAwNzg3IDAwMDAwIG4gCjAwMDAwMDA4MDggMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk2IDAwMDAwIG4gCjAwMDAwMDA2MjYgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNjA2IDAwMDAwIG4gCjAwMDAwMDA4NDAgMDAwMDAgbiAKMDAwMDAwNTQzNiAwMDAwMCBuIAowMDAwMDA1NTE3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo1Njc0CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDYzMS4wNDYyNSA5NS45NCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjUEKgzAURPf/FHOCJD8mX11ahNCl3fQAIbSVamkFe/1+XUgXAzPDDI8xku0YtwUOo+oLRoLty/rI5ZJOyAs57SeSio0L4qOm519qo2mDNu5wd6KZ3qiN3yUsRvZpwyYGfAqumGE7v2FZsaxYh6Qf4RobrvHHPU+wZ0b/wkAD/QBzWSXbCmVuZHN0cmVhbQplbmRvYmoKMTIgMCBvYmoKMTM1CmVuZG9iagoxMCAwIG9iagpbIF0KZW5kb2JqCjMgMCBvYmoKPDwgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0kxIDEzIDAgUiA+PgplbmRvYmoKMTMgMCBvYmoKPDwgL0JpdHNQZXJDb21wb25lbnQgOAovQ29sb3JTcGFjZSBbL0luZGV4ZWQgL0RldmljZVJHQiAyMzcgKP////7+/v39/fz8/Pv7+/r6+vn5+fj4+Pf39/b29vX19fT09PPz8/Ly8vHx8fDw8O/v7+7u7u3t7ezs7Ovr6+rq6unp6ejo6Ofn5+bm5uXl5eTk5OPj4+Li4uHh4eDg4N/f397e3t3d3dzc3Nvb29ra2tnZ2djY2NfX19bW1tXV1dTU1NPT09LS0tHR0c/Pz87Ozs3NzczMzMvLy8rKysnJycfHx8bGxsTExMPDw8LCwsHBwcDAwL+/v76+vr29vby8vLu7u7q6urm5ubi4uLe3t7a2trW1tbS0tLOzs7KysrGxsbCwsK6urq2traysrKurq6mpqaioqKenp6ampqWlpaSkpKOjo6KioqGhoaCgoJ+fn52dnZycnJubm5qampmZmZiYmJaWlpSUlJOTk5KSkpGRkZCQkI+Pj46Ojo2NjYyMjIuLi4qKiomJiYiIiIeHh4aGhoWFhYSEhIODg4KCgoGBgYCAgH9/f35+fn19fXp6enh4eHd3d3Z2dnV1dXR0dHNzc3FxcXBwcG9vb25ubm1tbWtra2pqamlpaWhoaGdnZ2ZmZmVlZWNjY2JiYmFhYWBgYF9fX11dXVxcXFxcXFtbW1paWllZWVhYWFdXV1ZWVlVVVVRUVFNTU1FRUVBQUE5OTk1NTUxMTEtLS0pKSkhISEdHR0ZGRkVFRURERENDQ0JCQkFBQUBAQD8/Pz4+Pj09PTw8PDs7Ozo6Ojk5OTg4ODc3NzY2NjU1NTQ0NDMzMzIyMjExMTAwMC8vLy4uLi0tLSwsLCsrKyoqKlwpXClcKVwoXChcKCcnJyYmJiUlJSQkJCMjIyIiIiEhISAgIB8fHx4eHh0dHRwcHBsbGxoaGhkZGRgYGBcXFxYWFhUVFRQUFBMTExISEhERERAQEA8PDw4ODlxyXHJccgwMDAsLC1xuXG5cbgkJCQgICAcHBwYGBgUFBQQEBAMDAwICAgEBAQAAACldCi9EZWNvZGVQYXJtcyA8PCAvQ29sb3JzIDEgL0NvbHVtbnMgNjE3IC9QcmVkaWN0b3IgMTAgPj4KL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0hlaWdodCA4MiAvTGVuZ3RoIDE0IDAgUiAvU3VidHlwZSAvSW1hZ2UKL1R5cGUgL1hPYmplY3QgL1dpZHRoIDYxNyA+PgpzdHJlYW0KeJztnftDFFUbx3dndpE7LrIqqYCmqIlFEJKGEqFZZBeRLpqZiiFd1EohSDKhEg210kzRIrULKmQFSilm4SVveIHwrimIiMj8D+8zz+4K7M7MzlkmrHef5wd0z85+93PO+Q7nPGfODLp3KCi6InR3GoDCTYKcRtE1QU6j6Jogp1F0TVicJmgT72gupqUWgbFqaSpGTnNNTEst9wAjp7kmpqWWe4CR01wT01LLPcDIaa6JaanlHmDkNNfEtNTqWrBLly4VFBw5ckQTsHPn4uLiUlPVgJHTXBPTUoucxhzUoS5pkdOYgzrUJS3nYqGher3+k0/UiDk5Zs+e+++/n+MOHz7cebCrV68mJHAcd+yYGjBymmtiWmqR05jjv96hDGJOj9q8efNPP127dq2zYI2NcAAnxsqVnQW7cOFCWppejLq6us6DJScno9iuXWrAyGmuiTk9ipxmJ0ZOc03M6VHkNDsxclq7qK5uaGhobVUjpnTAjh07eB7NsWzZss6CnTiBSt7e3tu3dxbs+vXr4eFgDQ+Pn3/+ubNg1dVos4iIiJs31YCR09oFOY0BTHunwds+PjzP63QTJ058771Dhw51Am/ePF9fX47z8/NLT6+srLx+3Qmektb777/v6Qlg9967evXqqiondXAitn17bGwsz/fo0WPx4gMHDjgVUzoAszvs0Ly8PKUD1YCNHo1OgyoqK6kBS0lJ0etB7OOP1Wkpig0dik775ptv1ImR0yxBTmMFI6e5BkZOYwXT2mnz5nXr1o2zBPSqwQAd8vnnn7uEN2XKFC8vrk2M5yMiiouLlfDkwU6dCggIaFMyGu++++6SkhKXwCDCwznRHfgD5GJj//zzT9fABCEsLAzB7rnnxo0bSgeqAQsORiZIZZWV1IDBeaTXe3l5XbyoTktRrH9/nU5nNO7bt0+dGDnNEuQ0VjBymgtgAjmNHUxTp0Eu7OExfPjwJUuysrKsXQoBGXd5OTPeH3/4+/sbjaNGjerdm78dUG+ZjFtJ6+rVq2PHwqdHjkxPTw8KQiXoEJgDlpUxgwnCfffdx3EGgyEqKiYmxmBAsZCQkP3797OCQcyZA0ocFxQUZD0ht2zZkpgoc7ATsC+//NLTEyqWktLc3Cz/lerALl4Ep3HciBEjnCqpcVpEBMzS4uJUaJHTbgc5jRVM0NJp9fX10AMeHmXWzquoeP311x9+2GIQk0lqh5MiXmEhJ1aztrZ23LgPPvggL++hhx4yGqFXhgwZIosnozV37lyeh76sqWltbT17dvr06dax3TB/vqyWnNiqVd27d9fpkpOTb92Clxs25Ofni9kjFx8fzwoGYTLh0Dl16lSUE4THHnvswQe3bdvGCgZV69evn14Pg5T8tzGAiacTtJLs3Ec1GEROTg6m19OmqQYjp5HTWMEEcho57T/ptKKiIiCbNatDYXNzRkZGsJgPSa3gyuNlZmYaDMOGDfv99w7FX3yBcxrJvXSyWjBJ8/Pz4/nnnnuuffGkSZPAbP37NzQ0MIBBBAYCwcCBf/31l7UAujgjw3IKSM2OlLTmz5+PPRARcf32qjQk72bzunXrpI5XAtu6davRaOS4AQMGyHwbA9j58+d79gQwT08GLRmxS5cujRw5Ek+B2lrVYuQ0chormLZOa2mBnoM6nj1r90Z1dfWjjz5qMBw9elQtHnwkPDxcp4Mx0+6d7dtxsUh//LgMnmNxc/PMmTPhU8OGgePav1FVVYWXR6TWsOTb7dtvv9XpQC4zs0PxM89Ystnb7nMOJghnzpyBcRjHzshILIE++f57nfgFIFdTU8MAhqcOVmj8+PH4uqWloKBgypTZs2eXlpZK7ThRchpMdrClu3fHl42NjStWTJs2bfny3bt3Sx2vBLZ+/Xq9NbDjDh6E82j1avnrlOQ0gZxGTiOnuYvTVq2Cthk1apTDG3v37sW2ZHDahg0bQCwp6ZY1GWuLwkJLh548KYPnWFxaavmI3YUncWqVge9cvnxZLRgEJMAcB/U8f76tDBrf39/yLadPqwYThPLyctu1trFjsaSiosJ6wQx+SK1QK3WoTevJJ5/E13PmYOdaCl94gQEMArJYdFqfPjdv3ty2DTStYlDPWXaTcWdgY8aMwQ/DKZSRERMTYzLhS19f3+jo6FdflQEjp5HT7qzT+vaFz77xxht2xdeujR49GuB8fI5LjHgyeCUlJdA2WVl2xU1NTdnZoBUZGSl1yVcG7KWX8AJ4rF1xTY3ZbMar2lLXbWTAysrKcPS224Sdn5+Pi4Y9e/b8+2/VYIIAzWVzx0cfYcnbb7/d5jSpJTWlDsXEEyItLU1s+GvWgRnleLP5xIkTasEEvOqPdoiJgRGd522jH8r5+PhIa8mIQXPpZUOs+0dSYuQ0cho5jZzmDk4zmwMCAhyLN22yLLXm50shyOBFRUVB2zgs8uXl5fG8h4fH0qVLZcUci198EaqT3/H7Gxoa0tI43I4hvXFZBmzz5s28iADvtd3/+Msvffr0wStbMPNjABMEmPzYXHXlCpZ4WffjoXHZrt8dPQoGwL47Kc5iIW9HmcmTL1y4ABMljtu0aZNaMIghQ4agmJ/f4cOHk5Lgf337xsfHz5oVEhICLxITW1pa1IEJwiuvvGLz1cCBoaGhYWEwCzSZevToAfXV64ODg+2exUBOI6eR0wRymrs4LTgY5j0dSt59d/DgwR4e0G4FBQXSCDJ4b775JrRRdXVbSVVV1bx5enEf9QtS6XqbmGPx88/D59reaW1tLS8fNGgQ9iX8ywT21Vdf4ecCAwNxFbipae3atV5euMABjcYGJgg+eA8ZBi6sVlbioi06LTs7+6zDKrhSh+7caZlGRUTAi5Mnhw8f7uPzqnUJYcaMGXq94754JadFiBsXAWfXrlu3bqWm5uTkHDrUKlLCXBW+yGSqr69XByYI/v7+aLMFCxY0NXV4Z+XKlVjfrVsdwchp5LQ76zT8nbp48WJ8UVOzceNGa6NBPPXUU9IIMngrVqwAsTFjIOsRm79y7VrbBRqOS0pKkmkbGTDsARhz584FpTVrUlNT28AgL2YC++233zD3hCxvzpy9e/fm5NjSMaVN0LJ9MG7cOOyBBx54QLxq1mw248nEieusjqvWSmCCuEIO4zd8fsIEQcz24H/W9VXIOWF8Skn52yEvVjN6Ll8OKX/7ZWroY2zN1o7Xt5ScBrML1CotLbV758yZM4GBkLJXVDiCkdPIaeQ0cpo7OG3RIm9vb6jrjBkvv/yyycSLT0uANANvHILURxpBBg+SLsyjnnjiiWefxfVIgOnXD9qL58darxDKiDkW37gBUxb0AsykxCcvQCQnJ2OHrlmzhgkMAmqHEt7ed911F97qaX1MEEys2MAEAWZOmJl7enpOnlxSUtKtm824U6cqacl0aDjegyom7bW9ekH+bp1IwcwWZH/8kQFMQKdZt2iubP8ctn37fH19/fz8mMBsZxScjI2NbcVg4RzxXA0LC5MSI6eR0+6s0/BxIR6YaordMH78+J07L1++bDRCT3ws9wQRebxFixbhzdgcB4lLfPx3333X2AgV5DgYlqW15NsNWj4gIMB6Zz3kw6Wl69atw8EYhmlWsNOnT0dHo7t48R4YqOfQofgSBlNWMBiAIiMjuduBnsUfsg+CUepQmFigDLZTbm4u7mvPzoZTNTExUWILu6LTJkyYYF0Ci42N3bgRUFta0tPTQ0LgC9566y0msMzMTM56w//XX2NGDXK7dyckJIC82ex4Vzs5TSCnkdPIae7jtJaWFpj35OZChrFzJ5YsXLiQ52F2dVpqf6AzPOHKlR07duzahQmoIP51DujQ6dNvyj66UkkLPw+Z44EDuJf76aefVsiInYHV1e3Zs+fgwVOnTlVWwstevQBM6dn/Slrgz9BQ2/ItToxwPin7zGFFMEF45JFHrDO9oKCgBQvwpjSYBX766aesYNCbjz+OTgMxkwmmzLm5oCRu7ohxzGKVwQoLC2ESbzXuoEGDYmJsa7kQW7bIgJHTyGl32mmOAYMUzzvuWVOHZx/r10NfLFwofwCDFnSIdRtXZ8HgPMBnZX34YSfAJk2alCNGTAyOni6fAnhjSXR02xofypVLP33HOVh9PbhC3zHi4qR26TsHA3MGB9tpgfug0rJ3yJLTOgQ5TR0YOa2zYOQ0dWD/sNNmzpzJ80uWLJE9gMVpgweD02bPlj9AtVZz84gRI3g+Kiqq82BFRUWYLMo+EJIFrLhYL64Fpyr8VULnYLW1r732mvgsR734uLiR5eWN7VdL2cCKi4snTnxe3A0DOXJWFlRWave6SrCDB8PxjxtYz4OUlB9++EFJjJzWIchpqsH+OafV1cGvR477RCEtU92hx44d8/fHlRz5YxjAsKq22yI7BfbZZ5/hyprsKh8L2JEj6A9NhnU1obmWpmLktA5BTmuvRU6zByOnsYGp1bpDTtu/3yhei3J8jIsLeOA07FClRwqqBvv1V7ztROnP8KoGs10BUpiMsnQozGwHDJB68gszmJogpzkGOY0VTE2Q0xyDnMYKpib+X5xWVsaLd0F+qLCoqRqvtraW5319fYuKnOCpATt+3GAwyG4SZwMrKSnp3TshIeHcOQ3AVAQ5TSLIaaxgKoKcJhGVlUFBQV5ejrdOu4DX0NCQnFyk5LN/e7tpqeUeYOQ018S01HIPMHKaa2JaarkHGMt1T3WK7tFuWmq5Bxg5zTUxLbXcA4yc5pqYllruAUZOc01MSy33ACOnuSampZZ7gFmcRkHxTwc5jaJrgpxG0TVBTqPomiCnUXRN/A/hW6myCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMzU5MQplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDkxNjE0NDIwNCswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjQuMykgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNTQyNyAwMDAwMCBuIAowMDAwMDAwNjQ2IDAwMDAwIG4gCjAwMDAwMDA2NjcgMDAwMDAgbiAKMDAwMDAwMDc2NiAwMDAwMCBuIAowMDAwMDAwNzg3IDAwMDAwIG4gCjAwMDAwMDA4MDggMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk2IDAwMDAwIG4gCjAwMDAwMDA2MjYgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNjA2IDAwMDAwIG4gCjAwMDAwMDA4NDAgMDAwMDAgbiAKMDAwMDAwNTQwNiAwMDAwMCBuIAowMDAwMDA1NDg3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo1NjQ0CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["pl.seed_everything(42)\n", "for i in range(2):\n", " interpolate(flow_dict[\"multiscale\"][\"model\"], exmp_imgs[2 * i], exmp_imgs[2 * i + 1])"]}, {"cell_type": "markdown", "id": "ac2dff10", "metadata": {"papermill": {"duration": 0.053429, "end_time": "2021-09-16T12:42:04.804401", "exception": false, "start_time": "2021-09-16T12:42:04.750972", "status": "completed"}, "tags": []}, "source": ["The interpolations of the multi-scale model result in more realistic digits\n", "(first row $7\\leftrightarrow 8\\leftrightarrow 6$, second row $9\\leftrightarrow 4\\leftrightarrow 6$),\n", "while the variational dequantization model focuses on local patterns that globally do not form a digit.\n", "For the multi-scale model, we actually did not do the \"true\" interpolation between the two images\n", "as we did not consider the variables that were split along the flow (they have been sampled randomly for all samples).\n", "However, as we will see in the next experiment, the early variables do not effect the overall image much."]}, {"cell_type": "markdown", "id": "e3c5cbb6", "metadata": {"papermill": {"duration": 0.052651, "end_time": "2021-09-16T12:42:04.909712", "exception": false, "start_time": "2021-09-16T12:42:04.857061", "status": "completed"}, "tags": []}, "source": ["### Visualization of latents in different levels of multi-scale\n", "\n", "In the following we will focus more on the multi-scale flow.\n", "We want to analyse what information is being stored in the variables split at early layers,\n", "and what information for the final variables.\n", "For this, we sample 8 images where each of them share the same final latent variables,\n", "but differ in the other part of the latent variables.\n", "Below we visualize three examples of this:"]}, {"cell_type": "code", "execution_count": 32, "id": "4c3c81a9", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:05.022247Z", "iopub.status.busy": "2021-09-16T12:42:05.021766Z", "iopub.status.idle": "2021-09-16T12:42:05.288274Z", "shell.execute_reply": "2021-09-16T12:42:05.287851Z"}, "papermill": {"duration": 0.323428, "end_time": "2021-09-16T12:42:05.288390", "exception": false, "start_time": "2021-09-16T12:42:04.964962", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 44\n"]}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDMzNS4yOTkzNTQ4Mzg3IDE3Ny40OCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjrEOwjAMRHd/xX1BYsdpk45FlSLGsvABVQVUtAgqwe/jMgDDSb6T7XuCiXwrOK1gTKYXBAW+G5+XYTyUHYaV2PKZVCsXmkaryuz130pKLmbL+DeeiRa6I7nwkQZ2eduOWXOC1Oo44zHiiAW+DRuAGIAYAKPYpQbBVix1/H4ZZvi9oLuhp57eJg8oJwplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0MgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjQ1ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLQ0NDPz8/Ozs7Nzc3MzMzKysrIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCurq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh1dXV0dHRzc3NycnJxcXFwcHBvb29ubm5tbW1sbGxra2tqamppaWlnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZVVVVUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDMyMSAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTY0IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMzIxID4+CnN0cmVhbQp4nO2de1xTdR/Ht7MLW2PcHHIbjFBgZnlNCtDHIrsAeUUyUUBLJcxLSKU+lhpRWd5CkC5eghREgxRLLSoNCbPyWqiVKcUt7jLG/n++jGeMbeeccc5vvJ49r76fP3y9zjnbm+95z/3O71z2+wn+jSGL4H9dwP990CBp0CBp0CBp+gwaHJZ/Hg4NkuLQICkODZLi0CApDg2S4tAgKQ4NkuLQICkODZLi0CApDg2S4tAgKQ4NkuIGa7CzszMvL6+y8uTJkzNmzEhOJqxRp9PV19ffvn3+/Pnt27fbqdE+rqen55dffsnMXL16tXGhu7ubBAeBwpYsmTp1am4uy4vQIEscalCv1x87plAoRCLRli1xcXH5+fl1dW1tbXfu3OFTI+AKCzUajUQiWb78oYceysnJ0ethfVMTU42sOPgsDh3y9fWlKOpfvbl2DdbcunWrq4sXrqOjo6gIcFBdfDxUl5enN1Zn/IcWhwYtgwad0uC+fftEovDw8Kqqqp6e1tZWUyOTkJDw00/ca9y9e7dQKJPJiouLzSthr+Xyn/jg3nnnHYoKCgq6cuWKwXD16tWWFoOxKUxP//HHH7njFixYIBK5uLhAo2+xfssWJhwatIyjDUIle/bAf+hVq65fv269Ef6P79vHqUb4Qrz2mlwu37TpwoUL1n8oIODgwYOccM3NL7/8MjQvy5fDgdh6Y3T0u+++ywnX1LRixQqVSvXoo6dOnbLeOGXKoUOHaHFosD9o0DJOYhAOt5MnZ2dn226prq728/Pj2NKUl5fLZEVFRTdvWm8pLCyUSn/77TdOuNJSOABv27bNdsuOHTtEoi7bwzErLi8Pehvz58+33fLBBx+IxT/88AMtDg32Bw0OiPMYPHHiBO2OtbRotdpZs2bZvoMVl5aW5uoKfVbr9R0dgYGBcXE072DFLV4slUorKyutVoM56HslJnLFLVsmFosPHz5sixs2bFhMDE3daNAyQ2IQviVS6U2rL11nZ+fDD0On8+233+ZYY1RUlIeH9ZcLOtOJiWCwro7mHay4kBBvb+/a2lqLladOZWVlwdcxP58rLj5+woQJdVZl9HbN09Vq9R9/MOHQYH/QYF+czCB0V4XCc+fOmde0tbW98cYbCsUzzzxD+w5WHJwNU1RZWZl5TUNDQ0ZGhrs77SHQHs7bWygUfvjhhxbVPfII/JXZs2dzx3l4AM54em1KU1NTcrKHh0dSUhIzDg32Bw0anNEg7JhYnJ+f39lpWnPgwAGBQPDII7SXV+3VmJKSQlErV640r9m4cSOUPW4c0ztYcdHR8F6orqfHtGbevHkiUWhoaI951eBxAQHQfEZERJjXxMXFicWRkZGst13QYH+GxOCuXbsCA6HM0tLW1tbGxsYZM6DTCV9ghgLt1XjkyJG5cwGXkgJfN8DNmgVd4rCwMNor8vZx77+fmpoK3YXUVDiCwl6uWwe4SZP0tFfk7eMyM2NjY6G6NWvq6+uhwLQ0iUQSFWUPhwb7gwad0qCh9z6Eq6urTAa1+vv7UxQ0FRUVFcwvt4crLIRjW3AwHCyNx/ng4OC//vqLNw5651OmTBEIYL+p/+abb1j+uh0c9AxiYmJEImj6vLy8hL25eNEeDg0OCBp0ToPGexgnT0ILAzw3N2gHp0+fzrMdNAbale+/78O5usI/8NnodPxxkNraPoMiEVT30kssrxwUrqICTvoBJ5dDlatWMfWMDGiQKY43CElKghrhm9fWVllZ6enpmZNDVGNMDBQYHx/f0VFQUODm5hYUZH2FhRNu82Y4aKrV6v37Q0JCJJI333yTqDovL6hu7Nixly9v3bpVIlmwYAErDg3aBA06pcHgYD8/P9PC0qVLXVw+/fRTnjVC11ej8fX1NS1Dr10q3bBhA08cZMIElUplWliyBHR+/PHH/HEBAdBdMC1kZ0NjaHE5yRqHBm2CBp3SYFSUpbHRo1kvOrKy2tvbJ00a8EwL9Irvv5/2vvmgcJCdOz/77DPzokajsbiCxhV34sSlS5dMCw0NY8aMyWV4mhoN0mdIDFonImLHjh38a7QI9FjHjmX9nnDC6fURERFwEHUQzmDQarWffPIJMw4N2gsaJMMNiUFoZBSKmpoaB9X41ltvubjQPELJEzd/PnTRm2h/GMAHd+KESCT6+uuvmXFokD2ONwgwpVIpl1dXVzuixtLSUnd3dw8P6ydLeOJKSko8PYcPH3779m1H4I4dO+bmBt3L06dPM+PQIEvQICluSAympKQIBAKN5urVq+Q1dnaOGzcOcEql9SNTvHCNjXDcpChvb++vvvqKHGcwjBo1iqLEYvHnn3/OjEODLEGDZDiHGuzuvu+++4qLixMTKYoSCoV33830ykHifH19KyoqRo0CFhgMCyPCGQzQllZVVYWEGG+vjR49mhCXkJBQWVmZmGjEwR5/9913zDg0SJMhMNjcbPyuKY37C9S0NKIadTooC0gikdB4y3jgM3s8cAYD9Ho9PT2NOJHo7NmzRLieHihNoVAI+sJ0rdGABpmCBp3S4ODzz8OhQVIcGiTFoUFSHBokxeGo8qRBg6RBg6RBg6RBg6RBg6TB3gwpDg2S4tAgKQ4NkuLQICkODZLi0CApDg2S4tAgKQ4NkuLQICkODZLi0CApbrAGu7q6bvTm9OnTkydPth3TkWONPT09WVlZL7ywcuVKmoF3OeMgL7744u7dGRkZFMXyokHiOjo6tm3btmlTZGSkq6vFyOM0ODRIEzRIinO0QZBXXq5SqSQSyeLFjz/++N69e+/c0el0tD85sI8DeUePenh4iMXiefNiYmK2bt2q13d2dtIO5DWoXd61y93dXSqVzpkzbdq07dv7BkagHR1hULj8/L6JMKZPT0hIOHy4vb0diMzVoUGbON5gdna2SAQ1rl271rxSr9d7enraPmxrH1dSUkJRPj4+69evN6+ET+mJJ7744gvuuLy8PKFw/Pjx31gNNHPmzObNm7njFi5cqFB4eXkZfwgGO2n8IIqKiuLi6uvraXFo0DJosD9OYhDetXYtNIHr1p0/f95iC8C1Wm1JCacaoUmZPdvV1XXaNJqfAsTE7OM41YTBsGnTJsDNnEnzs7zRo/fv388RB02yXC6PjaWZambMmOPHj9Pi0OCADIHB5cuXi0Rz5861HYLv1VdfFQgEts/bsuLOnj0rENAOWltWVubvn56ezglXVgbdg2XLltlu+eijjwSChoYGTrijRwG3Zs0a2y1ffvmlQkHTb0WDlkGDA+I8Bo0zZ1nPCGTobdDCwsJoRxNlxT3//PMSCc0Rt7tbrVaHhdG0Zqy49HTo99q2Wa2trf7+/mlpNIO2seIWLXJxcbGdhQQ6+8HBwQkJTDg02B80aIpTGSwoKBAKv/32W4uVzc3NTz4Jf6zEti9jp8Zhw4a5u9NME7Z5s1KppJvLiAXX3ftbP6FQaD1q3PXrycnJ0CexmfjFbnUjR44YMcJ6Xr8LF1asWAG4a9eYcGiwP0NiUCwWU1R5ebl5TVNTE7xeLk9JSaF9ByvurrvuoiiLYxocL5cuXermFhsbyxEH4r29KYq6deuWeWVLS8vixTKZbObMmdyrUyqhd2Ex/lRjY+PcuaDvqaeeYsahwf6gQYMzGgSeSAS7aFrW6Y4cOQJlBwbSTCBmv8Z7772Xoiy2QyMDx9OwMF6TOTz4ILSD0N83z9ywc+dOilq9ejU01dxxBQVgEAr6+2/TmoULF1IUdNlpZr4z49Bgf9CgwRkNVldXT5wIZSYl3bx5888//5wwARa0Wi3DjCf2anz99ddnzIAPICXlxo0b0GZt3CiVSidOnMgT9957UVFRgNu7Fz4AqC43F/pYY8bYG4KbaePBg3COALicHONdmzsbN8J5ckgI2dwQaJDUIMTYPVUqNRqNq6srRQH0119/ZX65PdzlywG9s/m4ubmNHDlSKAQcycwGsHfGMWFyc3N9fHyEQjBofyoC5s1wPhgeHi6RxMfHQ9siENxzzz0M4+AMwKHBAUGDzmnQmDNnYFdNsy8899xzvGs0GLvCNTXGAgXu7oBbtWoVCQ7Oj0tLwRw00HI5YDMzWV5sHwefycWLfTOdhIcDc/p0ponCDGiQNmjQWQ2uWQNE6F9WV+fl5alUqgMH+NcIBqOj4QNJSkpqasrIyICT5XXr+OMMvXcioboHHnhAry8rK5NKt2zZQoSbNw/6WJmZmTpdTU1NeDicsrNOb4gGbTIkBqdO9fT0NC2kpqZKJHB2x79Gf//g4GDTgnEsYItLIlxxGs3w4cNNC1lZMpmsqqqKP87PbwDulVegoaGdqdmABpmCBp3S4O+/W/Z7ExNpHowaZI3QDloMJdze3j5y5KJFi3jiID//PGAo4e5u6LLznz3F0Dsc5Dpzw9zaqlark5OTmXFo0CZo0CkNWmfECOsHHjnVaJ3oaP4fiHU6OsaPH0/UDloFcAUFBcw4NGgvQ2AQeqwKBevhiROurq5u2jReV/lps349dK+t75vzx125Asdi68dPLXBokD1okBTneIPHjx9XKpViMe1DM9xrLC4uDgoKUqv/Nt8gI8FdunTJePnIQfPPnDt3zsUFzttZ559BgywZAoM6nZdxXlCKcsjMBl1dTz/9tEAgCAzkdYPXNhs2bOgd5Vpk8SgIf1xOTo5YDCeJDpgz0RQ0aItDgyxxoEG9HhqX2trayEjjdAnC0FAdw8zhg6wRtHX2zk7Sh9NqmV43KFxXF5zIlZeXP/aYkTZgklZeuO5urVZbWlo6c6Zx5orQ0FDWngIatAkadEqDzc3QhTG2f9DIhIeHt7QQ1VhX9+yzz6pUKrEYTPr4+BQWEuEuXlSr1Xv27AkIgF2WyUjnhigpgQ5M39QzAsGcOWfOnGHFoUGbDInBweefh0ODpDg0SIpDg6Q4NEiKQ4OkOBxVnjRokDRokDRokDRokDT/AWICtgoKZW5kc3RyZWFtCmVuZG9iagoxNCAwIG9iagozNjk2CmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTEgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iagoxNSAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwOTE2MTQ0MjA1KzAyJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuNC4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuNC4zKSA+PgplbmRvYmoKeHJlZgowIDE2CjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDA1NTcwIDAwMDAwIG4gCjAwMDAwMDA2NTkgMDAwMDAgbiAKMDAwMDAwMDY4MCAwMDAwMCBuIAowMDAwMDAwNzc5IDAwMDAwIG4gCjAwMDAwMDA4MDAgMDAwMDAgbiAKMDAwMDAwMDgyMSAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDA0MDIgMDAwMDAgbiAKMDAwMDAwMDYzOSAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDA2MTkgMDAwMDAgbiAKMDAwMDAwMDg1MyAwMDAwMCBuIAowMDAwMDA1NTQ5IDAwMDAwIG4gCjAwMDAwMDU2MzAgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyAxNSAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgMTYgPj4Kc3RhcnR4cmVmCjU3ODcKJSVFT0YK\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDMzNS4yOTkzNTQ4Mzg3IDE3Ny40OCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjrEOwjAMRHd/xX1BYsdpk45FlSLGsvABVQVUtAgqwe/jMgDDSb6T7XuCiXwrOK1gTKYXBAW+G5+XYTyUHYaV2PKZVCsXmkaryuz130pKLmbL+DeeiRa6I7nwkQZ2eduOWXOC1Oo44zHiiAW+DRuAGIAYAKPYpQbBVix1/H4ZZvi9oLuhp57eJg8oJwplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0MgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjE4ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fl5eXk5OTi4uLh4eHg4ODf39/e3t7c3Nzb29vZ2dnX19fV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnFxcXExMTDw8PCwsLBwcG/v7+9vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKwsLCurq6tra2rq6uqqqqpqamoqKilpaWkpKSioqKhoaGgoKCenp6dnZ2cnJybm5uampqZmZmYmJiXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX18fHx7e3t6enp5eXl4eHh3d3d2dnZ1dXV0dHRycnJxcXFvb29ubm5tbW1sbGxra2tpaWloaGhnZ2dmZmZkZGRjY2NiYmJhYWFgYGBfX19dXV1cXFxcXFxbW1tXV1dWVlZVVVVUVFRSUlJRUVFQUFBPT09NTU1MTExLS0tKSkpJSUlISEhGRkZFRUVCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo5OTk4ODg3Nzc1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMREREQEBAPDw9cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDMyMSAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTY0IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMzIxID4+CnN0cmVhbQp4nO2d61cT1xqHsydAuIgEayMWIyqitUUrWm1VLoIRrEQFL1XQtloNXg6C9QZa79dqqoWi1SrWKqJoixIUJKCiQhFFqUAjSf6a8yY5rnVcjNnjeves41m+v0982OtZv3lmZWZPyOyt+RcFF83/usD/fcggNmQQGzKIjc+gW1jePRwZxOLIIBZHBrE4MojFkUEsjgxicWQQiyODWBwZxOLIIBb3BganT58eH19eXu5yuThQRbjU1NSIiJs3b/I7KsIlJSWNGSMON378+OzskydP8nFkUD5kEIsTb3DXrl2SJI0ZU19f73A4Xu9QIS4nJ0en03355fPnz/kd+bjLly9Du08+efjwoQhcc3NzfHx8v37FxcV8HBmUiSoGY2NjoePp07xxCnFGoxFwV64IwsEVQavVnj8vCDdp0iTArVihCEcGZUIGsTgVDD59OnLkSDjk8eMfP36M79jVNXr0aMAtWtTe3o7HtbWFhYXBIRsMdrsdjyspCQkJCQwMtFq7u7v5ODLYJ6oYtFoZY8HBwYxdv34d3/Gvv0AfHLJGc4XzQVaEu3wZWAEBAYzZbDY8rrbW146xxsZGPo4M9gkZxOLEG2xtbZ00CZ5yFi5cKMigywXXv3nz5jGm6LmJh6uqmjBhgveQa2tr8Tin8/jx4xEREYzduXOHjyODfUIGsTjxBl9m7NixjF28eBHf0Zv09HTGTnPm6Apxvb29ZrOZscrKSkHt0tLSGLtw4QIfRwblo4rBmJgYSVq2bJmgjgUFBZ7HRCEGIdu3b2esvLxcEG7x4sWMJSUl8XFkUD5kEItTxeCcOXMkyWKxCOq4Z88eSSosLBSEO3bsmCSVlJQIwpWWlkpSfn4+H0cG5UMGsTjxBl0uV2RkpLALl91uHzFihCTB1UsEDpKbmytJFRUVInBwsPCYrdXW1NTwcWRQJqoYbG5uZox98MGjR49EdLTZbBqNZtAgQf9cgxMC7d5//8GDByJwbW1tgBswAJ5n+TgyKBMyiMWpYnD9+vWSJM2axRunELdo0SLo+N13gnCbNm0C3Lp1gnBLliwBHGfW4SaDrw8ZxOJUMNjQ4Pt/4k8/Cen4xx8GgwHmC5y5h1JcXV1UVBTgfvlFCK66euDAgUFBQWfPKsKRwT5RxeD338NHeNy4cU1NQjpaLPAhiYuL++cfIbi6Orivr1y5kjdOIa60FHA5OTn+fib5Xzgy2CdkEItTxeDUqcHBwU18fwo7pqSEh4ffuHFDEC4hwWg03rp1SxAuMVGv19+9e1chjgz2CRnE4sQbvHr1amjohg0bXC5XaWl0dHRqaiqmI2BiY7Ozs+HPb74Bk3B5xeC8s9+ioiLv4LKysq4uP4P5uKqqqoCA/fv3O53O9HSY1TgcPBwZfDWqGARjsbFtbW1w6GYzTKtnzJiB6Qj1PDf1pt7eXoMBbvEr/P3alo8zm82Rkb5vVocPB1xZmZ/BfJzJZEpM9B3shx/269fP73SBDMqEDL6VBg8cOLB9+9atWyVP4HnR72A+btasWRoNHDJ01Wrhsur3Fs/HdXZ2Rkc7HA673T50KBRcuhTV7tChQ3/++eTJk4kTJwYEgEG/g8mgTFQx6E1ra+uaNWu0WtDJh/JxPT09eXl5Wu2pU6dE4NyeOWbKwIHFxcX8jx2fBSdk48aNjFmtVn47MigTMojFqWJQad49HBnE4sggFkcGsTgyiMWRQSyODGJxZBCLI4NYHBnE4sggFkcGsTgyiMXRqvLYkEFsyCA2ZBAbMogNGcSGZjNYHBnE4sggFkcGsTgyiMWRQSyODGJxZBCLI4NYHBnE4t7AYFpamslU5vdXtoo7ulyuuXPnxsby3lFRiHM6nfPnz//444aGBhE4t/enz++9d+LECT6ODMqHDKJwbjUMrlq1SvKs/OZ3cxLlHa1WK+C+/vrFixcicAUFBVqtds6cnp4eEbi8vLzAwMDPP8/NzeXjyKBMyCAWp4rB5ORk3usub9Jx9uzZcMi//y4I59s1QNAqEN4XNrSeVbcU4cigTFQw2NCg0+kYY198cenSJXzHrq6QkBDoaLE4/L47qRDX02M0GqHdt9+e52xRogj399/h4eFwQhITeS8Zk0H5kEEsThWDVVVBQUFhYWEazZkzZ/AdS0q03jCmqCMPl5Xlw2k0paWleNzZsy93rlB0sGSwT1Qx2NJSWVk5bdo0SRKys0FDw6hRoyIjIzUa3voSfBw80BmN/fv31+v1knTkyBF8u127srKyhnpeFO3o6ODjyGCfkEFsO1UMepOZmcmYkLudNxkZGYwpWm1XEc5kMjF27tw5QbjJkycr3AiDDMqHDGJxqhiEep6FP/2v/Kkct2/fPsb8LvTxRjibzSby/MbHxzO2e/duPo4MykcVgwcPHpQkcQYPHz4s8oTs3LlTkmbOnCkIZzabJUno/sVuMvgaHBmUj3iDLpdruGdVpm3btgnqGB4eHhioaDcuPqu7u9s7BRbXDnCMHT16lI8jg/Ihg9h24g02NTUxxiIihHwd9XIzh+jo+/fvi8C1tLQATq+/d++eCFx9fT3ghg9XtNUEGZSJKganTJkC0B9+4I1TiMvOzgZcSoqQb/nd7nXr1gGOs2uWclx+fr4kSQUFinBkUCZkEItTwWBzc6g31dVCOlZUGAwGnU73889CcDdvwn0dcL/9JgRnt+v1ejghP/6oCEcG+4QMYnGqGKyuhuuC1WoVtJnDsmVQcPPmzbxxCnH790O7pKQkhZs58AZdugS4lJQUzu/A3GTwdVHFYEJCVFQUb36pvOPkyYMHD759+7Yg3NSpcFFobGwUhPvsM/gUV/MvWGTwdSGDWJx4g11dXTrd0v+slP3s2TO/PwTm43p7ewMCMjIy3J7VuOPi4trb2zG42tra0NCioiLvzhUmk6muzs9gPu7atWuMZWZmuj0bLC9fvryqiocjg6+GDL6VBhcsWPDRR75XZw4cgGm131W4+bjCwsKEBN+rM199pdVq/b7jw8clJydPnOj7L+enn8I05PBhP4P5OIvFEhMDGp1OZ1AQ4Py+k0MGZaKKwbq6ugEDQOOwYcO8e0N0dnZiOsJkWqPxPTdJEjyL2e12DM7hcFy9+vTp07179wYGwgn59Vc/g/m4HTt2DBoUExOTnp4eFAQ4m42HI4Ovhgy+lQa9aWlpSUtLCwvbsmWL33EKcR0dHXA77t/fwtkkWCEO7u+rV68ODeUNVoirqakZMmSIwbB27Vo+jgzKRBWDSvPu4cggFkcGsTgyiMWRQSyODGJxZBCLI4NYHBnE4sggFkcGsTgyiMWRQSyODGJxtKo8NmQQGzKIDRnEhgxi829rp63ZCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMjQ5NwplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDkxNjE0NDIwNSswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjQuMykgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNDI5MCAwMDAwMCBuIAowMDAwMDAwNjU5IDAwMDAwIG4gCjAwMDAwMDA2ODAgMDAwMDAgbiAKMDAwMDAwMDc3OSAwMDAwMCBuIAowMDAwMDAwODAwIDAwMDAwIG4gCjAwMDAwMDA4MjEgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwNDAyIDAwMDAwIG4gCjAwMDAwMDA2MzkgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNjE5IDAwMDAwIG4gCjAwMDAwMDA4NTMgMDAwMDAgbiAKMDAwMDAwNDI2OSAwMDAwMCBuIAowMDAwMDA0MzUwIDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo0NTA3CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}, {"data": {"application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjExIDAgb2JqCjw8IC9Bbm5vdHMgMTAgMCBSIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDMzNS4yOTkzNTQ4Mzg3IDE3Ny40OCBdIC9QYXJlbnQgMiAwIFIgL1Jlc291cmNlcyA4IDAgUgovVHlwZSAvUGFnZSA+PgplbmRvYmoKOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEyIDAgUiA+PgpzdHJlYW0KeJxNjrEOwjAMRHd/xX1BYsdpk45FlSLGsvABVQVUtAgqwe/jMgDDSb6T7XuCiXwrOK1gTKYXBAW+G5+XYTyUHYaV2PKZVCsXmkaryuz130pKLmbL+DeeiRa6I7nwkQZ2eduOWXOC1Oo44zHiiAW+DRuAGIAYAKPYpQbBVix1/H4ZZvi9oLuhp57eJg8oJwplbmRzdHJlYW0KZW5kb2JqCjEyIDAgb2JqCjE0MgplbmRvYmoKMTAgMCBvYmoKWyBdCmVuZG9iagozIDAgb2JqCjw8ID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9JMSAxMyAwIFIgPj4KZW5kb2JqCjEzIDAgb2JqCjw8IC9CaXRzUGVyQ29tcG9uZW50IDgKL0NvbG9yU3BhY2UgWy9JbmRleGVkIC9EZXZpY2VSR0IgMjQ4ICj////+/v79/f38/Pz7+/v6+vr5+fn4+Pj39/f29vb19fX09PTz8/Py8vLx8fHw8PDv7+/u7u7t7e3s7Ozr6+vq6urp6eno6Ojn5+fm5ubl5eXk5OTj4+Pi4uLh4eHg4ODf39/e3t7d3d3c3Nzb29va2trZ2dnY2NjX19fW1tbV1dXU1NTT09PS0tLR0dHQ0NDPz8/Ozs7Nzc3MzMzLy8vKysrJycnIyMjHx8fGxsbFxcXExMTDw8PCwsLBwcHAwMC/v7++vr69vb28vLy7u7u6urq5ubm4uLi3t7e2tra1tbW0tLSzs7OysrKxsbGwsLCurq6tra2srKyrq6uqqqqpqamoqKinp6empqalpaWkpKSjo6OioqKhoaGgoKCfn5+enp6dnZ2cnJybm5uampqZmZmXl5eWlpaVlZWUlJSTk5OSkpKRkZGQkJCPj4+Ojo6NjY2MjIyLi4uKioqJiYmIiIiHh4eGhoaFhYWEhISDg4OCgoKBgYGAgIB/f39+fn59fX17e3t6enp5eXl4eHh3d3d2dnZ1dXVzc3NycnJxcXFwcHBvb29tbW1sbGxra2tqamppaWloaGhnZ2dmZmZlZWVkZGRjY2NiYmJhYWFgYGBfX19eXl5dXV1cXFxcXFxbW1taWlpZWVlYWFhXV1dWVlZUVFRTU1NSUlJRUVFQUFBPT09OTk5NTU1MTExLS0tKSkpJSUlISEhHR0dGRkZFRUVERERDQ0NCQkJBQUFAQEA/Pz8+Pj49PT08PDw7Ozs6Ojo4ODg3Nzc2NjY1NTU0NDQzMzMyMjIxMTEwMDAvLy8uLi4tLS0sLCwrKysqKipcKVwpXClcKFwoXCgnJycmJiYlJSUkJCQjIyMiIiIhISEgICAfHx8eHh4dHR0cHBwbGxsaGhoZGRkYGBgXFxcWFhYVFRUUFBQTExMSEhIREREQEBAPDw8ODg5cclxyXHIMDAwLCwtcblxuXG4JCQkICAgHBwcGBgYFBQUEBAQDAwMCAgIBAQEAAAApXQovRGVjb2RlUGFybXMgPDwgL0NvbG9ycyAxIC9Db2x1bW5zIDMyMSAvUHJlZGljdG9yIDEwID4+Ci9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9IZWlnaHQgMTY0IC9MZW5ndGggMTQgMCBSIC9TdWJ0eXBlIC9JbWFnZQovVHlwZSAvWE9iamVjdCAvV2lkdGggMzIxID4+CnN0cmVhbQp4nO2deVxUVRvHnTvDOiFUshgYWxiEgCmyiKWGBiYYLUCoWWgFhAuWBi2vIgapQQWUAWFstiiC5DaJZUHkVhhWWGmlo4YYsQmIf77PnJk7zMC9M3M5w+ed99Pz+wNlLvfLc7/DnHvOnbnnjPsPhi7j/tcF/N8HDdIGDdIGDdJGafCm0fLvw6FBWhwapMWhQVocGqTFoUFaHBqkxaFBWhwapMWhQVocGqTFoUFaHBqkxaFBWpxQg21tbTExMQUFMpmMH2o4rrOzc+vWrdnZBw4ckMvpcdeuXWttjYyMTExM7Omhx0EGBg4dOlReXt7fz4dDg7pjVIMHDx5cv97d3Z1hGJEIvkyePJmmRngO0tKAweJSUvhqNAhXXV2dmurj48OosngxFQ68bdsmlUolEhGpLiODD4cGuYMGTcrgjRs3GhuhcRk/fjyjDnAlEsn+/cJrHBwcrKuLj493cXFhNGNj09jYKBz3ySefbNwIz6uVlRWpisVJpcePHxeO6+7u3rcvNTXV2tpaBRKT2NvDGYAThwa1MwYGL1xISEgQi5U0MzM7O7ukpKSmpry8PDC4e7fQGn/5JT09XSJhcba2tkuWLKmvX79+vUi0X/AT8ttvvy1YsICVJpH4+flFR588eRJ2Eom+/PJLgbirV68uW7ZsCPfYY48lJNTU1CQnJzMMNBOcODSoETRoggahwxsVNdSySLOyoBVjN7a2tnZ2Cqzx5vTpjOrcC7g1a9iHe3p6EhMvXbokCNfVFRISwlYH8tauZatramqKiNi0aZMg3OAg7GZmZkZwEyZMyM4eGBggW6C/Hxx8+PBhThwaVAcNmqTB7OxsVYMFzU1tbS3fzxlY42uvvUZaVIkkNjZ27969lLgdO9gGC6qrr6+nxG3fruwMicXQ14IetYE4NKiO8Q1CT/rBBx8UiZycnN57Tz9OX43wCnvllVcYJjo6+quv6HHQ9fX2hkOeNGkSpzvVC9BQHOTxxwEHw4a8PEHVoUF10KB2TMHg5cuXyXlzxYoVWo+DipKSzMzM8+fPC6kRTt0iRZ566qnhm3Jzp06dytnK8uP+/vtvMg6OiYkZtiUnJ2fevHlXrgjC7dmzh+CWL18+bEthYeHdd9/d2sqHQ4PKoEGNmI7BhoYG0llYt27d0IPnzp0rKysjj8OgVEiN0F0lu2VoXqzs7e0Fs+bmlpaWGzfy1ciJk8vlpO/x8ssvDz14+vTpyEgYrzs6Oo7s7BtU3Ztvvjn0YH9//44dYrHYw8Pj4kU+HBrUrs6oBmFIN3PmTIaBc7G/f0BAAJyY4+LmzJlDfhO87Dh34sd1dXUFBwcr+r8Se/vAwMCHHnpowQJPT0+J4koN5zsGOnHXr18nQzqoLjDwiSeeWLx4cWLi7NmzSXVgUCAOhpXkyO644w43NxjS+fv7+/qGhYURnIuLCz8ODSqDBk3SIAS6wEPvPIhUId9UVVUJrBECBzpE0viXYZ555hnhuFWr2N1Fw5LH0yvWc7DscTKaAdyuXbv4cWhQ42DRoAka/Pnnn1eu1CqTtFqMl1dHR4fwGhsbG6dN08CNG2djYwP/tbODXyQcd/58WloawbFPLakuNDT02rVrwnE//ZScnKyBE4ksLCwYxtfXF3rv/Dg0qM6YGLypOOfB3/CLL8JZrrKyMj9/4sSJcMje3nw/rg+Xn5+bm7tzp5eXF4zuzp4tKSmBSu3tR4nr7u4uKiratauioqKuri4iQqy4gBsSEjJK3MWLFz/44IOjR/fv35+fn79pU3R0NHn7QCcODWoEDZqmQa3k5irbQR+fUdaonYICZXvI0zsXilu9mrRi06dPNwrurbfIEzJjxgydODTIGzRIizOywcHBwW3btpmbK0/zQUH0NRYWFrKd1tdfp8RB3woaMKmUGHydGgfZsGGDmRk51qysLJ04NMgdYxvct8/NzY3tVE+bNq2piarGY8c8PDzgJaLoU4+75557fviBCnfqlJOTEykNcH5+p0+fHj3uxo0bra3e3t7swc6c2dzcrBOHBrWDBk3QoFwOPEtLdqCYnZ3dw3mzi4E1NjVBN5Udd0ok8fHxUPOocXB05JKvSPWBl+rq6rNndfx2PbiGhobY2Fh2XGxuDuPkP//Uh0ODGhkDgydOnJg6leDgNbds2bK6Oh08/TUeOXLE3191GYVJTEz8+msqnEw2ZcoUFjdr1qyRHyIWhNu3DxoU9mAjIiJGfuqXE4cG1UGDJmmwuLiYlCeRlJWV9fb26kPqqfH5558nFbq6fvTRRzqbU4Nws2axDepnn33Wz3n3pRBceDiLc3d35/hEMg8ODaqDBk3SYFxcnEh0yy235OTo7HMYWuP8+fNFIqlUmp6ucUvA6HHkE5gTJkzYsqWvr48eN2MG4CZOnJiU1NLSYjgODaozJgZXr14dEsL3TpXwGoOCgubOhVeckXBRUVZWVq2cn+wbDc7Z2dHRsbS0VCAODaqDBmlxY2JQeP59ODRIi0ODtDg0SItDg7Q4nFWeNmiQNmiQNmiQNmiQNmiQNtibocWhQVocGqTFoUFaHBqkxaFBWhwapMWhQVocGqTFoUFaHBqkxaFBWhwapMUJNXjp0qWMjIz09L1793J+9kU4LjMz89Ch2tpa4ZO1jczly5erqqC+yMjIq1fpcXK5vKVlzZo1KSkp7e18ODSoI2iQFmdsg/X19QsX3nnnnex0GPHx8TQ11tXVrVrF3ugDX1JT+Wo0CPfpp59mZnp5ebG3RlDiZDLZSy/5+vqyOK5jRYO6YlSDnZ2d7767cuVKsvqC9kQsNTXCa/z9999PnfL29pZKpYx2OCYb1o87evToxo1OTk5kXg62LvKVY3Zg/bicnJxFitjb22sdqVj8w8h719AgR9CgCRo8e1Zj9QWGcXZ2jouLKy9/4YUXzMzMRt6prQ/3xx+PPvooCxOLra2tH3jggePHYSeG4ZicSQ8OTuWw+xDOzs5u0aI9e/a8+uqrYnGN4Oe3vb3d1dWVxUkkUN2zzzY3NxcXFzPM5s2bOXFoUCNjYLCvr2/KFJbo4eGhdf/99evXR+6hE9fT05OQwOLGjx8fF6ex0dX1ysjZZ3XiurqgO8Di4IU8NAltR0dHaWlFRYUgXEdHbGwsuesZcCCvuFhj49tvc9zNgAa1gwZN0mBJSYmqSZg/f35DQwPfzxlYIzR0KtySJUsOHjxIiVPNSaxcaqJO/32nenA7dypx5uZRUVGGL4SBBtVBgyZp0MbGhtxq9u23yu/13w3HjxscHCR3QPv7+49crwe62cKvzXh6Ku+ry8/XV5V+HDTRs2aBPuhJb98+fKNcLj9xgg+HBpUZE4MwKmIUI5pHHnlk+KaCAh8fH85lgvhxZCkCRjFN2Yh5yqqqQOv33/PVyImTyWRkuDVv3rxhW95///177733zBlBOOibE9zs2bO1Hu/rg9YmPDx85Np3N9GgVtCgRkzH4OHDh8m5CWQNTa0LrdnSpUvNzS0sLIqKioTUeOrUKXLVRKuj29/fDwfMMJaWlpw78eN2795N1nXdsmXL0IPwLG3YwJDV5jhnouLHsb0OrT+Lrq6u5cvJyV6yahUfDg0qgwaVMTGDvb29oaGhDOPg4ODjk5aW9uSTT86caWtrS34TPCjwkKEcstQEdI68vBISEmJiYiIVITg7OzuBuM7OTvKWzW233TZlyqJFiyIiIsLDod9PcAEBAQJx7e3tiuKY+++/PyRkzpw5UFhAALuopaenJz8ODSozJgZvKmaPzlddNRexszOpohPKh4Nzmmp3dmYX5XdicWBgoHDckSMirYwbxzJ9fX2F43JztasijQT5d+HChfw4NKgOGjRJg2fOnElK0jBIpkUDpK3tsWPHhNf4xRdfpKZqHDKpUaSYxIdjxXD9uJ6e7OxsreeXvGEHLeNXPOta6sRduECWYxl+sDY2NidPnuTHoUF10KBJGiRlwrh49eq8vLyPP/44KcnPzw8Mzp3LNw+XPlxvb2VlZVxcRkZGcXFxeLijoyNU6uzM9+N6cG1tbaWlpVlZH374YUtLy5Ur3oqpuB5++OFR4np6eurr69PTn3vuOZC5dm1QUBC53qUThwY1MjYGtbJli1FXNnjnHeXC8+7uRsEVFZkpZoLXt6CIobg33iCvYldXV504NMgbNEiLM77BdevWWVgoT1IFBZQ1Dg4O5uTksBOFL11KiRsYGKioqLCyImfS5ORkShw08oWFharq7rrrLp04NMgRNJhMiRsDgwcO3HrrrexwFtqZ776jqvHYMeWSYwQXEBDA+RaJ4TiZDDpFLE7xhgsFDp7aH390cXFhcTNm6PigARrkyBgYPHAA/oytrcnrg5FKg4ODz507x//j+nD//FNeXs6OmxwcoKNOg2tuboZer0SixNnbJyUl6ZzyVA+uu7s7NTWVDDQhlpbQ0HC+066FQ4MaQYMmaPDzzz/39WWvVCQkJFAu5lBTU+PsLFJdwVyxYsXx41S46movLy/2Ikp8fPw331DhamsdHBxUF3kUa03KZAbh0KA6Y2Jw0qRJBJmY+Ouvvxo0L7VOXEpKCsGFhf3111+cHyIWhFuzRqQ6/ba1tRk0T71OXGKiEjd5cllZmX7WTTQ4PGjQJA1CP1qxZpY5/8KLgmoMCwsTiaytrTdsMAruvvvgeG+//fZt24yCCw0FnI2NTXq6IBwaVAcN0uLGxODWrVuffrqJf41JgTVaWVnNnXuV8/790eAqK0NCQgy6w8AgXEGBm5ubXC4XiEOD6oyJQeH59+HQIC0ODdLi0CAtDg3S4tAgLQ5nlacNGqQNGqQNGqQNGqTNfwHPD65oCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKMzYwNAplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDExIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKMTUgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDkxNjE0NDIwNSswMicwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjQuMykgPj4KZW5kb2JqCnhyZWYKMCAxNgowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAwNTQ4NyAwMDAwMCBuIAowMDAwMDAwNjU5IDAwMDAwIG4gCjAwMDAwMDA2ODAgMDAwMDAgbiAKMDAwMDAwMDc3OSAwMDAwMCBuIAowMDAwMDAwODAwIDAwMDAwIG4gCjAwMDAwMDA4MjEgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwNDAyIDAwMDAwIG4gCjAwMDAwMDA2MzkgMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDAwNjE5IDAwMDAwIG4gCjAwMDAwMDA4NTMgMDAwMDAgbiAKMDAwMDAwNTQ2NiAwMDAwMCBuIAowMDAwMDA1NTQ3IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gMTUgMCBSIC9Sb290IDEgMCBSIC9TaXplIDE2ID4+CnN0YXJ0eHJlZgo1NzA0CiUlRU9GCg==\n", "image/svg+xml": ["\n", "\n", "\n"], "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["pl.seed_everything(44)\n", "for _ in range(3):\n", " z_init = flow_dict[\"multiscale\"][\"model\"].prior.sample(sample_shape=[1, 8, 7, 7])\n", " z_init = z_init.expand(8, -1, -1, -1)\n", " samples = flow_dict[\"multiscale\"][\"model\"].sample(img_shape=z_init.shape, z_init=z_init)\n", " show_imgs(samples.cpu())"]}, {"cell_type": "markdown", "id": "8cb113a3", "metadata": {"papermill": {"duration": 0.054683, "end_time": "2021-09-16T12:42:05.399905", "exception": false, "start_time": "2021-09-16T12:42:05.345222", "status": "completed"}, "tags": []}, "source": ["We see that the early split variables indeed have a smaller effect on the image.\n", "Still, small differences can be spot when we look carefully at the borders of the digits.\n", "For instance, the hole at the top of the 8 changes for different samples although all of them represent the same coarse structure.\n", "This shows that the flow indeed learns to separate the higher-level\n", "information in the final variables, while the early split ones contain\n", "local noise patterns."]}, {"cell_type": "markdown", "id": "256310bb", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.054446, "end_time": "2021-09-16T12:42:05.508976", "exception": false, "start_time": "2021-09-16T12:42:05.454530", "status": "completed"}, "tags": []}, "source": ["### Visualizing Dequantization\n", "\n", "As a final part of this notebook, we will look at the effect of variational dequantization.\n", "We have motivated variational dequantization by the issue of sharp edges/boarders being difficult to model,\n", "and a flow would rather prefer smooth, prior-like distributions.\n", "To check how what noise distribution $q(u|x)$ the flows in the\n", "variational dequantization module have learned, we can plot a histogram\n", "of output values from the dequantization and variational dequantization\n", "module."]}, {"cell_type": "code", "execution_count": 33, "id": "665bc626", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:05.628077Z", "iopub.status.busy": "2021-09-16T12:42:05.623860Z", "iopub.status.idle": "2021-09-16T12:42:05.658084Z", "shell.execute_reply": "2021-09-16T12:42:05.657594Z"}, "papermill": {"duration": 0.093163, "end_time": "2021-09-16T12:42:05.658187", "exception": false, "start_time": "2021-09-16T12:42:05.565024", "status": "completed"}, "tags": []}, "outputs": [], "source": ["def visualize_dequant_distribution(model: ImageFlow, imgs: torch.Tensor, title: str = None):\n", " \"\"\"\n", " Args:\n", " model: The flow of which we want to visualize the dequantization distribution\n", " imgs: Example training images of which we want to visualize the dequantization distribution\n", " \"\"\"\n", " imgs = imgs.to(device)\n", " ldj = torch.zeros(imgs.shape[0], dtype=torch.float32).to(device)\n", " with torch.no_grad():\n", " dequant_vals = []\n", " for _ in tqdm(range(8), leave=False):\n", " d, _ = model.flows[0](imgs, ldj, reverse=False)\n", " dequant_vals.append(d)\n", " dequant_vals = torch.cat(dequant_vals, dim=0)\n", " dequant_vals = dequant_vals.view(-1).cpu().numpy()\n", " sns.set()\n", " plt.figure(figsize=(10, 3))\n", " plt.hist(dequant_vals, bins=256, color=to_rgb(\"C0\") + (0.5,), edgecolor=\"C0\", density=True)\n", " if title is not None:\n", " plt.title(title)\n", " plt.show()\n", " plt.close()\n", "\n", "\n", "sample_imgs, _ = next(iter(train_loader))"]}, {"cell_type": "code", "execution_count": 34, "id": "0e0a4cb8", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:05.770898Z", "iopub.status.busy": "2021-09-16T12:42:05.770412Z", "iopub.status.idle": "2021-09-16T12:42:06.769321Z", "shell.execute_reply": "2021-09-16T12:42:06.769715Z"}, "papermill": {"duration": 1.056909, "end_time": "2021-09-16T12:42:06.769866", "exception": false, "start_time": "2021-09-16T12:42:05.712957", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "b632bd849f444d76b9cdb1ce9b458ecc", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/8 [00:00\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["visualize_dequant_distribution(flow_dict[\"simple\"][\"model\"], sample_imgs, title=\"Dequantization\")"]}, {"cell_type": "code", "execution_count": 35, "id": "c3deeaa6", "metadata": {"execution": {"iopub.execute_input": "2021-09-16T12:42:06.894994Z", "iopub.status.busy": "2021-09-16T12:42:06.894523Z", "iopub.status.idle": "2021-09-16T12:42:08.443397Z", "shell.execute_reply": "2021-09-16T12:42:08.443791Z"}, "papermill": {"duration": 1.612603, "end_time": "2021-09-16T12:42:08.443933", "exception": false, "start_time": "2021-09-16T12:42:06.831330", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "e14299cd737b4a32ae911b93b2592618", "version_major": 2, "version_minor": 0}, "text/plain": [" 0%| | 0/8 [00:00\n", "\n", "\n"], "text/plain": [""]}, "metadata": {}, "output_type": "display_data"}], "source": ["visualize_dequant_distribution(flow_dict[\"vardeq\"][\"model\"], sample_imgs, title=\"Variational dequantization\")"]}, {"cell_type": "markdown", "id": "37caab54", "metadata": {"papermill": {"duration": 0.062935, "end_time": "2021-09-16T12:42:08.571623", "exception": false, "start_time": "2021-09-16T12:42:08.508688", "status": "completed"}, "tags": []}, "source": ["The dequantization distribution in the first plot shows that the MNIST images have a strong bias towards 0 (black),\n", "and the distribution of them have a sharp border as mentioned before.\n", "The variational dequantization module has indeed learned a much smoother distribution with a Gaussian-like curve which can be modeled much better.\n", "For the other values, we would need to visualize the distribution $q(u|x)$ on a deeper level, depending on $x$.\n", "However, as all $u$'s interact and depend on each other, we would need\n", "to visualize a distribution in 784 dimensions, which is not that\n", "intuitive anymore."]}, {"cell_type": "markdown", "id": "2cc36677", "metadata": {"papermill": {"duration": 0.062538, "end_time": "2021-09-16T12:42:08.696736", "exception": false, "start_time": "2021-09-16T12:42:08.634198", "status": "completed"}, "tags": []}, "source": ["## Conclusion\n", "\n", "In conclusion, we have seen how to implement our own normalizing flow, and what difficulties arise if we want to apply them on images.\n", "Dequantization is a crucial step in mapping the discrete images into continuous space to prevent underisable delta-peak solutions.\n", "While dequantization creates hypercubes with hard border, variational dequantization allows us to fit a flow much better on the data.\n", "This allows us to obtain a lower bits per dimension score, while not affecting the sampling speed.\n", "The most common flow element, the coupling layer, is simple to implement, and yet effective.\n", "Furthermore, multi-scale architectures help to capture the global image context while allowing us to efficiently scale up the flow.\n", "Normalizing flows are an interesting alternative to VAEs as they allow an exact likelihood estimate in continuous space,\n", "and we have the guarantee that every possible input $x$ has a corresponding latent vector $z$.\n", "However, even beyond continuous inputs and images, flows can be applied and allow us to exploit\n", "the data structure in latent space, as e.g. on graphs for the task of molecule generation [6].\n", "Recent advances in [Neural ODEs](https://arxiv.org/pdf/1806.07366.pdf) allow a flow with infinite number of layers,\n", "called Continuous Normalizing Flows, whose potential is yet to fully explore.\n", "Overall, normalizing flows are an exciting research area which will continue over the next couple of years."]}, {"cell_type": "markdown", "id": "d5a736a3", "metadata": {"papermill": {"duration": 0.06309, "end_time": "2021-09-16T12:42:08.823219", "exception": false, "start_time": "2021-09-16T12:42:08.760129", "status": "completed"}, "tags": []}, "source": ["## References\n", "\n", "[1] Dinh, L., Sohl-Dickstein, J., and Bengio, S. (2017).\n", "\u201cDensity estimation using Real NVP,\u201d In: 5th International Conference on Learning Representations, ICLR 2017.\n", "[Link](https://arxiv.org/abs/1605.08803)\n", "\n", "[2] Kingma, D. P., and Dhariwal, P. (2018).\n", "\u201cGlow: Generative Flow with Invertible 1x1 Convolutions,\u201d In: Advances in Neural Information Processing Systems, vol.\n", "31, pp.\n", "10215--10224.\n", "[Link](http://papers.nips.cc/paper/8224-glow-generative-flow-with-invertible-1x1-convolutions.pdf)\n", "\n", "[3] Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. (2019).\n", "\u201cFlow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design,\u201d\n", "in Proceedings of the 36th International Conference on Machine Learning, vol.\n", "97, pp.\n", "2722\u20132730.\n", "[Link](https://arxiv.org/abs/1902.00275)\n", "\n", "[4] Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G. (2019).\n", "\u201cNeural Spline Flows,\u201d In: Advances in Neural Information Processing Systems, pp.\n", "7509\u20137520.\n", "[Link](http://papers.neurips.cc/paper/8969-neural-spline-flows.pdf)\n", "\n", "[5] Hoogeboom, E., Cohen, T. S., and Tomczak, J. M. (2020).\n", "\u201cLearning Discrete Distributions by Dequantization,\u201d arXiv preprint arXiv2001.11235v1.\n", "[Link](https://arxiv.org/abs/2001.11235)\n", "\n", "[6] Lippe, P., and Gavves, E. (2021).\n", "\u201cCategorical Normalizing Flows via Continuous Transformations,\u201d\n", "In: International Conference on Learning Representations, ICLR 2021.\n", "[Link](https://openreview.net/pdf?id=-GLNZeVDuik)"]}, {"cell_type": "markdown", "id": "d586cc6a", "metadata": {"papermill": {"duration": 0.065018, "end_time": "2021-09-16T12:42:08.951852", "exception": false, "start_time": "2021-09-16T12:42:08.886834", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "{height=\"60px\" width=\"240px\"}"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Tutorial 9: Normalizing Flows for Image Modeling\n", " :card_description: In this tutorial, we will take a closer look at complex, deep normalizing flows. The most popular, current application of deep normalizing flows is to model datasets of...\n", " :tags: Image,GPU/TPU,UvA-DL-Course\n", " :image: _static/images/course_UvA-DL/09-normalizing-flows.jpg"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab,id,colab_type,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7"}, "papermill": {"default_parameters": {}, "duration": 19.664801, "end_time": "2021-09-16T12:42:09.722798", "environment_variables": {}, "exception": null, "input_path": "course_UvA-DL/09-normalizing-flows/NF_image_modeling.ipynb", "output_path": ".notebooks/course_UvA-DL/09-normalizing-flows.ipynb", "parameters": {}, "start_time": "2021-09-16T12:41:50.057997", "version": "2.3.3"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"0989ef9cef0e410d8ba8708d70a44485": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_bd437d01e4504868a0eab569e54fc653", "placeholder": "\u200b", "style": "IPY_MODEL_ddf110132d7a4b79a55b81c2445d20f3", "value": " 5/8 [00:00<00:00, 34.55it/s]"}}, "121d677b3fe64b379404376f171236bb": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "20eced8f26104528a837da146da0288f": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "212476674e8742c5a36d5a6a59b8d2cf": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "21d9114cc8a54f54998866b9b4c3f542": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "301d6460224d44a69468ee761437b216": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_20eced8f26104528a837da146da0288f", "placeholder": "\u200b", "style": "IPY_MODEL_52759cf555ba4bb79cbb72b597401e1c", "value": " 0/8 [00:00<?, ?it/s]"}}, "3420b49f1b7245029561a3537f3672e5": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "360f088f7aa34336bba895d5688f18f3": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "52759cf555ba4bb79cbb72b597401e1c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "5cac1945d4b94e45a8cb5128a7462916": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "63e7d0109110426fad97b7d711fa9919": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_e7fb1721fcec4c4eab30b9ff09edbb4a", "max": 8.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_212476674e8742c5a36d5a6a59b8d2cf", "value": 8.0}}, "9caa927d490648dca2d1d7c54e448858": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_ee2e2fb65a534444bfa51ad704a04f35", "max": 8.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_121d677b3fe64b379404376f171236bb", "value": 8.0}}, "a26ef8f75ac44f1c9a6872f079431d43": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_21d9114cc8a54f54998866b9b4c3f542", "placeholder": "\u200b", "style": "IPY_MODEL_edb3a164cc18459694b2c1dbb050b44a", "value": " 0%"}}, "b632bd849f444d76b9cdb1ce9b458ecc": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_a26ef8f75ac44f1c9a6872f079431d43", "IPY_MODEL_63e7d0109110426fad97b7d711fa9919", "IPY_MODEL_301d6460224d44a69468ee761437b216"], "layout": "IPY_MODEL_3420b49f1b7245029561a3537f3672e5"}}, "bd437d01e4504868a0eab569e54fc653": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "ca544fad59514bc6b35ed43510f6bb85": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "ddf110132d7a4b79a55b81c2445d20f3": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "e14299cd737b4a32ae911b93b2592618": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_f797208673e64a9aac664d03c1b76f3a", "IPY_MODEL_9caa927d490648dca2d1d7c54e448858", "IPY_MODEL_0989ef9cef0e410d8ba8708d70a44485"], "layout": "IPY_MODEL_360f088f7aa34336bba895d5688f18f3"}}, "e7fb1721fcec4c4eab30b9ff09edbb4a": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "edb3a164cc18459694b2c1dbb050b44a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": ""}}, "ee2e2fb65a534444bfa51ad704a04f35": {"model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "f797208673e64a9aac664d03c1b76f3a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_5cac1945d4b94e45a8cb5128a7462916", "placeholder": "\u200b", "style": "IPY_MODEL_ca544fad59514bc6b35ed43510f6bb85", "value": " 62%"}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/comparison_GAN_VAE_NF.png\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/uniform_flow.png\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/normalizing_flow_layout.png\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/image_to_gaussian.svg\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/dequantization_issue.svg\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/coupling_flow.svg\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/vanilla_flow.svg\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/Squeeze_operation.svg\")
![](\"https://github.com/PyTorchLightning/lightning-tutorials/raw/main/course_UvA-DL/09-normalizing-flows/multiscale_flow.svg\")