{ "cells": [ { "cell_type": "markdown", "id": "33ee711d", "metadata": { "papermill": { "duration": 0.009299, "end_time": "2021-07-26T21:27:21.119445", "exception": false, "start_time": "2021-07-26T21:27:21.110146", "status": "completed" }, "tags": [] }, "source": [ "\n", "# How to train a Deep Q Network\n", "\n", "* **Author:** PL team\n", "* **License:** CC BY-SA\n", "* **Generated:** 2021-07-26T23:14:45.695289\n", "\n", "Main takeaways:\n", "\n", "1. RL has the same flow as previous models we have seen, with a few additions\n", "2. Handle unsupervised learning by using an IterableDataset where the dataset itself is constantly updated during training\n", "3. Each training step carries has the agent taking an action in the environment and storing the experience in the IterableDataset\n", "\n", "\n", "---\n", "Open in [![Open In Colab](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAHUAAAAUCAYAAACzrHJDAAAIuUlEQVRoQ+1ZaVRURxb+qhdolmbTUVSURpZgmLhHbQVFZIlGQBEXcMvJhKiTEzfigjQg7oNEJ9GMGidnjnNMBs2czIzajksEFRE1xklCTKJiQLRFsUGkoUWw+82pamn79etGYoKek1B/4NW99/tu3e/dquJBAGD27NkHALxKf39WY39gyrOi+i3xqGtUoePJrFmznrmgtModorbTu8YRNZk5cybXTvCtwh7o6NR2KzuZMWNGh6jtVt7nA0ymT5/eJlF9POrh7PAQl6s8bGYa3PUum//htmebVtLRqW0q01M5keTk5FZFzU0oRle3+zxwg5Hgtb+PZiL/ZVohxCI+hL5JgjmfjPxZ26+33BG3dA+ealHPM4gQAo5rU59gsI8bRvl54t3Ca62mvHyUAhtOlLd5WSQpKcluBjumnoCLs1EARkVd9E8l3p9y2i7RbQ1B6pFwu/YDgW8KbHJHMTQrwnjz2oZm9M4pavOCfo5jWrgCaaMVcMs6/pNhDr0+AMN93XlxV7R6DNpyzi7W/OE+yIrsjU6rTrbKV5cd/pNyItOmTbMp6sbBB+EqaYJY4cWE3VUciNt1TpgfcRFv71Fi54xT5kSoyLvOBEJMOMxWXkFlBeBSX4u6Zkcs+3KszYRtiapbNRqF31UgetVuc8z9vBXIv1qD+F1f83B6uDlCUyfsZGepGPpmg01OB7EITQbhS9ribKy+DmP1DUiClLz4bnIHVOqa7BY+Z1wg5g3zgUvyehiNpnJKxSLc/ts76LKm0BzX3c0RNy1yXjDcB5lWoro4iNHQxM+f1kWeWQARAWQS++trISJTp061Kep25X/MycwtjuctSC5rxo7ppi7VNUox5+PhPHtrsS2O1qJ6yx1QujQUzm9sh6hbkBlvvGcN8hYnwjUjH6kjfZEd5c/jitz5Jc5U3ENnFynKl4eB7nyEgP2UZ+Yz3/rVEbyYr27qELrtC4FIC0J7sc7xWnmccdHfRRTs0VB+cA4lt+oFcRR/wUeH8FG5w2Mbx8FQ8TXEvv1xYf4wBP3O2WyL3/UVjpXWgIqaFeUPr+wTmDvUB7njH6/bOv+HRg4SqioAg5GDe1aB3ZeMTJkyRSBqkLsWqSEm0fZVBEN94zEZnYvrdx1JL5cxe+a+AbhSJecRRHW/ikTFRTa38dtQlNZ5CRKwFvUtZU/kvBoEF9Uxni/XqIM+dwKbTw3rhcxIf7gmr2M+H6SMwx8iBzJbw5oxeG3Lv5FX9B3AGaHPS8e8z77H7v9VMpvPG5ug1enh7eGK8h0LBTwUb+GInqzInlRUK65DmTPQu4c3+uQKjwKK77zwUxBX4Tq7yR1RuiwUsqlrABCM6esHdXoy47fk4+prYKy8ZF574x4V5BnHQBuf4g9Z9ld8U36L2aktZNNplNfw7zotwWTy5MkCUft4aLEopJj5/OPHl1BQqeAVOnHgNSQOqmBzq9V9cfEm/yx5ubMGKS9cYPZ3vx2OS/c6PVHUuUO7Y1Pci3BO/1zgq18byebfGemLtNF+6JRtOvMk926ibussZqM+1mNz4TWkH7rCbM5phwGRGDAaoF8fY5OHFnlldAA8sgoEXKnDukA1NgSeNjqkJT9brbN4pC9WRweYXyLugR73c+MYvyWfu0yC6+mjzN1Isfw3FKJS98CU/zI1IHFkFPR52cHL2FJk0sB6kMTERIGo9GzcPkLNfA0cwdwi/hfEYO86ZMd9w+y1egfM2T2Eh/vesMNwljSzuZRT420SW3eqy8N6aHMmwmnFUZ7/PGVPbIoNZvNU1BURdHs0bT2+HjL8sDSM2e6vi4Lj5NW8WOLVA6RTT2azxLV+bglaFNqLieqemS/gWkw7NyoAHo+2dEsiivengjKsPFoqWOvbSh/kxPaxyW/JRzH2Fl3EzD9/xjAefJqB3usKUFn/0Gb+S/d/jy3FN2yLOmnSJJtn6oehByEiHPSeXnDxFGPRnoFoaBJjcdQlbDwcjL1zTNuQpoxD7R0OG0uUTMi0fkVwdzBdYIwcwZunxrVJVLplNm54BZp7jfDfYLoNyqQi1K6KxIdHzmN+QQ2WjFIwUT2zTGdlRXo4NFXVUO4sgX5dFC7f0aP/ZlNeUjFBuL8Xjl6uRuP6aMjSjpjzsH62FDU7JhBuGccEXIvDfJFFBc/gHw80dklfCVYnRaDfpiJcutPA4F7qJsfJeUPQI+1fqMlNhFx1FM0GDqkjFVg7NojlQ0Vt4aM5ReSqcbpaCg8nCW5lRsBvbT4T1TLfFptsfh7gItzuKTdJSEiwKSrt1vcmnEXXrsLbYnWDA1bu+z2WKy9Arq+1KRqdfKsoBo0GcdtEpS/B1bO4v0cFiUhkjskvKcMrWwtAPHuwQq8Z+4LZ1vTQANfXt4J0DwZX9gWa9qh4XDM/voC9JXfwYEMMHJcfNtusn82ihvliVUwg5KrPGVf6GH94ZJpEZBen6EC4qYTHA1dXhW0JIex8txzv//c8lhzXIi/BFxOH9jGbQhZsRalTIBZZ8KkGyZAxeRQvXkFF1TWz/Hm46jNYUnjPbt3JxIkT7f6dSj8qfJJyVvBxgaIlblOyjtysNHWN9fjjqWi7glJfW3/S0Hlj2XnA8PhKT9w6g3Qx3XiXhvuxQsuT1proxBKI/AaZqY1Xz5muvY8G8XkRRCaHsfQsRAFDH/tZPbcYuHotOG0FRIqB4HR3wNVoIPLtz8ycTguu+jpEigE218vd1YCr5m+HpHMvEI9u4LTXwNWaLjl0iPwGAmIpeHx1VeCqTJdPs1/vweweQPO3HC24NhOhnTphwoQnfv6QSY2ICbkNmdSA4h87oaLaiYfn5diIEd4att2erOwJXbPUHp953p6orQVSUVWRAXBT8c/dJ5L9xhzaJGp71GR/wFP8P5V2z10NSC9T93QM2xUg8fHxT+zU9ijeU4naHon8CjFJXFzc8/kn+dN06q9QgF98SYSo2Xen2NjYZy5sR6f+4nLSK5Iam2PH/x87a1YN/t5sBgAAAABJRU5ErkJggg==){height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/reinforce-learning-DQN.ipynb)\n", "\n", "Give us a ⭐ [on Github](https://www.github.com/PytorchLightning/pytorch-lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/latest/)\n", "| Join us [on Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)" ] }, { "cell_type": "markdown", "id": "f9c378db", "metadata": { "papermill": { "duration": 0.007663, "end_time": "2021-07-26T21:27:21.135216", "exception": false, "start_time": "2021-07-26T21:27:21.127553", "status": "completed" }, "tags": [] }, "source": [ "### Setup\n", "This notebook requires some packages besides pytorch-lightning." ] }, { "cell_type": "code", "execution_count": 1, "id": "621d9c7d", "metadata": { "colab": {}, "colab_type": "code", "execution": { "iopub.execute_input": "2021-07-26T21:27:21.157525Z", "iopub.status.busy": "2021-07-26T21:27:21.157055Z", "iopub.status.idle": "2021-07-26T21:27:23.530688Z", "shell.execute_reply": "2021-07-26T21:27:23.530184Z" }, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": { "duration": 2.387983, "end_time": "2021-07-26T21:27:23.530821", "exception": false, "start_time": "2021-07-26T21:27:21.142838", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mWARNING: Value for scheme.platlib does not match. Please report this to \r\n", "distutils: /usr/local/lib/python3.9/dist-packages\r\n", "sysconfig: /usr/lib/python3.9/site-packages\u001b[0m\r\n", "\u001b[33mWARNING: Value for scheme.purelib does not match. Please report this to \r\n", "distutils: /usr/local/lib/python3.9/dist-packages\r\n", "sysconfig: /usr/lib/python3.9/site-packages\u001b[0m\r\n", "\u001b[33mWARNING: Value for scheme.headers does not match. Please report this to \r\n", "distutils: /usr/local/include/python3.9/UNKNOWN\r\n", "sysconfig: /usr/include/python3.9/UNKNOWN\u001b[0m\r\n", "\u001b[33mWARNING: Value for scheme.scripts does not match. Please report this to \r\n", "distutils: /usr/local/bin\r\n", "sysconfig: /usr/bin\u001b[0m\r\n", "\u001b[33mWARNING: Value for scheme.data does not match. Please report this to \r\n", "distutils: /usr/local\r\n", "sysconfig: /usr\u001b[0m\r\n", "\u001b[33mWARNING: Additional context:\r\n", "user = False\r\n", "home = None\r\n", "root = None\r\n", "prefix = None\u001b[0m\r\n" ] } ], "source": [ "! pip install --quiet \"gym\" \"pytorch-lightning>=1.3\" \"torch>=1.6, <1.9\" \"torchmetrics>=0.3\"" ] }, { "cell_type": "code", "execution_count": 2, "id": "eac80594", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:23.552854Z", "iopub.status.busy": "2021-07-26T21:27:23.552379Z", "iopub.status.idle": "2021-07-26T21:27:24.701512Z", "shell.execute_reply": "2021-07-26T21:27:24.701070Z" }, "papermill": { "duration": 1.161732, "end_time": "2021-07-26T21:27:24.701630", "exception": false, "start_time": "2021-07-26T21:27:23.539898", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "import os\n", "from collections import deque, namedtuple, OrderedDict\n", "from typing import List, Tuple\n", "\n", "import gym\n", "import numpy as np\n", "import torch\n", "from pytorch_lightning import LightningModule, Trainer\n", "from torch import nn, Tensor\n", "from torch.optim import Adam, Optimizer\n", "from torch.utils.data import DataLoader\n", "from torch.utils.data.dataset import IterableDataset\n", "\n", "PATH_DATASETS = os.environ.get('PATH_DATASETS', '.')\n", "AVAIL_GPUS = min(1, torch.cuda.device_count())" ] }, { "cell_type": "code", "execution_count": 3, "id": "0bfffc03", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:24.724549Z", "iopub.status.busy": "2021-07-26T21:27:24.724078Z", "iopub.status.idle": "2021-07-26T21:27:24.726193Z", "shell.execute_reply": "2021-07-26T21:27:24.725793Z" }, "papermill": { "duration": 0.01604, "end_time": "2021-07-26T21:27:24.726293", "exception": false, "start_time": "2021-07-26T21:27:24.710253", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "class DQN(nn.Module):\n", " \"\"\"\n", " Simple MLP network\n", " \"\"\"\n", "\n", " def __init__(self, obs_size: int, n_actions: int, hidden_size: int = 128):\n", " \"\"\"\n", " Args:\n", " obs_size: observation/state size of the environment\n", " n_actions: number of discrete actions available in the environment\n", " hidden_size: size of hidden layers\n", " \"\"\"\n", " super().__init__()\n", " self.net = nn.Sequential(\n", " nn.Linear(obs_size, hidden_size),\n", " nn.ReLU(),\n", " nn.Linear(hidden_size, n_actions),\n", " )\n", "\n", " def forward(self, x):\n", " return self.net(x.float())" ] }, { "cell_type": "markdown", "id": "0f0e84d9", "metadata": { "papermill": { "duration": 0.008295, "end_time": "2021-07-26T21:27:24.742720", "exception": false, "start_time": "2021-07-26T21:27:24.734425", "status": "completed" }, "tags": [] }, "source": [ "### Memory" ] }, { "cell_type": "code", "execution_count": 4, "id": "aeab7bc1", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:24.762210Z", "iopub.status.busy": "2021-07-26T21:27:24.761745Z", "iopub.status.idle": "2021-07-26T21:27:24.763836Z", "shell.execute_reply": "2021-07-26T21:27:24.763445Z" }, "papermill": { "duration": 0.013053, "end_time": "2021-07-26T21:27:24.763933", "exception": false, "start_time": "2021-07-26T21:27:24.750880", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "\n", "# Named tuple for storing experience steps gathered in training\n", "Experience = namedtuple(\n", " 'Experience',\n", " field_names=['state', 'action', 'reward', 'done', 'new_state'],\n", ")" ] }, { "cell_type": "code", "execution_count": 5, "id": "c8a2fcb5", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:24.786200Z", "iopub.status.busy": "2021-07-26T21:27:24.785725Z", "iopub.status.idle": "2021-07-26T21:27:24.787827Z", "shell.execute_reply": "2021-07-26T21:27:24.787369Z" }, "papermill": { "duration": 0.01577, "end_time": "2021-07-26T21:27:24.787925", "exception": false, "start_time": "2021-07-26T21:27:24.772155", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "class ReplayBuffer:\n", " \"\"\"\n", " Replay Buffer for storing past experiences allowing the agent to learn from them\n", "\n", " Args:\n", " capacity: size of the buffer\n", " \"\"\"\n", "\n", " def __init__(self, capacity: int) -> None:\n", " self.buffer = deque(maxlen=capacity)\n", "\n", " def __len__(self) -> None:\n", " return len(self.buffer)\n", "\n", " def append(self, experience: Experience) -> None:\n", " \"\"\"\n", " Add experience to the buffer\n", "\n", " Args:\n", " experience: tuple (state, action, reward, done, new_state)\n", " \"\"\"\n", " self.buffer.append(experience)\n", "\n", " def sample(self, batch_size: int) -> Tuple:\n", " indices = np.random.choice(len(self.buffer), batch_size, replace=False)\n", " states, actions, rewards, dones, next_states = zip(*(self.buffer[idx] for idx in indices))\n", "\n", " return (\n", " np.array(states),\n", " np.array(actions),\n", " np.array(rewards, dtype=np.float32),\n", " np.array(dones, dtype=np.bool),\n", " np.array(next_states),\n", " )" ] }, { "cell_type": "code", "execution_count": 6, "id": "ce6fab64", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:24.809140Z", "iopub.status.busy": "2021-07-26T21:27:24.808674Z", "iopub.status.idle": "2021-07-26T21:27:24.810365Z", "shell.execute_reply": "2021-07-26T21:27:24.810736Z" }, "lines_to_next_cell": 2, "papermill": { "duration": 0.014597, "end_time": "2021-07-26T21:27:24.810849", "exception": false, "start_time": "2021-07-26T21:27:24.796252", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "class RLDataset(IterableDataset):\n", " \"\"\"\n", " Iterable Dataset containing the ExperienceBuffer\n", " which will be updated with new experiences during training\n", "\n", " Args:\n", " buffer: replay buffer\n", " sample_size: number of experiences to sample at a time\n", " \"\"\"\n", "\n", " def __init__(self, buffer: ReplayBuffer, sample_size: int = 200) -> None:\n", " self.buffer = buffer\n", " self.sample_size = sample_size\n", "\n", " def __iter__(self) -> Tuple:\n", " states, actions, rewards, dones, new_states = self.buffer.sample(self.sample_size)\n", " for i in range(len(dones)):\n", " yield states[i], actions[i], rewards[i], dones[i], new_states[i]" ] }, { "cell_type": "markdown", "id": "5bf5f70e", "metadata": { "lines_to_next_cell": 2, "papermill": { "duration": 0.008338, "end_time": "2021-07-26T21:27:24.827809", "exception": false, "start_time": "2021-07-26T21:27:24.819471", "status": "completed" }, "tags": [] }, "source": [ "### Agent" ] }, { "cell_type": "code", "execution_count": 7, "id": "497f6af6", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:24.853026Z", "iopub.status.busy": "2021-07-26T21:27:24.852318Z", "iopub.status.idle": "2021-07-26T21:27:24.854718Z", "shell.execute_reply": "2021-07-26T21:27:24.855093Z" }, "lines_to_next_cell": 2, "papermill": { "duration": 0.019026, "end_time": "2021-07-26T21:27:24.855209", "exception": false, "start_time": "2021-07-26T21:27:24.836183", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "class Agent:\n", " \"\"\"\n", " Base Agent class handeling the interaction with the environment\n", " \"\"\"\n", "\n", " def __init__(self, env: gym.Env, replay_buffer: ReplayBuffer) -> None:\n", " \"\"\"\n", " Args:\n", " env: training environment\n", " replay_buffer: replay buffer storing experiences\n", " \"\"\"\n", " self.env = env\n", " self.replay_buffer = replay_buffer\n", " self.reset()\n", " self.state = self.env.reset()\n", "\n", " def reset(self) -> None:\n", " \"\"\" Resents the environment and updates the state\"\"\"\n", " self.state = self.env.reset()\n", "\n", " def get_action(self, net: nn.Module, epsilon: float, device: str) -> int:\n", " \"\"\"Using the given network, decide what action to carry out\n", " using an epsilon-greedy policy\n", "\n", " Args:\n", " net: DQN network\n", " epsilon: value to determine likelihood of taking a random action\n", " device: current device\n", "\n", " Returns:\n", " action\n", " \"\"\"\n", " if np.random.random() < epsilon:\n", " action = self.env.action_space.sample()\n", " else:\n", " state = torch.tensor([self.state])\n", "\n", " if device not in ['cpu']:\n", " state = state.cuda(device)\n", "\n", " q_values = net(state)\n", " _, action = torch.max(q_values, dim=1)\n", " action = int(action.item())\n", "\n", " return action\n", "\n", " @torch.no_grad()\n", " def play_step(\n", " self,\n", " net: nn.Module,\n", " epsilon: float = 0.0,\n", " device: str = 'cpu',\n", " ) -> Tuple[float, bool]:\n", " \"\"\"Carries out a single interaction step between the agent and the environment\n", "\n", " Args:\n", " net: DQN network\n", " epsilon: value to determine likelihood of taking a random action\n", " device: current device\n", "\n", " Returns:\n", " reward, done\n", " \"\"\"\n", "\n", " action = self.get_action(net, epsilon, device)\n", "\n", " # do step in the environment\n", " new_state, reward, done, _ = self.env.step(action)\n", "\n", " exp = Experience(self.state, action, reward, done, new_state)\n", "\n", " self.replay_buffer.append(exp)\n", "\n", " self.state = new_state\n", " if done:\n", " self.reset()\n", " return reward, done" ] }, { "cell_type": "markdown", "id": "a0c997f1", "metadata": { "lines_to_next_cell": 2, "papermill": { "duration": 0.008391, "end_time": "2021-07-26T21:27:24.872244", "exception": false, "start_time": "2021-07-26T21:27:24.863853", "status": "completed" }, "tags": [] }, "source": [ "### DQN Lightning Module" ] }, { "cell_type": "code", "execution_count": 8, "id": "6a4d50ff", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:24.905411Z", "iopub.status.busy": "2021-07-26T21:27:24.904908Z", "iopub.status.idle": "2021-07-26T21:27:24.906984Z", "shell.execute_reply": "2021-07-26T21:27:24.906522Z" }, "papermill": { "duration": 0.02641, "end_time": "2021-07-26T21:27:24.907082", "exception": false, "start_time": "2021-07-26T21:27:24.880672", "status": "completed" }, "tags": [] }, "outputs": [], "source": [ "class DQNLightning(LightningModule):\n", " \"\"\" Basic DQN Model \"\"\"\n", "\n", " def __init__(\n", " self,\n", " batch_size: int = 16,\n", " lr: float = 1e-2,\n", " env: str = \"CartPole-v0\",\n", " gamma: float = 0.99,\n", " sync_rate: int = 10,\n", " replay_size: int = 1000,\n", " warm_start_size: int = 1000,\n", " eps_last_frame: int = 1000,\n", " eps_start: float = 1.0,\n", " eps_end: float = 0.01,\n", " episode_length: int = 200,\n", " warm_start_steps: int = 1000,\n", " ) -> None:\n", " \"\"\"\n", " Args:\n", " batch_size: size of the batches\")\n", " lr: learning rate\n", " env: gym environment tag\n", " gamma: discount factor\n", " sync_rate: how many frames do we update the target network\n", " replay_size: capacity of the replay buffer\n", " warm_start_size: how many samples do we use to fill our buffer at the start of training\n", " eps_last_frame: what frame should epsilon stop decaying\n", " eps_start: starting value of epsilon\n", " eps_end: final value of epsilon\n", " episode_length: max length of an episode\n", " warm_start_steps: max episode reward in the environment\n", " \"\"\"\n", " super().__init__()\n", " self.save_hyperparameters()\n", "\n", " self.env = gym.make(self.hparams.env)\n", " obs_size = self.env.observation_space.shape[0]\n", " n_actions = self.env.action_space.n\n", "\n", " self.net = DQN(obs_size, n_actions)\n", " self.target_net = DQN(obs_size, n_actions)\n", "\n", " self.buffer = ReplayBuffer(self.hparams.replay_size)\n", " self.agent = Agent(self.env, self.buffer)\n", " self.total_reward = 0\n", " self.episode_reward = 0\n", " self.populate(self.hparams.warm_start_steps)\n", "\n", " def populate(self, steps: int = 1000) -> None:\n", " \"\"\"\n", " Carries out several random steps through the environment to initially fill\n", " up the replay buffer with experiences\n", "\n", " Args:\n", " steps: number of random steps to populate the buffer with\n", " \"\"\"\n", " for i in range(steps):\n", " self.agent.play_step(self.net, epsilon=1.0)\n", "\n", " def forward(self, x: Tensor) -> Tensor:\n", " \"\"\"\n", " Passes in a state x through the network and gets the q_values of each action as an output\n", "\n", " Args:\n", " x: environment state\n", "\n", " Returns:\n", " q values\n", " \"\"\"\n", " output = self.net(x)\n", " return output\n", "\n", " def dqn_mse_loss(self, batch: Tuple[Tensor, Tensor]) -> Tensor:\n", " \"\"\"\n", " Calculates the mse loss using a mini batch from the replay buffer\n", "\n", " Args:\n", " batch: current mini batch of replay data\n", "\n", " Returns:\n", " loss\n", " \"\"\"\n", " states, actions, rewards, dones, next_states = batch\n", "\n", " state_action_values = self.net(states).gather(1, actions.unsqueeze(-1)).squeeze(-1)\n", "\n", " with torch.no_grad():\n", " next_state_values = self.target_net(next_states).max(1)[0]\n", " next_state_values[dones] = 0.0\n", " next_state_values = next_state_values.detach()\n", "\n", " expected_state_action_values = next_state_values * self.hparams.gamma + rewards\n", "\n", " return nn.MSELoss()(state_action_values, expected_state_action_values)\n", "\n", " def training_step(self, batch: Tuple[Tensor, Tensor], nb_batch) -> OrderedDict:\n", " \"\"\"\n", " Carries out a single step through the environment to update the replay buffer.\n", " Then calculates loss based on the minibatch recieved\n", "\n", " Args:\n", " batch: current mini batch of replay data\n", " nb_batch: batch number\n", "\n", " Returns:\n", " Training loss and log metrics\n", " \"\"\"\n", " device = self.get_device(batch)\n", " epsilon = max(\n", " self.hparams.eps_end,\n", " self.hparams.eps_start - self.global_step + 1 / self.hparams.eps_last_frame,\n", " )\n", "\n", " # step through environment with agent\n", " reward, done = self.agent.play_step(self.net, epsilon, device)\n", " self.episode_reward += reward\n", "\n", " # calculates training loss\n", " loss = self.dqn_mse_loss(batch)\n", "\n", " if self.trainer.use_dp or self.trainer.use_ddp2:\n", " loss = loss.unsqueeze(0)\n", "\n", " if done:\n", " self.total_reward = self.episode_reward\n", " self.episode_reward = 0\n", "\n", " # Soft update of target network\n", " if self.global_step % self.hparams.sync_rate == 0:\n", " self.target_net.load_state_dict(self.net.state_dict())\n", "\n", " log = {\n", " 'total_reward': torch.tensor(self.total_reward).to(device),\n", " 'reward': torch.tensor(reward).to(device),\n", " 'train_loss': loss\n", " }\n", " status = {\n", " 'steps': torch.tensor(self.global_step).to(device),\n", " 'total_reward': torch.tensor(self.total_reward).to(device)\n", " }\n", "\n", " return OrderedDict({'loss': loss, 'log': log, 'progress_bar': status})\n", "\n", " def configure_optimizers(self) -> List[Optimizer]:\n", " \"\"\" Initialize Adam optimizer\"\"\"\n", " optimizer = Adam(self.net.parameters(), lr=self.hparams.lr)\n", " return [optimizer]\n", "\n", " def __dataloader(self) -> DataLoader:\n", " \"\"\"Initialize the Replay Buffer dataset used for retrieving experiences\"\"\"\n", " dataset = RLDataset(self.buffer, self.hparams.episode_length)\n", " dataloader = DataLoader(\n", " dataset=dataset,\n", " batch_size=self.hparams.batch_size,\n", " )\n", " return dataloader\n", "\n", " def train_dataloader(self) -> DataLoader:\n", " \"\"\"Get train loader\"\"\"\n", " return self.__dataloader()\n", "\n", " def get_device(self, batch) -> str:\n", " \"\"\"Retrieve device currently being used by minibatch\"\"\"\n", " return batch[0].device.index if self.on_gpu else 'cpu'" ] }, { "cell_type": "markdown", "id": "5d09b815", "metadata": { "papermill": { "duration": 0.008511, "end_time": "2021-07-26T21:27:24.924163", "exception": false, "start_time": "2021-07-26T21:27:24.915652", "status": "completed" }, "tags": [] }, "source": [ "### Trainer" ] }, { "cell_type": "code", "execution_count": 9, "id": "77233e6e", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:24.944963Z", "iopub.status.busy": "2021-07-26T21:27:24.944501Z", "iopub.status.idle": "2021-07-26T21:27:40.167092Z", "shell.execute_reply": "2021-07-26T21:27:40.166617Z" }, "papermill": { "duration": 15.234387, "end_time": "2021-07-26T21:27:40.167206", "exception": false, "start_time": "2021-07-26T21:27:24.932819", "status": "completed" }, "tags": [] }, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "GPU available: True, used: True\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "TPU available: False, using: 0 TPU cores\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n", " | Name | Type | Params\n", "------------------------------------\n", "0 | net | DQN | 898 \n", "1 | target_net | DQN | 898 \n", "------------------------------------\n", "1.8 K Trainable params\n", "0 Non-trainable params\n", "1.8 K Total params\n", "0.007 Total estimated model params size (MB)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "/home/AzDevOps_azpcontainer/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/data_loading.py:102: UserWarning: The dataloader, train dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 12 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.\n", " rank_zero_warn(\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "292eb170164e439ab3c69c33cde369f8", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Training: 0it [00:00, ?it/s]" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "/tmp/ipykernel_728/3502201211.py:32: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.\n", "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n", " np.array(dones, dtype=np.bool),\n", "/home/AzDevOps_azpcontainer/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/deprecated_api.py:70: LightningDeprecationWarning: Internal: `use_dp` is deprecated in v1.2 and will be removed in v1.4.\n", " rank_zero_deprecation(\"Internal: `use_dp` is deprecated in v1.2 and will be removed in v1.4.\")\n", "/home/AzDevOps_azpcontainer/.local/lib/python3.9/site-packages/pytorch_lightning/trainer/deprecated_api.py:92: LightningDeprecationWarning: Internal: `use_ddp2` is deprecated in v1.2 and will be removed in v1.4.\n", " rank_zero_deprecation(\"Internal: `use_ddp2` is deprecated in v1.2 and will be removed in v1.4.\")\n" ] } ], "source": [ "\n", "model = DQNLightning()\n", "\n", "trainer = Trainer(\n", " gpus=AVAIL_GPUS,\n", " max_epochs=200,\n", " val_check_interval=100,\n", ")\n", "\n", "trainer.fit(model)" ] }, { "cell_type": "code", "execution_count": 10, "id": "2d1234a3", "metadata": { "execution": { "iopub.execute_input": "2021-07-26T21:27:40.192086Z", "iopub.status.busy": "2021-07-26T21:27:40.191606Z", "iopub.status.idle": "2021-07-26T21:27:41.734141Z", "shell.execute_reply": "2021-07-26T21:27:41.733727Z" }, "papermill": { "duration": 1.556234, "end_time": "2021-07-26T21:27:41.734258", "exception": false, "start_time": "2021-07-26T21:27:40.178024", "status": "completed" }, "tags": [] }, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " " ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Start tensorboard.\n", "%load_ext tensorboard\n", "%tensorboard --logdir lightning_logs/" ] }, { "cell_type": "markdown", "id": "de74debc", "metadata": { "papermill": { "duration": 0.010636, "end_time": "2021-07-26T21:27:41.756088", "exception": false, "start_time": "2021-07-26T21:27:41.745452", "status": "completed" }, "tags": [] }, "source": [ "## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/PyTorchLightning/pytorch-lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://join.slack.com/t/pytorch-lightning/shared_invite/zt-pw5v393p-qRaDgEk24~EjiZNBpSQFgQ)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/PyTorchLightning/pytorch-lightning) or [Bolt](https://github.com/PyTorchLightning/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/PyTorchLightning/pytorch-lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/PyTorchLightning/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "![Pytorch Lightning](data:image/png;base64,H4sIAAAAAAACA9xbWZeiSJu+71+Rk7cMxb716eoziiIKioqCctOHJdhk3+GPzf38ssHMrK7srqrvVA/pXHx5TipLxMMbT7xrSPzyy/3vt/9YKPzpul8++VUc/f7Lb/evp8hMvM/PIHl+cszKhO00Sgs4Th3w+dmsq/TtchR4fgVXPojH6y8nbzccs7h9uX4/fv79l6en33xgOveD8TAGlflk+2ZRgurzc125MPvaJgqS21MBorFfUsJZAVxQ2f7zkz8efX72qyorf0UQL6j82vr0+mWWI0j5yU7jfwJhNqOgRfmGUZegsNOkAkn1T4FeAWA7SmvnU0l8MmNzSBOz/ccS3WWAg9j0wA+kQka08e8LoF2kZZkWgRck46wkadLHaV0+P8XACczxShQ9PwVjZ68Iqv7zc+mbFIbDsyGksROjHSWVQ8jWu6lVdhyCJaHPZsft4nJRgmFF88asO/oaMGJqRfa3rEgODenrQYLsijgI7YQSD/7yxteywJ4g+fD58/PrAMuqj0DpA1D95Jwhr4eIW5gxaNPiVsIoQbCu6VAEcCiSNEnLcV3WxlEOwy2aI5lPdjkOE7lT+6pNE+gohy1gvC7JChYlLyXuYc3yIh+vLr8cTLy+xqzh3HjC313zmeuie98MQhy70nPmhHJ8fhKDG2MdvZg+DUswEx0CYyj8A+iwgG82QVqUsEUQBAYAQG2aMWnLJUiAMRZBswRl2SRlv2PjjY+vH1OYscJhsBY9cgIqaVW8DO0HilhrrHYKcKdAitvAnxKSFG7rlVw1JplnmdQCpqICTzeggT1tjTNFz0qkGmbJUl1tvYLNvenMvNkbDQjbJWibsC2XdABpsTRwUNZyWdd2OecrLXd7Ke0iyKofEuGMBlmMxnn/+h4V/ACu5Iw+BSvSXRmqbpxZoYsHQYjtg5Dus1khoP5NRJENojJnmQTAQWRpedkeLu6pDVZUv9zM1xA3d6QVKx02O8u31y9KUvXZ3aVmWRTYZhWkCRKOjulV3OensrB/mhaQNEGRJvHoK2DUIhiOBeinsHz+/TfkFe7N806kQglzHZzOa2leSfmN3yfY5eZSkltbiqhsKm4uM1jqG3kbbknfgy5IEqGls7yAPluQRUmgPOTvnbOEzbG8PxQ1TxuE334sFbZfJzf4nUMhTMLkaIZ7AB9nFVkf/GbFgZDU3HA5MxIIEmv8cvYu0jlFiGufJXm4x27hjteHbrXpdy29UQUATra6piR+wQdIsHAu1hAwAXmakYz6warxykcDEmcM4qPQrs2S+DdcfICdiDJlNvv0Rsw3fC56jOvXTqj7vZCz1NycH0yWC6lqkwueXmJGCnA8pl13fquik7dbWu1ZH9QZoC/qRUVDYPBIbCfn2ceS8dWxYrYFKNOyv8fEdC6oSpfUgon92aD6pNpR16PgSW7lrDqpFaVEWV1bahF1oejfDtr5eBXOxPxGrbfKaivVs2BzinhCuVy3RDes7D4c6qVw+AnFeMm/xkytjgAcOJ+fP71Nv5mMqUUF4DSBx6ZpFN2H/dr6n+vSN2AwoB2KNUnzASaGbo20aIn0kkg9BYDaHrhuVUhn0u8OgL90cd/5CViLO3+xzmu20X02FsuSEyKeoS4OYu31TljwjaOkFFMlpyNJJ9ZPaNWPmLTHNDgOiiItplD4FQV2xtBOs6b7AO62tI6UK1mpNxd+fWshJ11E9BAxGy7HF3HutG514Kt+GRhb5LzZrY7awemldo3sZ9tugeElQFq+Iwy735RcMqApqqjaT7jrH3P3pZAop5H3JwxMECZludYjgh2DiUaNnaiNnh93AtcWMb67GLP8qjVDZ7CLJi2oMxUhcr2cWyt+fqEYNelp9BhC+dkwZs1yvhKiXTlgDLdYJrgotJ42SfOSsXKogmnUvWLAwKUwh7OsB/C2x7kzGu1i/xwSFKEVvJ7R9uD4u2xbCQzaaZJLrkZqNIlP81XVZWjZt7Mti5A0FPDozRGCY7PFiOW8WSQ5nfb6sQ0maV1SFYFV3zuUcJkFzlhpeYWZ+ROZ/AEqTLi0S5HOt0F2Ordm4UZbwdYGbXY193jCxIx3i/a8r5dayS+q495scN6irVwGG5fNPCsN9Q2xLIF0Wcv8One7CGlos5g7tcGpcpxtl8YEbh0QmT1w4Cg1nSDxYBCBew46hdgfQMK0aTGAIh5h6XSIpKoA3Swd6kG12zdrmiW1YVZz2VB7tBbiPkTUmOzw9dE/kmFSB6LhgoMUlUTS2SdEJUQoXVeMUAXeEjFBGkKeN4HVwvTGjzSbxOMXEBiYhAtYkn0Ac5qqOtB1CPBgq0JZYF1pf4HoTp91s6WTWpSAH7dSV0v2TFTxZSZcN3twuqRaV6G7i4cSlRvizs3E42UlqlATZlfVXEzQx7GmrWA/TW9wCewCVB+hkD/ChCkKJznHfURhcVEsw1unYd52HAluhcngbK3tT+JYQykE05KQs+MXu3NhXI7JKhaOdlA5fayn6jJmhspu/MFKOzG+KOoeP4F0r7oHakL+eOdgMoeUPeaJHPsICzaDfLterYxVys4O29BO17G0355KQ9OOEe1XvHpl5tkxzlb7FTITID2H9ogSIk7l6wdlfeNsfrfdL82SUfBhdqTTPEGrCbEaxGkYwFlg38ao8BE6+B280SPi5n1l7AF85vnRq64A0uaASw+NdLqqM67WLCLnJX+5jYhSdMpkz84s7ejWcagqtXfilzvsKsdJEuzm24vFMFWPt45jRZakmoclNkH/XDOIrHQShW8QsGmaJI1a1ANY2wUXkYF20KrTGNtcEBIVL3TxzLidpG7p8zE6s1ARM+jKVzQGP5fKqQ1JEMciREr0qkjYbW4prp0leTKj5yzhEZSITvCGbhBV91XlJKunUfcOByZonLIZ8Ih6xcBau1eEgzj0W3WfZ9RClorUKojFDj/Qi4HAbHdftaSFN/uS21/QfdH1i4uYnax1ZBtcfsK22Hw36zNRklpWzuhrh0zQuiAp7z9rlNOzxb8iwTRD2ZhtPyIi8/mJWQibmAVM3zXlYmnhM0POnKPYCyoHciOrab/ZkNqtoXtLO/mpSVXeqel3h4XRz/We1qCbLuadQ3WEejtlorhcTdDBP0ee16DoP4TDFyQYHe14LPy+XcCZzmEUp+Wq7wGEdMrufCg5tV3tEJNzD81ZDWbCjRILmQWyGIJa6bjFhTvplH07SsICooUtRTZ1JbhIPI93go6gR0LaLOeTOGyC6qX5pLr5HQzM0WP0sLFH1H/ktlsd/HKhLBrDk62aMguF5s3MTXaNQbmXbWmQjBDcorN0jMWcnBM+W5/Sm7rWZ8qsKxt+eyyMAl1oBhqgXkwcZ0k5IZsO6ziDq3QKc28QMEBtm6S/s3z6AZl05ZjtFlHUsoYcusEp40qf7YOJrFbyTK3UpRqWubNsOwj0LLvaXVe3A06m0EYQUjK6Qkwrb48uY/KF3N1mlJrX83Q3wfdFZp3Y/stC1UckMN+BgynaYSjTfkT+cnRqU541wRzvveYoIeBCBe1wVUgiAEzZFBu0uG5KlO6YS54dI2Qtsfv9vLTn+pbbpillsPRCwoOjvMQh88icNtTZxiewGYOqCOyxgIiAfe/0EZT+CBMmKcfiRut+xJqYb6AElniBQNDtqnTttO95/ZS31y68QDk4yRWvMfwGWitQO9wGPsaO6ZHQB90eXD1u5V1eQGJN3A6Kd5ubq4RMKn0Cr0laBe5bBzgKygp2U7ue5CV/AAkDgNEM6Twi74lNVFF6IgnRMSvI5x06v9nxWijl3FYNaLuub4euXwqjd9BPg5GtoMPWU1J9PxONgSS2tNuHs/MKC0XaRmh5HwTSnu8nxJusSMeUD4z1xhgpPkBTv4cHc5zpoIB4RATyvVSV5FMks2tirRfISVZ2/BnakTeHr5xC5Iuidsid5HvHSC5OLXo57/18bi4c4xwI6rA4HFskEQ7YrcU1cIpRj4OiCdXg2/g/gkKWRk0Mxx+ROZIXj1gvdLq+BcZCrY6cF0o1v24G1Nh2yu1azxd0icVUwqhNofArbOuxdSJst4a4KvVlIHHpLGOJijeWm0HD2kwZZuUE2y6A6cRgDLv2R+jgt2gwwBiWYB5SPyNue8pWeACteE9cGjZCUAKUXRzVb7Cju9l2fdDZp3U3dAunjxRd2zThWk/r+Kov9yHjnfO5U+gVqQoux3LqEHcRMCdYdAHct0Ax7Xer9ziw67g2RpjkA/hbINtuHlbHfQhKetEk11WAO0uRmR2WidyozUlZmNbypqYQNuxpVV0amcWmAoXfBH15nHFMhckY5wrInrASMvPW5XZfT7DgApTZmDYHDYDrxAFFFCQgMZtpVH4XEkZdlyAwlH4Aq6reE7SKs5sSQq5D2Vy9OIPWS4dH82prz126vPqX09IOMVPY8/JeOSmJIuWDEOx6EU932sKBauoQsCC/rnfqusZZWZhi4XWSvPxqktbZpJj9FyCY5GjbYsAj1mVJus3XlUbtA7oZ1tzhJi3C1mv7NR7V+0CK5AwKHf0cAaw9z/Th7BPzCwEhixj4u9oktRNqNI1dtspa3eq2LisXXnEm2HUJigCUcGVa08LLexwYEKaNm/gjqpzqxtAgVeVzflE1g53thn1WFMo6FtYaNhwQba5B+rBijdDItL3I2vJtKyJ1QYmbORZ1HYf0KaThbbgQOLoic/fcGvIEuy5BA+4CwrYZ2XVk3t3aBwSbfwELWzQJXJN5xLsSIbO3MBHCLxV5XUtqsZHlSxchu93tlukIQu3lU1ZT7Ga1YLP8uFcWi3Aj5aFCX/eGA0HzOaltakvch0NTby++jnChnU9hNy1eNerLazeTOP07GMy6FuFYziPiTyKZxNmZ4QqjAXKT4KRnznG0P7ubRmXaKHHYA8tDgb5gA7y6yJ4pgXih32QAOnBwihIyE44V+IhylaBHN1AwJFA1wVNWfgHAp9cbU0h8jwNztklbDIU9gD9dYrVLS0TuebTpo44OkSftfXt7zkW0PiVzRBrA/rx3DJ+PETHr3MtFlE6rJqR4VAEdAw3Weosv+eEsW9WJP5G1R/QT1oCq1CwnWfQLAEyxpotR1CNqQKxRjabNijHyl2wbqjthJ6y0WDjhS/IarvEde6M3eaBaOLtaXbe35ebsNyCt4maOogZpHYwtjRleo9sXs8POxJZkPXmKxrVgHH3qBHU8ibevMLBDuyTHWY9gb+46GaImSRzOh1Djz7G3l9P9Ai82V2pYdldFgObFDHOxjNnV0EUNlJPcJ+tV6jpNn/XlatQ1ei1uHdbzl+Z8WJilOZsQl+ukAcGkl+xeEWCUchmac51HvLyqCDsd6eyuT7JIc3n2SrgHRxACs4xr73o+i2dyx6ErtsznxkrG+iPB9KFm3jqNRQSNcsB8JnecctjO08AyEUIRV4sJ0eJlB0RZmVVdwmVtxdN+cf4WDaZQgJEO9YgVB1uqEejCeCdI10SyvEgV6hErhrHcg2t7bV0cFWsLORV76aN9zIGlvoYQU5d8zbRorbzWTVOvqfmpCKSLf3E3ralenAnW2wLLi+DWLCa9QvIVBWZQ07Lc77zr8AEvDjfNUTeKQxMcNAtC1zMGEnO0Vs49q8hoyOJMdjaNCMMtIO5uura76oaKXWc7LTNRkawufHATZzeFF3GjEl38tlib5vIngsU/IaQAWVoGYx53T5Mt4JIUTf+9RJucHKvJuXQiK7e0iGsxfLE/y+7sOmCkxC9qXssGG7FIX6PI0DhBrHyW8nm3ouL9ctgNcu/7WCLNQA108doT5aZBF7L2M4sG/4QHJ3DdkQAK5xzS+nvm8MuXvV2Jed/71QSgzcas7fnpbRvT5+c2cCr/swOawAbwy8nzFyWqgioCv2f9yLHtv24uS4LEQ6LUSz9lifdkVk/xGIRB8fQ///2070/3dvKfzb7peG+1Ciqxtn5DXrHfbT57FdABr5KPhLyT8eSDpxeUFtw/vzzqqS1GAsenu2nx5I934PFkPI7NxAZPs/XTWNADc2z46Ukd037w1Kd18XR/SzUq//MpSaunagS20iACRTaWBODTE/z0ccN9/v3dnp/XPTYv0nxv8tMMJK933zEAdXH0ZTPOuxafXi6/EPj5+ZXPv29rc6M6cODAvrP43c08d/15aXRvcx/cdwFfJiYrxkcXd6NwrV9Hmf8InHdTg5EoRrIszREkQVM4+67j64zeRzl6wqpORjMdT971Hc/uLhIjGYYYQypK/blt6j1A1QbVyPmvLzvxfh0t4x3Cl1H96Qv6f71hD2NYGqcZHEUYDnM4kkXhuxODMQxYMEuZDMyxDMtYKIYxqHMX53uSjE8C74T4r9dn/ai1bRbvCSvrODaL/o/ILDzwx4u0P+r5MiPvun6gav7gif9W5vdXZfqqxqn36xvt/+969K0Qv5pR9W9B9XcGeLeTP+4K9m6Ab+7l+x3ufvFd29QKgV39qO3/g3H85YF1EX1HY9750/8z8r+d1b1sih5ziL8Gpdd85Wd3l75mIeP//wIAAP//AGYCmf08bWV0YSBuYW1lPSJyZXF1ZXN0LWlkIiBjb250ZW50PSIwODA1OjAxRjU6MjQzQkRERTozQzM2RjlCOjYwRkYyNUM1IiBkYXRhLXBqYXgtdHJhbnNpZW50PSJ0cnVlIi8+PG1ldGEgbmFtZT0iaHRtbC1zYWZlLW5vbmNlIiBjb250ZW50PSI0ODk5YjYyMWE3NTBlNjQ0N2M3M2Y3ZDU4MTJiMTRmN2IwN2VkMTdlZGI2MDIwYzkyMjdlZWZhOWQ0ZDEyMWU1IiBkYXRhLXBqYXgtdHJhbnNpZW50PSJ0cnVlIi8+PG1ldGEgbmFtZT0idmlzaXRvci1wYXlsb2FkIiBjb250ZW50PSJleUp5WldabGNuSmxjaUk2SWlJc0luSmxjWFZsYzNSZmFXUWlPaUl3T0RBMU9qQXhSalU2TWpRelFrUkVSVG96UXpNMlJqbENPall3UmtZeU5VTTFJaXdpZG1semFYUnZjbDlwWkNJNklqVTFOall5T0RFME1URTBOakExTmpFek15SXNJbkpsWjJsdmJsOWxaR2RsSWpvaWFXRmtJaXdpY21WbmFXOXVYM0psYm1SbGNpSTZJbWxoWkNKOSIgZGF0YS1wamF4LXRyYW5zaWVudD0idHJ1ZSIvPjxtZXRhIG5hbWU9InZpc2l0b3ItaG1hYyIgY29udGVudD0iYWFkMzIwZGM5MTQ0OWQ1Yjk4NWMwZmQwODdmM2IzNzFkMGZjN2U1YmI1YjUxMDAyZTFhNzdmMDQ3NDU3NTI0MiIgZGF0YS1wamF4LXRyYW5zaWVudD0idHJ1ZSIvPuw9a3fbNpbf+yuw6ja7exJKfJNKYs1xkzjJNHGysZt2O52j5QOSGFMEw4dtpaf/fe8F+JYUSZacxLOdZkQSBO69uLhPEIC/+47A/x7PaeaQyJnTo96MXdLEcxJfSnP3A/UyKXOmPeKxKKNRdtRLaMzSIGPJ4qFi2aZqWqrcI76TOVL8wbmWssSJ0gCqjr777rs26GmQzXJXuqALl3EEM5ZkXp6lK8E/SFmeeFTymE9XIjjqZUkOrwYjRNRFltIQiKe+FAbRRY9cOmFOBfixALyO6C5DpoxNQyoBUVQC1gSTwHOygEUNoj3lIn8q/aS+ePn+xNSMcBF76S9v2VP9IgiZIX04Hl/9eLI41/PX/9MbdZmyGfxP58Y0tWfy1aXjTF/9dPz+l4+269LTq+i3335SEuXX/16kv2rXefLq55uA/+3T7P3i2cmVa11pVJbyN+dhNrf/nnoX6olxlr0PZk9len2lTtLjm4B//mtq/MR+/vni9Alzjn+LrKu3p1ImK28XV/HwtZY++xBlY2382y9vvd6oO4rMy1i4yAIvlWYszZo8ZyEOMEv6QqycOO57bI7isAYA1JACvwFCNPxMC3oJ9aQ8CRuNZlkWpw8HgzX4B4WQ0+uMJpETDtyEXaU0GXNYpbA28TlQS6AL2RLzBvfC7FEO7SWsfG+aPeIlKMZ1iRsydwDKdLWVmiyxOM6CefCJhgsJW0+CkDYI+OPex5xlj2BsU6BMPDwk4qqLy4PiMQGOMNDmstI//lm+yRYx9Y9zH8jx6Ir3TsSiBdLw8m35EmkuX8cJQ0P00m+jV0xLsyxTViy5TcilkwSOG67CNKFOlif0JHSmK97S6xhkeA4dr18WDEgzaJi2CXiXR1EQTdvInaKbL/0VCDhlOMRL8D9Dc9DptyrrmmposqKrbdRgWNs1cRRhVMTDnw/IjXAptq7qumVvwJUlwFpkXYHts+RbhqwqqrIBZEJBoQCkT/1xHDrR2In8sUsjOgmydBxE4zSYRnnchhI6C5p0RQVwqrasGYo5bNcGLZmAzToOS93rjgt0KMgWK+CtZgsQ+2byzommtGxgyLJcs34tuNUjugxOkRFeLc8MPJn/fkms/vizruMsKd4flXj4QauZuP7jd3Flye8lOX8Ud3Mn82a/l9V/L9XG8bKqanGDxqVb8d/B1oz9fD5fjJ0sSwI3z2i3HdqKbjsvTzM2X9+GO/fPIGu+//OfbRZ3xbNJY8G4dgNudltN3tQW9DlIZ+JA2EFKc0dgiMiPjndxBSFPSp6weQxD5QYhyEFXVaYJy+NV9rHs+NIQdqlXTFtVdUsGDdugXOhUxmXzWkCXAFqKroGNVQ19A8A0doBpG+HZimWYpqzJww3wWDJ1ouATl2uu/ptB24omy4rR9QdLoIN0HLLpFMxKsBnqULY021BsdRPBU8o2AVPBWRm6qYIl2mj7PuY0BUF6kicJyNGSrLgsOwlCCDIqD/SQTJwwrbym43ksj7b3mjxAWRKw2iWudGjL4jJUYIgVXd/UwdnCT9jYCwPvou876YynBP2MOt4MBDNjLHTZ9djLnC5Pd6XIVlRdgV7L2iYBzt15kPWbcpdiSBeHNKPc14xLZ7MPNbphD1XNtjZQE7ExBGlJ4PXBR3kXIKsYWwU+HbPJuI7Z9qfHsi3DMMxN/l2MFDBnzCI+ViB3Yxgr7yIM0qzv+P4Yo9L96bFlw9DUjfJTjFadL46DeQz5JAwZRiL0AISYCthRdTvGzGgY90PqJNF4zhI6dlwYrnGDOk5VFWDsRZfNQ4VNRg68RSayjr7PxhHLxmDyMWqCcNrPvcPQMlSGpmbqm3yD4BGdx9lijCLE03DQ7T7PUiag8DWnDkCUYSqqZW83cCuIEoJ0SJJURR7qhj7c5EI2KFkQXUKiDXZh7kI+dgCqFNkEVbsxo4SmHZZRhqWChO/JqDz2kTCQdExn96dKxRjIsjfFVBuo8oN0HqQHGDdNtzVVtc3dHdoYIvXgkt9Wrm1/egxVQ5e2vYNtiMyhzLVqgBtTDG2TX20QIfRozLWqYMmhaBkaqqqam0z0qgEqZLYQ4b1JMUwdgm1T3SS6lwG96nuhk6YJg1RLzJ6Om5QdgBTwqBABbxJboUUNWjDyAr/Fy/enwoLRgWxm09h0GFJnfXvh1lUZkont7EiNvLCzhx0NW5UNTVd2peVWJMMe6qYGvLkZX6qC/QkZKpqqQUBzM0ImQQIx10HJMcG2bpwo65IzhagqA1Pv0QN4G3B8iqbou1qQ3eJdcbM0NfgZsmxNMSFV2Oh08hi4AVlTkQ6vnEnYlSdD2dThn7Gl8jSnKfdBaoF308yNmaNAisNRddoNwhBjkdiZ7m3FgA5LMwx7Y4ZW5PYi6CnmatG9QRx5ABqGIJVDbbvQFefBIfAZp0416b0HbvBhkPTIG2eZStzz2Ena8ncAGnAqW9aU7WShqwMHmFUZ6pCiDzVl46xYERPHNKrQp1dBhrM8e9NggDfVZHujaSo1AiLOehwOkxsMIZIYWpBGbWkKeNfrgcDPbNf7q4OlycrQtjdO4hQB54zNKZqC8YEsE4QStiab6nbSeGjs0HHDUmR5Sy8pXDWAp0mcBCkdZ0ngLH0j25EKVYYg29Bsdbhl6LAc0Y0z6uwbMgAZ4JgMTd1ypiGI4jxb4sV4wpIDUGKCdsryxmnX2kQHUU7HoCCzA7HChLDSgpTwxiMySei+BgLIsE1TM01jO1dVuMmSHX38THOA3APIgFhSt4dbzop1ySi4U9itvYlRQFs0y9wpfqqJoXMn2F9hFUXBfGxL8ejSEEOce8WSpe9nu5OBOqsDMRvI4OE1KKl3oDgOMIP7NHFmcjuncWgRAI0AoRxuWsNQYC9DmNhZIJLDmCjFQh5oym4c2D31LfFtm92oMs4e6YZpbedNlhZKVGsjMPz2cgi/FyxPxhGl/r6xJ9Jm6Tp4uy1n2viysAXEW8HcSRZj/t15Bw2+Ae80HXPWjbNvG3mXAbm1b7w9cvGTuGXvPdTbO82bEDm0dQuq7ktk5ee39603oNYAq2ZBerKB2i6dBT0V5IRiztBd9maVWdeffFEdIY//TZLIOSNxwj+3EbTOZBI66Qxs9QOSzSipP9iSv58RrockY8SlJGRg1nwSRFhPLD6dUccfkcyZwnuwc5RDePrmNUmQ1CQlksTRpl4SxBnxEpamLAmmQYSrCXEpHcvTHvHphCZHPX7pAYKMTpMgWxz10pljKKp0/8PPhqycUTu+uK8Mj38JT2dn7wb5p+iE/vL85TDyz3801CfP6al98okOzpL5ffOFfHJqnjyJLs/8y/evsxdX6Xvl9PlHJf3VeHf549Wz58fTo6MewTU8QEoch8Vy0MEH59IR5PZImnj1MkqxXLJcQ5mmNOPf3AfidtBYnDhxNEMb+lr/Q9obPR4IaCOxnlGs2u0uZE4zvFta8MmXiY6W1vnyVZYhAy42WvTEKuZuXRRUvrh4E5Ll5cQ0wqV3vlSsRGyufX59/O6nZ+dvXx0/eTZ+++z06cvT5+OXp2fnx69eHZ+/fHN61lwfiwyU6Mc8uDzqXYsln8UizQZEY6KaNtV9Vxt6mkENiIkt1xoOKUQ+wE9Ttg1HVx1zgtMZQ8dXbfCHjuVSyKoMz7U5m0Qflhb+SuIz6cqeD94uzlnizV4F01mGayQH8SLDAiksSwjUJh0p2K4lCstnVwr7aEKQE1KxyKnJEVuzzaGhfGbVb6d5VyS6BG4FqblMoUlOYwn9blCiK/YZmpaZtiv8OHdBeRsoykXEO3U1xTjtogGFr1Daua80g1D7Ypwwlh2KfU2QN+Hld6Vq4/YCMMzhUc9D2wsGL+yRWUInSzZuO+kWq7nnTgpxx8BnXjoQ2xUGY1yCHHgDCKOmNB2AWLJ+jMRstfeiWIYuIZB0zZJ2Jw76DVoxZLuEgLdcwj7gbZc1rwRNk4QlN4UtGq8H3nAEEGPgsvy90NXgxgJcm4B6UGEgLqTAQ6u6clCXHFccRBGYdxRA8H399JJvngkZeOLvZf4/sYmiRuGEfKtARonAw7+hHPU+pGJbxcS5FOXCq/LhH/CB346gon1a3nCh6ZCwPWLoz/3r+ToR34wc+QE8bg4xRDdz3OYDPGqM5PcKVTXVgsrNuryWlHrYpFGZaw8oAtqaQSH8zTGMggnF3SOC6EFZAKEE7zcGUG+KAApsvgSJno8rtcFYdYkt3Xc5iSl0LuRRjgQh5DRH7exEEEDNgAd1/NZl/qLktViGKrE8IzS6lOqlWTyIBIuVxsC94JKKZ7QNEEBlixBIwekH6Spx4ofEhcz0QsKCR0VkU1gnP7gsUXHbJ8wgkgsgYaCRKtAuhAKhNykaQ0On4NX3oPRJJrGJVPSoEpP4WlJJvJB0Id6SO5WCaAJhQYRxCC1KcXpbupqBPBHcoSIB/gmDjBSRpxdBLGWsgjw6gwIMi4uCxwOnoieNnUhYOpwIkfCrCosajqkkKmFTYFoq7CGPrRNyFfjZTJrkYYhYG28k10nI26IJqRg0Ca6pX0aIHHHBcdfxLnCNduQLcX1IvreGrj2ZPOIoHhL5h0e9LagsUUrAljlZRTNSxqNcwA4aIK6CFQVHqri0Kik4JYa0RPVCDDALfVKMdUPkgBs+iHYQpnwMnCCChk9FCVmWl4mO4632SMIEN8DSJcKUNOSsgiRdh8SXwqk0Cek1wR/eX8DFk2oSN8S75HYDjt9o9yFPs2CykFxw25RGy8AqweWiW0CYJ5L+GV/cI7htApTWRSPxolDoYvymjsRz2aPef77iDCPAsP8igp0PyJShnJZG4AG33A/RJXMlnKMhGjWG5TGYPeA/WqmjnqZWggBeAluS4iphQ0mQuKQ/PYIy9SO7PurJRCaKCf+gTMT8EAr1IablYigw8L7NAh8MWSF9o8exk83IJAhDKclxBDFdZT4EVD4kHzaRn2h9Ay7wX3Fje/xWJ2pfHRKzb2jE6OsWsfrGsK/3ZbgaUl8RFw1awMNQ6suK1LdVflX6+lBS4a6vWXA1NEkUmFBtqEMVuKpYaItnaAY9gQIVChSAasLVELCgdd/U8JYofdlGEhVoDG36loq3WG5b8A/o1aCuCRRiaw4MXhhIDxQpfQsKVAmqIMqhuNH6QwN7ABU1rAhdBGpUToXM6/c1EwGo+AjlKmdV3wQcgBqwICUIUUHCNKivAiX4C73AIYP20GGiEnyhIDMAGmcudry89nUdXsJ/MML9ISIjiAyo7xsm1lMQIq+vGHyseJeg11rfMrAvBsG+wJD1VQNGr2/xMYShEFQgUzgPh4KZgBf7oQJKGF0YZQPG8tiGVxoRv9BPGcgBcZD0vo6MM2yJ//cJrRTKFRqry2lDDdF4N1WgpdpgFSBYBicBZg+EM/Iw/Gk+SOB2pjQrbTfkijFYnlHL1LXADwB+A+E6S7LGaHS83kDMVv0NHvBj+dEZPN7P43vOPH6EZSHzjoQ5vS/M6X2wDtVbNAlHP6gnP2hPqq2bP2jPREm1dVOUoEeHC/pG3l4E+wVwnrH0OkSSultBBEEOjwq8C1IzdWIsO9+ISTlOHmED4oKNQhch/DZ/gLoJBDzJgnD/Bg4CnHvh4RUCllTDn3QuGcv0cIPJ93OUNrOcz8PpsDGGj8UMmrg4OYR8EF55xaLLel3fA3GJnQW6waJRAazcoceXsFffPwqQGLAWDq4FqZV3Fy0iCAQe1JQs03f28vnp+Oe3LUBifg0IiKZjSDla9b9InteiprV5i3fozz97S+MgzeaOB/mFQ1VLparrOhZVFV/VqeHpFqXu0LS8iUxN356oskU91zSo7biGQi3DNgxHNw3Z87sD3tUaQlA97kVuGj/K465Kdc2Am2cZeLxCht0s4hv0G9I7z0DgGpGJMARlFiKat133OahgCCIOKca02EDNX9NriJpAlMtphzbhLZ+s6mt9cgYmgWI8lu7jkhHBzi4ZTbNlHMP/4R8aYUXBG3WG3sCpihUFrbrxQulW5s0/zWWyDMPaGoaFMF7zRoraaCDzBl0wsiyVYNa6BpQKMY5Nd8Htd9eUN824iL1e0ygn9a3UjGfbETxEaLEkkwRHGK4uA4xzuBFjLnxK2QCErwpzOzHrxthTlIApYXX6tOx9Gua5CZVGfmGOJUFgnU2lFDfookmOJa0Z4K5RoVvXmW015rqpKFUvOsqiQtyjH1JZDAwC8cdpybpsEvkVhGqKDPFlCEGRKuFPU/J5Jfx5pWj8VuX1SLee1KgIALFuKK2oKYuakqiJeAEkJw1CaMtuVpR5pc8EUW09WYpyYPyqbAeFHFy2hj8gazKZu5KBP/jQHv3n4HucsD24edgKz7jQ4rYViSe/XHSXA6YwqBs1wpBae7jQ4/RC9dhSnbboF09QqVJWCDr4D5RphPpTytujgq9KUBFFjayrueviQLLUL9TgIg1eMj6lkpFS2fDAnNBZVM+Q1daBK7jmFdBRl/I5fg9fAb980yji+g3DKmM4ppV8kjGq491tMb8ZFa7ETcgvswV5HmQvcvdvK99zTUdFja97ZFFc275OJ3aPXM/Dh2nsQMDai7HbCWTyXEGPelxeyt5xs+DN6CVcfHYFie5FNq0H0HFTFuYQwa2yx2u6AERyc8D95AMlNPvqA1TgB0qlTKu71vVFjReC8StfNg16wmLRCXRFlfEvA2cxTkIbXUkH4w2d0VdIa7OzFQdCOsGW/IK6o6/pfp2qND4kFnNgCwjbwxlO6YDxTME35mGWOJWQvAJhkqRiPUY7MZgZ5Md8HnN5kyR+FNQu8yEnJSmj8q6YPysoq2Gni7nLQmAeczKJe2jCvUXEkrkTNv1HlZfECXrBe4mTJI+KCbHGBGGHO7Ux65gwTI5gWLTGh92VAGrD1jI6IFsV4+fMFWfl7ML22qm3GD9ZwfjRa47g9ln4eBAG65RsW2aUUjhw+PT1rtK4C1uOBYY7xhc8xoybyttkzZMKyR3jDtB8UXw4uS3evC1Q3DHOQBdzXLZzi5w5K1DcMc6gRkm4RIteDW5Zp4hAc8cYFKRpDpdb5M1LjuGusYWvhHNu2029bKD5UhxaHdI8HuThaM2r9YESJI5yIyPLWExiHpNq6yPyLYaBf/HDlRbbcr7F6duLXc9KukYiMaoKvq50b8NScTwaX9zDkmAHJ/qFWPukoI+U9I26JV+RxUI5VpVXE5KdYpHmL02FrMDQGLydJjpWzUdUo43LvKsR7s4RtIaNzxWsnxrYYQTPEecIf1ekXF+s5/Va/C/d/2c15lF9v5YXf82S/UvMkj27jsHA0L9myOoX5QzZIabIIJpQcS7mC0yWfTbK2S+eoUJIvoHZt0JcwUKJm2KG92vGL5+LPGd6vWJsGVWTFH4/ZxHDQBsGTF0Xlr7C8/eIE/l8BaE4MvXxYKZ/eakAsgLvFrOLXXy3IGUkrl87V9sqnhVHq99ufrZL/NqgZ9R4+PYzgwx3jPEdC98AG89LYkbl3TfPwHJlT+i4/dbK0W+AndzYIROBuML04dMr56ta/J3YiqfCiFVP/Wke+Lf4LWUHvr4BooiginCqQOmXy/4fedUnLIrA4hE8mYOwbEaT9KZOVd7Hqa5YZpfAyM1v8xPc+YySd4Dk9TPcJ4B/heLOKFeDTeJo729Cu54JUkbiegeZOc+jW/0as1NcUlCDf2Mgn6OmtgruDHepnxdrgRubp78FDhcZ1LOSvmpyuCq5MzzGfVzpWv7yNPzrMvkMCRS7sXCysVUai9Kv73RXld9knvhLTJfiZiKaxaHjffH50tcN1KPGw18zpv/aM6ZvxflUd3fG9I5MmerFlOktLy0sjhurXYX7teY2C8Hqjd7i8alfYFXhF58xLHj9fbFMQRLnxgZpY2P+V4418Rhbwo+v5ZFm/XhnYqD6A2JzNqc4HfgbYTOnhZzhScU89a4fv370820sIdk+a9gU1e61/mGXpLfOH76JxOG21jy04T4eRE69WWVp/++6UwGaO1h4P4sX/B5K0X+1d66shDwPIknESnwjjFZthOnuIe4cX1DsvE2pk3izhh8uT0UpXqwJElMaTkADEpqVjee+KORASkKgsA5B4UEjqcdi6pfQBaZWEeD/gEKS8Q3By8clLBFUb0bD4+n+o/e/eGwcv388QF6C+UYZuZ7HI/6Gn4nDUHTwbM3RY/wtgbc7L+Gr8hQIUdTeXnTG90Zy/eCdkMQ+tHfVZtzWy8BvHeRUvyn7jmf8gIfcYk9tSQyHAcEUDuQKSCyZpsvgWo3zaFXDqrP8A9BuNOGfYwDr580goaXZUe/n8xMJwt05zWYMOID79Rq6wzlZsh/5LRV/zJXwM5uldE5aklod4YJbHmG4vFnC5jTkh4zwBvUBHyvblcK1QrBvehAH9IKjLnYhosy1txLv0DsBqKAR/AEN/YZG1AVtKS3PmWkW8poBkIt/rA3P8mnTJPZQs+yCLo7SB4PWO3EC0Md2g+IPi7cLebI7Y6GPBzCeCQFYRlPJ2IrqxazIilbr27Qro8HxnDjInDD4hEeiTSbtCkJ/wWm6zGXXncZ896WTxizOY+HYV9fp7hddrsHpKP7UlIC0olIx/GhqiiGF1A+cdbqibmlk1rG1hJDm4I9T/qVWwtQN1BX/Mko8+xgOntPsTLym/mm1B/YpVo+KlcEtwGlMw5D/Ja/VHW31scPppkb8HwAAAP//ALcASP88aW5wdXQgdHlwZT0iaGlkZGVuIiBkYXRhLWNzcmY9InRydWUiIGNsYXNzPSJqcy1kYXRhLWp1bXAtdG8tc3VnZ2VzdGlvbnMtcGF0aC1jc3JmIiB2YWx1ZT0icnJFVW1sbGVhNnlOUlYwZFU5R2dNd2FRQTBHR0U5L1F0WmUxNlAyWFVwMXZ4dUtuUVJSZXpCL3Y5SW9xRzViR2V3U2pneVRlZDcxczFFU2ZTOTRxMkE9PSIgLz7sfdmW20aS6Ht9RQ77tHvmtMHCvpRKNUeWLcszJdmnLWvunZeaBJAkYWFhAyBr8ZnX+aP5ofslNyIzASQWVpEl2VqaOioQyH2JjIyIjIg8Ie2/8yRfb2pS367Z09kqiWOWz0iU0qp6Ovu10qqkZlrFaBmtNEyjLRKWxjOS0wzSY8iMXJyQ3r/zJFuSqoygvLpeV2enp8ukXm3CufiBolldzaMiO00yumTVqSz/HbvVKqh5Na+2yxmhaf101rYlKzWTrBiNWakN088uTnpNOI+TbZPvm+KGrAvoRlLkGg2rIt3UjBRbVi7S4loTPSaxlhc5I79usrVWF1q1WUK7MEtFYBAmgrWoyGua5KycDbt/cr5Jm9qbcqfLqFm2TikMcL+wk5PzNOlKWKTshuADyoCMC+h0TctaBMH0ZNAYltesJGtNJwsH5mabLCnvMEZj7RNB4wbNSFmkMKvFmn9hU84pqWmY5DG7eTrTjHY28kLbQGCZQqOJ2kS6qYuJlk31fk3r1XRD2iil2cUapmmtmTOyKtkCASOmNeW1aAJ4lY6IKVHBoKmkiOoERluttwnira5WZZK/g3Hk8Fazm7rpgpjKdrbPAUYBHpPlCqDUcGfkOonrlXiVdTYFy1+tZOtiUMu4GSKVrIzUSY0z8jfGQbgob2FZlAnVUhqytB++Tdg1APvTmU50YrgEGwJQXsF4QKvmRjO7sDhnF+d8hBdJmmrlBkPZluVFDAs7fjp7ZRJz7jyDP/yF0nTDxt+VP/ccCn/wn4eK160BqdRgTbyvtF64YeiaMXdWmGurmSvNpwYxMIMOGQybQIRaiDHXPRMCdWyJHWBb8IlxEAwlbbXgLjP0uaMZbwMoLtLmluMSKM0NbMhsawaZm7r/FpshqzKg3JV/98oh0GjT2VrwoPAH/3lL5vbcTI257WD1vqdEGXOL6Jf+HAYW2tjPo83NrTYsCgId/FtBzCgC/t/BLJziNMAPwNJ7wNW6LH5lUf0waDUJB9D1kwjug1Yb+AHhyuCT+4z/GHKidZ2/ciB6ruPgmoY793yb18VTuiuMHGTDWJgs5y2GDONsPtYvMeAOahVRvcmUcyNAtxeOYDEMllP2clhMM8dvZU28BZayFnS5FrZeb4no2CZ9qw1CMTH+3WWaj7X0V5vI4wzWmiwJgu9e+dN1W5N1Q+gzJRTyfjiIFBv0wwAp0w3g8Wce2gfHJuxDQiMOhEdtQCsBEU/EUjD0QeADmPYjYPx5epibuW9CsGdTl7g8C+AKFxGGm1pz3Sb4UPEejyI8XsM4/pgc7PNT2LIkMcOpKDm+dAt7XSn2pJ2DquyfMn0zrpKSUsfzDaPZTBBp7YSa/qydZ3hv2jGxiypVISG4oxU8qqMKhjQX31xTtqgJUBPXJV2TqAKaqNzkEdBEvQ8NerNk9WxilJTWhUUJJAkAAJImMWD//lBFRVqUWrgEsqusE1rekvUNJBLBvDFtRJZibpeMybeQxsuGIFZogTXNFaJ5KrGoIWYLukk7DKxOSQJDskoqUnabekda/jCKbOo+xcoPb8kyLUKa7moITVNSLMj3Sf1yEyrNeAbhIvCe6nlZYpKBRSg3bNYRs0mOJKMWpkX0TgzzVqNpssy1DDKkQF/9v//5X7XQ0WTvLv13m/9Y49C81RZFtKnG4IBf7SD9G3yQupiakMd3H1/oxcn5aZrAWJyfblL8uY/PgDVVsgpgbSeb8QCTIcneMS0PHMZ9bA0n0mXj1c4r41wxaElMEcBfF0RmZTHnEmDoiGx4Nw6i30q3k7hDRjJ1g/nTpKrD4qaFCmSKMoEpFXaiHYpJ1qQZOSXVNAc3SvgxGLgIeKN4uJd+3qwc79JVD80eubkjN3fk5o7c3JGbO3JzR27uyM0dubkjN/dFcHN/NL9QXOd4hvblcQ28Y1dH3uHIOxx5hyPvcOQdjrzDkXc48g6fMe9QlEuaJ3dU0WlRuQc1+sg/HPmHP4h/ELDyRXEOoktHnuHIMxx5hiPPcOQZjjzDkWc48gyfJc9wPG848gv38wuNHhVR/rUJm08+mZhlUZTZxaCsbkx7ubvsyrJqbW9KlgI5vmWIh6ye0QxwCoIwP02LZZL/a8nqTZlf1cXTP5svfrp9UwDYXSJiyZN8CUHr2xqDtFQJgzEJ4SejFUwDvMRFVMFPVWzKiMHLFbBAsGvAm7A5gheorJiv8+WsNxKy2S+5pdErlm+gmvzdABhUHqafnXMWq9u4LLQoTaJ3T2e/ffX3TVE/wc2jvkJ2Q3yfiR/AbSuISCLOqsx5HhHztfhZ09u0oLHMJAsDAODpr5L8ag3d6RWJJlvSUopk0IFecR0GuEqaQvNNmn7dNWfcyJ9/+P711S8/9QoqygTmClqRL682ZdpLPzD7Qkuv4TSejibxFKfwVEzgKU7fqZi8Uzl1jbFYM2291mwqVvY79N//PRtNhrbKKGxZkeUvbNOL8GHQQGcssEJ3ETLqwtPSY9sLIicMqBd6LPS92PTjOA4Cz7MCN/LdiRlf0ma6//mygC0sJsWm/hcioOhrwuMg8GdYqyTJv+a711mFKzfJe2uBNGl6y4uqS3O4+tQFhEVu1rCCFldRTZ9iWX/drL+i2foJhgHcPBWQ8deUN/Ov0Mw2FiEJl9yfrec4nnwH/rP1nQhBwFFC5HqrVsU1zy/mShbOuVAl+Aq/n+6zkvtDu+dqHCDGnoBBYnWJvflHh6glrkdMbpL1rWYMqz+u5n+01fyIOc9pegs9quY8wcPTjSTpEqasPym46mGlqrlphFDRS8ZbhCrSlUhPYHsmNIoAkPs18/27l7VZ4WenX6ViiPly/mpZP+Eh7QLnIXz6cHU/kbjkTLawwSNnEibTFt19KQClu2ag2xaNmO+bzHP9yPBMz3R0GgZWFC1Mw3StRayzSNcjPbIXdsgYpb7uG0x3/LgPUEPrYDmMA+qrRfAKITYkuM5PxZA35Gy7EXByCzXiuZhXKxZcHR1gsaVtcSKRCuUUKLKOIiv+40VhAXw01r/SG8TeKfReKNlX2gJNrHsa7rzSxqpA4RdEyiZiQo7KExCRbAGzQMhvv4m418jtwaRcnPRzcNzcmJifh5u6LvJeaTB1RcWZLOVzJu3ZRfo+f/JtUmVJVQmGKGNVBWDTl8tO8AgPCw748GEyGKlsDXR/XjeZD5N+3DTjdq/4wZp7JsFHX8ajuyhmJO48sFNrbpooVzRVaUIrTLgM8M3nqcgwVSdzgMICKXoYpRNiViG7uMQqic+bhOINX02m8ySdzOKkJ7QQkyS+mmlA+ATQkNMDgKFAe7Ma2tXRAFwbKKCaJ07yKN3ETFuUdAnbZ61Aa17UyUJu1UBMsHShDVPLaQ1pBezvhEMDRF1qMdVpyGp6ygvD3g7Lu5BLDvt3otA3dL1Om5ZksMqIwCG8cqgjS2pthQKQiJZxpbGchimLuyRxUsG6RmC8NxlA/YZpNI+1dTmdUCEzOU7BcxhUp+Jv0kUEjAAMQBWtWEbnRbk8/blY1Ne0ZD9zfPy8iDueugFl3ieJTvg+w/FMh1IU5NOZ3DSNaWhdOalyCNtxbOpa3fDtTZMkaLgk61qzCKxjxnGNVkUlYznJQs3pHEaoJcjDKEhgIdrhjyzWbPxNl2quIVZrhFRZkmt8hQNVKn4XST1meCH7ymgxnHoGxgVZ4zOwsGT0nXYN5CtZWEL0lQNfTtPW0Gq7vPfcaMr4igsE90FtCs5SUdkYTR4Phj7Jg6GeFAl5gkKa9VWI86q6ZDUKxxHi1mWxbtI0sEWbrEDHkQV6CcFtVKaRRFSDTBpNTFVxYZgEigFOFZJUY7KvTdWc1Y6SzC6GIZJ6UqwF1f5mN60oT+nt5IKYXZyqhdRlkS+VUUHauO+OZjyI7ZC1CO3p7E+7kN6OHo5p39nFKKjrNG8mtzoEAtFoNr8+hdiGdBajiCsRVWqCy6hGPLWUXWYNdz0jVX2La3pN4xhacEbM9Q3RnwiseI6iRVlJCzB1UaR1sl4Df9C9ahUJ6xz/tCrrk2b/t9iQbFPVJGScwYF8KNYG9LWiOZAB6lZLKlYjd1FJeMyLRZGmxfWYrP8ITHu/oZsQtp6Ea0ZwFp5cU4CV92fkL3/8/uqH118K2xWHzHAcGnsMmKvQifTQYY5tGKHNYs+KfDOwYsewLWZZ+kIP48iLbeoZC88xHJO61uw9hcjqVvpxOQBYC2mPCcDN04caKeyg8hA28HEv8ZxorhueNjcseOiBJ3cHq9siIoz0Cd8ezLlhWDwaPp41hWHJyq5x7xYOaWELh2KJJbZwHbdwZ2sC9e9FOplbtgsV6ubc9bEm2DEDL8WW6hbu+o5LocE+/slNS9ct+Ern2Cldd7EMHbKLP1v+8Vwe/omtEUIwBnN5mHMF+3Xgim8s0sD0Lv616TVRVD/UxlCXV45vXr8a3jgNWyc6gISHY9OetoDBO6lBL9/2x4WP1N0rizgUBkgQCfDoRkp3oGT4C2A0bJgZK0i7YUKdBMfgZYiRgomb2z5QKNZLE8PoMB6aaLrYRMtNldY+UygEi3hY+VtnmhN7rTIyJ+0GI3XMFATfR/IdMuc/14BdNC6PImQH/t8T6aM0g1D1QPMTxPS8kYKDfRChG57vmq5n6v9AWN2n8YItIs/1A53qoRXqMYs8FlET/qJoAYFO7LqWZYSh7kRR4ARWZIUL04uZpdvhB8LqnwRe57DSnNhy0jMsAHAyZMSMvaQ+PuHcgMIquR7gBcOHJe/7KJ/xDYcAswERRk9SA4lQoGN6gdAicQHNBMAYGaijYgQjqY4PTFlgXPrI0Fi2h5k8F0sI/HFSYLSCdO6ZGhZlX0IjUARlefawBYik/BTbB9yUYVx6c8v05q7rK7pEBu/kXaYjZ2Y7ly6kBAw65AUd14YSeWegHzbycaYND8Pv14r6YC5sDZDDRPGU7mMqzw005Mfc/mCi5k4KsR7hkZDH0niWYZEaLxOr1LBKniqAUHvcSgjEcTSRfdziTjONfRu2ZTe0dej3ZwCkE8mqtLLjDiNXRZTQVCJh9OGpfB/AdZwivC7pHSyGRqSt4m3Ddm2dIDKoOKouEcvv0EABRt99J8V9VErxpvaVfXaV99tUFkX57pPfVIQSLwM+Ma1Ei487zO7zP4OFLg1c3bGCOIxj6pjMNeLIMHXGfM80DTOK7cCJGYMkegSsBHUsarquEYU6XXywHeYT2WW4qAGBhsV77SlIsprOQPqPL0PlUYGSDdOkKOMipiArhfgNCNu57/nP1Bhd5/l9LpxDFsEwfZ7VIOKJmXnetzIN7WfHF43L0zQsfHe1k1JCu6ehCluI25Z0l1kYDLuPmveejltaX0Zp6Lx6RQFW5+T+DsyO/17AjJyoB330AEycs/oa8p9mLAsRHU9heuXYsYemPduTWFoAxT1qgsA3vGvPIRXyv6cwNlAV4zg8p9u9JF67rA/uAf7uvEYeUG/5lpbA4qLbrmxeFeGpKiigSras0wLBTEP90V+ayNcQeZ+0H2VmD7dPLQ7IuviWoB8yjYvM+qYffMvbo0RxLNCY0uDhSVTEqMsaHiI5FBVBJn6g8HSW6F0pPKpiKYtq2EFRsQaqxVG+EriaA8lVXFznuH8JmLkSx1HyA7ZORismv2q6lG9hSfNo1YSvafQOkb0sj63T4hZPwyqyfw9WRf2O3T6dLUk0G6pcdZD0NXndmv0IpauvCR5Ika6/EkI5MMlj8mkwjTZA0uCs4Pa8D4j2QIAbTa1urlQYFIGNuvJuWG0trtq5GVpjNRGz3qmp+u++EyGEgB78t0ZMrdS3ylqp7x93PmSLM21zfKbNT5jN5sB6dPLdO7CeA76D/cQaH0Aj4+FoNucu3Dme6agYXSnCsPhhNmd6RnXpnT5+W55qICHr0rrKpg0jBiS/hMWnM36AeoHPvna0mkOaOrwuakK3QCXi+e0+IPocNwpU4WiKFuCDyF5g+vdFUfyIuToQSZ2KXGNcZajF7cZWIpXALnxzke9ZkrIKyGa2F6Lpt6LDN8mB+OYH0ZrDMM6nhF0eh1SEcgGaWgKJ8RGRixCTBHg2yy23GnLNIt03kmvW/iJvnfrEF8VAA5+JD5Tw6sI8zKeATYg7lxo2Fvx0ASgjgZADkYAAotmF+H0QEZhofHPA8of0vxcGWKOqRcn+Ds2uD0YEmHkCD5gThe5GB7wQAuRH9G6/ha/W2q379YHr/ifUZ2va+A+3/JcJ9EqZpU+RvvDmhuehCNCzLgFBeNbc9TzlgAQPaOa+Y2+BEEGJqGqpaZs892VTBnyrCXRUErHvXnG+sq+fIljDAYGARpqmNNLssbWGJXjsLRbtDtllyaG+dYBzNfvMNmdVkZNHQ1Jswkoz3toro1NRIQYU6u7kwXuaOboo4y6DXPpAPDBikkWjZOcN8+G+H4YLewtrdtH7fBAzOochRud3w4udjt7BWFHJOsaN1qjg3ZhRSboXYhxX3KHH5YHo8Vul7n805Ig8M/asG89PET0afPlPW7r7k4buzqSdu9RMUy3JHeTqLvFoHLCHbb8NBjxTY7oOOGvKQF4aqIuD95EDAGwzNuW5DmRfa/6vS/N//aW5hZ7ZFsUjHXGw40nxou3jt24BVtddxHQra274wdAfgEn8phHDKMSNxFhpPm8f6kLY/RFU9PWmtBbnTjQPXG4HzuN8m7Q1bEf1GcI1gWG+NOzpThma0qsAVbZRIdHvo+/G3NzEeHxs8RArGKhg8vkwehqJ6mwchsOV1T+7UD4+QxZX6ssdisRltjECt3sF7kbejZ7e4ZV1SJseiLSfyTq7pn3WeHid0ttPFfPuxUQ+wIzimXtAYL2aKlmruxBk20hSGkZfxRiSA8a0UZDl2lxhKFDjdaR7/Uul2AOXvYSf2YV8+QyXu/R6czgrK/ONF7zTL/IeRlamIzm7vlJDSO/jEa3pMEJ4KJfbtOlLQQlyED9JrHD0d/T5+TvahQqbhQMMrHx7EBnaByFA+/fCgBWLNmVS3x6KAZt8Ywzo9oucxIBNEn5wiy0mNGUlIJ51AZjoltTFO5ZfVRHNsTKCZ1rd1yOa16HE6kCU+HPT0i8EJVYrvOL5U8SI3tx2vblhWQMdbORmgIdKOaYx5q4/ZJtQMmb7b1HvGtaz62pzC8/1DN8EjGDpFnBtLrzPAx+PCezAR1rIR3bYNyzAFjpP7jt4qIDMlddgEwuosgh5LttDnspGFUjdDlAN2/FERo1nfG5glcAvGqjEjQeUqE1JvLfYsFF3TAOLc12Pdwlf/btsjiwqEGRBn700HGhDqrV9H8R6gN2tpuuWA5iSSyhdrsypOyjL9OUOgsqbXPqIsj/LhkGxIBWw7gFSjXNhLQD12dBnVH0E7Bs4mMCBsYN8usyCNmuB/9xApt8BnpzXhcy/D7HiTfZ72FS0TfBEX0SnXwU4Zo2dnQHsLf5IiaaJqjl8U+kj/1Y3yJpC5Afi7WZ1zy6at0m8PbIz5ubCh+ChU0R0NG31NGkUsTXiDnZTnzbF/nVVZ+kOy+IPjPYTQDjQxsec4VRsjPO9fnm7qd4lIL1Vq2yS12USbqCOisRszQAn5dGtSCMDaFjUV5s1FMhgc8DKyZoV65QRFLttcsTMh7Z777Ne2acvBfWLcf0EMX9HlSoEm1jRSH2iPY0NdKxluSIBl13Z85FiBhKlkv7MkO5FvWzHGpTauDQwdG4wAw9AscBCe6N0RL/kfhH8viOGTkXk0iOeELXZHuqze6NUYtfS7vHpugszNcA3u2jePgJ7LVQDZVVoEy+NRLdJlYRJCovvTMzzk4PXRyPwum8ZNFp9pHUzR8OqSDc1IIESdfpKqdNXcp0+UuI4afrsommxVLLeA0GKlLxGFCM139AyVpORlzuBSatNlglHU6nijeQeFX9l3hQHA2O0kZWaLn1bYW8GM/7RdZDfsRAw/6ookzs8x5myZgwe1M/gi/7hZBlKxvZX9hhr5g4MxqtSK/IU9vtXRTmhgtW6H5Fz2zqs2XNWJdhwO2ABFvi6z+KIy2KNypgib+9Lq67FxbO9XnUrUwlsqAGebawTKkBmnI+TBV17+Raj6OY+qKnYtVdk3ali+enrox5gtj8xjIQrpHIMOhrh076N5T0z1tOM+9BzNpiqP0zx7mCFwenxFTpT7zvCUzpHH22gH6vSdKjm1fSA9jQu3ndcx/oKH21U30sd4nHqG9MDrByIvu/w9k8TP9rQPuqw8vDD1OnhlIdN7zuUg3Oaj7f4f49joEccZO1ADjL6fUd7IBP+GKP9B4qcHyFFnx7+Rij1vsM/EM98NGD/yNKfQ8VcOygQOZZ7Too0rwPOQmEOLjo/hyK0C4DBVT0hQk6FV+ws4m7Q1xOj5SK5QdShegmsk4xx98g7DN9I51P0XtU8WWdnTNikJOiyU5VASbmBNB1s/I7STvzYXJWwZujXDp1QVcA/1tGmHhzQHLR6uBUyo9RwI8/2dbZwPdOP3cAwrNi1dEc37NgKIpMtWMT2sVOeXfzUNLAxZYd+/JOmEayKSJAl0NIzHtID4rOtaZ5ZlkPDIHJ85htRYIZmENiGtVgwSw8gKtJ15umO6cReGLLICGPXCC07NE3TDm2dWUTTJm6vUJ0ICteByp1frVevNbo2HDgazHDsu8/myrCQ1dfoLrFJ07tKrIX7k3td/UtIkjKWgSBFCE6GYhUu2MhCeMw4+AnOTeIKvjBaP01StqI4DFAv/Zhyyy8h6Lof19wbc52gY7iWVSxKggzkp+RABJXsRfuED0hAhLc90co9l9V8IL30t27PgaOh41HWS7fzuYhq5saUqXdPd31QuJTkzn3L5V4ZTSKeWIVLvJXdlg+lbDXUYh9ovwet9rtUkXy4p6+s+dj2fZdy+70+OxrUO3HnjLLTSj+3wifCPRevtHtlREu8taZ/oUcrdppagT/zZfKqWSXTUVpWxJ1ITrrxHqcSEdO31CjJ+OqZXQxWzykunUEf+86blSK4y+ZvpHC077hZnCIVy2XK0KfBND7oVqY0LX/OXUILuVpvxS4o3+W7S56Ojp1/D8fOfZ/OQ7/lSb7e4F1G2eZGDK7g5mBQaagY9Aufj2dK6j9VsImyHzCATKWNe4kFUchTd0Dcyza1d0ZFusn4BY8NKGsIykgVV60HTHTzK+b+TDjAbCm56QUHc10rS6kbheaUFFEFnri2rZ8LKmR4NwH8E44AcnG3JJd9JuhMndcAPyyNJzKNmySqx8t9+HjAipjIxtcOYCJO4i007tGgGZaJ5O3xFJSG+/tCchjxftnRgzJ3UD8dhasvZTV6l10sdjVWIoAXfDj62GgiC3dzvypSAM698wj8hMOvRiogcM9dSCra5PAkEBG849jMdkNDM0W92wR2I1TOwvURaVuTxIgN89UcUXwju91fuu9V0cUbvgcMy5scoDYXbBgsnZFpmBEDJGH94cm7Z31P3CuHXzPlytr+csXGiDEryj71KHrPK56gOcVsVoPFjQzvROIGEQ6gTmTlnsABx6kt+ROP+k7EDHIhqmtmuZ8JYn6WEYM8fPnxQe/n4OEvkrKqLyHuFfdC08vab/LfNwxYQeDO10WCu+U+nBrUNxiS3kdEYV41Tr5v9TPDNT3LsNFZp2dYruMNbu6RB0sSQSKGhCn6jxd38fdvbwfjLq7UkzC2KxW6fhY+yAAbIrz98jL9JbT+7Vf63c36P82VHmbp5j9vg2v2f/62jbLXxeV/KOHXT5/2cUWf+28v0ugtf3X85wt0Lv0CGCUWv+nfrqGUs4PYkzdcyCsABGz/9hvhb/zGjefFJo3RtzLBY7V2DQ38/PAqlKsWhrXv0Yu8eIWOj9/svh9kot0Xr4t6hQI29Ai6Kq5bKYjSlH4jkRl/8+O3P57JKz7ojdysSYIy1AgLBr4R+DuG4qJ6xWSfucsecl3kf6lJFQFKSueEXK9u/5Wz3KOx3t1RLOd5JyVRj3nHHeWVdqRF29ozYln6+mZ/Q8k9AQInGceT346wBk6775YJ0+BAyWQTEi5ObsvmhsWNViV33B140ywIeyL8q5+RLS3/WdOEs3Wpb58g+/IvT6ZJb6SK0Rlffr92iiS2LbOjwfG9cXGfJ5lWFnUDY+dRUkYoEYS6fEh1y39gFXs47GXxDjoiMcZzbGkTqhVrGiU1JEfTtza0ue8UGQWcbo0tFvDCm61VEcqGlppIOyOnvAGt3oOBhj7ostgk4sl5Zk97qCVdnTIApXcRXQPg4R1m+zZlQvdh5NBLCXrEGkfZU2+Bt37O3sONIKCrTZl+l6PEPf4bW8jLgvaS103RvlwG3ZdilwDtxcgXpNBSwGNXpJQQa1bPxfSQ1iPhFEo9adfJxxUY8aYTtesYO/jWRLL9BEnABvroYLbHLeryemIPRS/ewEVtoyjHrU2DnjGqbihsJir6AnProh9Zd5L/3XGBkCqY4HSdcc9VwK3CGJ9MQTYCnD3H7HzqILxsIawneuH1NFPJs8PW9K28khfBQbbhEhEp4U9NuW1FvSyCi1Al6aFcOSEuad21sUGCRVHUk8IaEQHjc4hXwZauvngrDrtSZfNHBT9Ral/7SV3tF7vvmthJ1XNuWlF5mqbm85hQFL++Lw0/cZD1ECmPtX65pPlBTMmHoeOPpPsfRboLoev+ZPvjifYP0PZpun6vdh9JkSMpciRFPiIp8qH53yP7e2R/Pz32d5DuwxLfQregJbzF1r0f0T3ZTFis6tlWc/SmUHgXncpQ7zZVqV50ctKn4c9XptDAgG1PwxlsAQxvo4yjcpOFCg3e4RR5Z6V6F6ZyYyXJ8LJ1eTkmKhaL1yxWihF2LUareyK+uYZQhssWHhCq2r30MG/jK7yEkSTynpU0xnPifiLsFDRZ2GsOLpG/DjVx6SbFu7gGF/rdqxN1gHkAb9DU3X7tzR7tq/jp2amwNS1pjfc0nk7Ff/wOntYlY+odEE2P8X2ik5957yT513RSfH2x3Wyo3Ka78vOL76+k6ptui6/H9FpcL9oQi0lOU4FlLxp+Qb3bUyDzldm+H7LVQOGx7NGA1+4GnQ8XXdebkkndAFSo47ecqleB8mmYkChg/t0Kd7WCqb8v+DU8wA+33aLdntPoCrYbT1aUSIgImwYZu5ex3B9hrqnoHk6Y3w21ktCekMiufCwdpM/AMBPH8dHmld39tntYSyr50qQTNApUEwvF2HxC4kFHdQgd95EKLt4bLyzlxXLY/1KpTqN63wulBCPWK+TFD6+/vXrxw+V3V9/88ubNj68Pvj7qM74VKnCY7tq+GcSmrts0xqtlvXCx8JhBvZh5Ruh7OotMpjtBZPuOFUZetAji2PI85uvuvY4gEKFyFPY1ae7xOuN3djWGI7Mh/lM3KrluH4W+p9jRocyBC8EvOjw75RVglFPchJ0hKzJ5NfaUy4Y3TclDM4cJE4fG5qSny7kTwY+WnNRAekA59NdNtq4Lzhw2RWlxQgGaVFTN7U0GFikPjM49S3vSRubeOcFsu+bkA8zK5c75ngyfUvjaNYNDtAd4OYk7CY/AlpNZR7XCqlqHBV4HHxXrW7Kl6QZKOESK2ke+0aasYHz4QUXrNPpxwIYW2tAk3JRGY9Vv9h5jtmfH3wObflDrlg8+uOSPWVlixjrLnMNWwCGzKoy1RsZZIkraY43Nc4C8I7uOMFvrnCzUrE5V+fEepVSTowM223byxdEbl5T2tpxBh6SFAOleNaCdNmyCEBIiob5GpSqefcdShvidbim/9Jn/aLjhD4co1XID2RJ81vwZwrOVH3MK+My01zdPpBiZv88uRiK9yfqbF40jWsS5moN1mrOLr/KwWj8ZFHOfsi8OC78IbudwkMe0x9jRlKEtSrddxKgbX5KVS8SMaqwskRV9TnM8OSwZAAuQoUQFG0JroT2HloLDZdMC/8gJ2WAViJXQLYbBiQGNMwZoJJpLpn2mrhYu6BvxabyszvpMlMu/xnC5vgXmdS042G4GWlBqhItlcT0xNyeyfHHyoU4OH9OsyIG4chX5J3c3pEgvzYH00vgwKztm/Bj7oHW9212cqc898u/fnKiQPGFVCCNp4AOFrkOR77SsVva+yrQljG+Ta2hbCIQLwxt++W8WzybsGr+p8+/LYrPuRu8QyUdJrw9CgFztnF5rwOzMHiMFkNcVk6bVHXEJ/7+Vjmskg9tc26ygkPPhilavQ249sU1fgJxfCz9ZWcF9ZKWsqlCppkYTj4HWiuBFZFyFSDOHvb76GhinQcr+cXnMqnd1sZ53RMoguSrt+HENE8xRCPIgePvy90n9chOSb0UpU22a5rbwTjGCGhxENmDMHnz84+2YbZMIuQ61hXv4d8aLena4VfYO86os/SOPvDbzeynQJPLlhO9neXuGJ7xJG503aVN6kzZXaPjlRdocnY/qNt5pjnG+h65TTW6p6Tme6hvZlm6o3b77vbnjolNP07agMA+yz30fA4zAgqcnLocP0JEolCdqnXBirTc9nXRibdy9CuZ4g7xhvnTngW9SB83W+ENeDuyKMbfQAetULHp+vc9bmCqu5N+obv6X2X+hjjh/x1PzG9jkGMqCb7L1BY85FYQxP98rs4tzfDawJEXcfKWSRtFsL88ySB3sIaxWMJzwKooGc2WFu/Avb15o/oxkrF4VMTfhrmcX/x8AAP//AKMAXP88aW5wdXQgdHlwZT0iaGlkZGVuIiBkYXRhLWNzcmY9InRydWUiIG5hbWU9ImF1dGhlbnRpY2l0eV90b2tlbiIgdmFsdWU9Ik9kOTJTb0x6Qms1dXk3MmpDTDVqcUdEZTdsaW5KR0Z4UmtEZE12c3dNRzh2bDVlemZ0bFZOUXBwWWZVRmY1eTF1bVRkK0xuWDA0U0JrS0VIS2pHUmN3PT0iIC8+7DvZkty4ke/6CpgPsy8CiwAJHlJ3b8xowh5vaMIK73gi/KQFQbDIaV4mWd1d2nCEf8O/5y/ZTICsIquqNa25O7ytUhFMJIBEIpEHkPWCLP6u0t04tg1RlRyGaycdG9qqsVQAWpRpJput7snYttVYdp3OFkXa3Dtk3Hf62hl2aV2Ozguy+pN9KWklU11dO39td6TeDSNJNRnKbQM9lQ10Rmp5q0nbk65vu3bQRBU45OCQTI6SZuUg00rT+3Isbk66vxrutnaMoswy3Vw7Y7/TDrkr9f0X7cO14xGPsBA+ANP9ULaAwlw2dY1oVLV11za6GefGhS63Bbxho/syGwtbnNg0s2hmz9jLoXAsYVedHAuSl1VF+10FTNF3ummzDIa7dr4OXUGYGwnpcgEfoMxj8KTwv+DuKRg+3/oF9b/FJh/qABp7AOAAlwCBj8GE6bniK266XUApQoVp+0YQN4oDIsy3R0KDhCO+YZ7LgTvQB7M48DTDfR24QRICarjs13QbJNxlonJDU/05ops2OK4n3MCDV1EIlyXKTbCRGwpO3TA2XQcMKYtCbE+h/YLoqXPqsiCp6NT/kinAqCBxOXCLYvdnVRTrTEvs+YNzc7XB9bh5cbUBMVmLztXGCj9W5m1f31hgVt4h3lSwawrFee0zmlf6gYCsaVpt7fOhIgikbZ/pnnL7su3be+o585hXmR5lWQ2HfkDQs/a+IROc9nrQ4+GtBVGt5J5ktGyqstE0rVp16xxncDXs6lr2+wtb15m2gxy6ttt1s0wv9yHssaGEHUUkNGmbwVmy5jewo251KlNatH35oW1GWa02F26kmCSSmc1kpdKjPjm+e55H/A9f48v3o9XMJ973Yj0qSvBqV2ISFgPbVafrTGvd7MjqjQ73zolIVuWpegOgPOurHHX9qDYm36Ew1SBBdNdUehhoV8kRJfxUM5s/s2YzBgxyXzYwxvCyluoiftHr/NopxrEbXm02mR5ux7Zzt6CZd6kLy/74IFtJVVWq22vnzxoEsBzbfv+StJ1uCCp2MvXlnLOAkD8dsL60WOds2sjT/b3m5hlzH2GsM01x827/Tdur4i0KblM22023HxFAqwOkl/ebWg6j7jdZq4bN0O56pTfvh1GCIG/KWoIR21TttnW7Zns2sy9hzKqV2Qndq4nYaSzEbVcddMpmUhfT7rAqa9ZcE9KLSX/h1NC6wobUD+Nh58GGpmmb7UlHPQJKJqX3vQRJ6s2aGcNOsQEh23IYAdXoONRrB8V2VI6ISBXsemjeUf9Eq3Ty6GjYbnAo2FhlvSVDr57GcqTxE3j+n7BE15MCrED/HNcCNi5QtODlBcV/ZOWLFxd0+MdV9/odnKj+1iElaK/vdnU3ttTo9RkrKyWQNuu5WbmbfVO0463eA+lrHf5f0Au6TtiNmc2shuyqrPpdrDbBFV/YF2POjMVSbbWrGyKbsqa5BLsGrlkOjDYjIM0fG39m4+8oJf/h/A+h9MaUrzYoFLLXEkh8qLsbU7NpO7Q7N5PlvcLvmUZQXzgYnTlk6g5iagl2JsN17WBJ6Q7ErpA9LMO185dvfk9jh9R6LFpg9laPS7NZNt1unIfCrmHazdi3kwmXO3RGezAA86SpbXFGVamrbHZ87Y4CJap00VYg2WvufFboqiq71x/hH8GB81bthgWtk3NuhEBV4BZPq4mzXnncHzO8R//AsAE+f2iNPD/m/6wEZ9oRs5p5cdQuL84Ujm1fy7JZ76Nl/aSTlkojb1vUF7gM0BS0ArhTxkOg+a6qQI/A9oIpDeUdzAYWCqaNuDDFsslbM6lld8aygGhAMzBq0Ggl4+iU9Rq9Fm0B1RZhtozq6ABt2uPrdxCylPl+Vm0zygxO9XivwT6BFIakS8EDrLGUhzCpqu2N/qSDBqoz3NST8gMjNtVPgCPGrAaObkSFyncY9xAENbC8qzkZOqEnyrh5AGXik+iuDckpVi0c1qqcBzf7Ab5s/Weq7favCfc4I38ox6926Uvyx0a5a3v7aHOwuicuBCjxhf+w0c3Gvm1gITXtWvAZ9hMIWAlOCm1zYFZ/Vyo9iX6xBxM+exf/+9nfdu34GgOv8T3uE/v+yj5kI6s9GIrBNQgW+NI+OrlHczzhT/0oOeotuCqrXn5vhHbV2GqkFda2xQ1uaF6hGg2wwkQReXWOCA7wtmxAjJvt+12/bjLzb8G6n958rqjZAdPflzN7Gtidf//7+QLQAlzHaycSMecqU0mchXGcs9hnoZJpluuURXkgeJhLxXWY5FnM8jBiSZ7yNFdaizhPFUjKN8gO9IZ+Ecnq+vJOqj1FPmhwz8fnIFoT0U8Rrkuoz1i8QhkJ7qU8jEQQebkIfO7HCkCeJ+Io9sMo1CJKPRXlmRZK5J5SKs51oPwszXns3LyzDPk0AfvtiwRYkV1fjk+SiYu4z1gokiDkoc6YH8pQMaY9KbWKsiTOAxlo5WVxkoYqCXw/krkIVZokQcJTAc5uELE0d04UyGKaM6ucm/+eSj9GM93f30+KCVmwG8wQz0HhWHKfJFvnmM9YsgLUMDIRvox9L4v8PEh1lAmdcSYjrlQWipSJBKycH3kBU2GYZToJYhA3wbnMQWwMPy4KzUGxHE9I7Jq8JJbrED90L4lh65cwy1MxPbFzzg0irUeyxwa2KFdhyFdtrTvgEoQU5YjOtfXpnLV7frgWyGb/EyTbnEmS2viUweN75+hS/pwHizw4HCxi8ZGDRYiQb6kl7UkH9jHx3viuiPEckEyFWJliQLjLExK6wifCDSISuSJxA9eDp6Ausw8fWsBLQl08oo65eeIpN+VQcv0InsKnFhACWhIACh5kIzC279AM2AAADgAm8CidusL2Ba3d0MciYa4XI4kMGkMbN+JYRHgcwQfo9QE3BAqxtekMKgTSAyDmRgDgFFBwyMQWfDcROANA9M3BfYLUcEOFZ/BdP8QOOL4CnBtWuSGMAUPDKEgJ9siQMB/wOVCC3zALczngA0UJ4QQrGDIDejPMxYnPTzcIoBL+gXi4CQ5GcDCg3hUh4jHs0eAzYdbKTAlm7buRwLkIgnOBJcNLAxG4kVlDWApLBTLF8DCxzIRxcR4choTVhVUWsJafx1DlE/uNR8NADogDDdwAGSdiav6dnxYfDvV+1ZjuU0zUsOu6th+XeuUZ2CdziqDGpxioS6jP2EKFYRJyKb3cC3mSQLTFuWaZYErlSvoiS2Mm8iBTOveEF/lexGMWBnES5X4MyhhE4I1lyBTU/xj3ZsEiCDsU8OE5yM47S+pTZOcS6jOWnSwLdZKzJFCxl+g8jdIgDrxU5AmPeaQiJsM8V3mcR0KIJPfTAJ0eHzxsxQBbm2AKGXIqND84Vn8O8iK78imycor2jOWE8YSBWEgPlYvHVchSkSah8oM8zNE3DvIgiEF0QGSSIPV5FikV5DpMOI+14M7N5+/++MNlZDrze25yMvaybJ6oWC7iPmOJycHEZFnOci9Qmc88zwt9IWXghZKlvso0xEncS+JMMpYwFanA0xFoFbBmTPtMOjffTBz5CexRijdsz0BgkM6nCMsZ3jMWlDhKvTBF45N4PBcsD5Qf51ng+xBSMwZSw/IozLQMOMTe0vcSxTzuZTxQIY944tx8AUT8sPBapu1unOJrU/7IOZCtv/kcH5dC7OVN8nna0CUPvktpeLj3XdyRr6+PAHMsZb9f3lsfL9bW13Lmuh+iV/mdfKC676GPWg+DCfKnzs+rgDY5FPbb1jjEhukzbb96VpCsgAlPC9tdzk1KWrBIEYMoFoJI7y2DOBtiUR8iar6shjYQi8YFRJXgGZ/UYJgbvz32+6HGgBkD78iLFESXCcWwG+PWgEdTGYLQOCReBQEuRqgY7sfrdDnGAlsnvoLIk8VyXYsBd+DjEUEk3oZuIDBC9oLow9cJdCchXDWZexRDVfviYQAOxGFWoVgnCpqMQOLdmUTDI9QCcYzLSU7zNbS9cbYvx9MhIy7mchqvyBdilZVDXQ7D+s77SwskVr5+kXOhp4nWw5PEyscjFfxaJlrCgsAav41J6CZB5aOs4NciZ9Ki4NfbBEuxwSKnWPSIFhNErOgFPM/ima+3OCSJDUkkcKP4NAHUeyQH8nj3j2+Ylatk869//HMkne5NwsVYyHFKsiBQGotyIGNZa3d9tb9QcbD8YBQqTQdQKHgBv1Aok1q5l30zFVPZNPqgYgwZ/69n/r31zNL4XRYmanPGadkcbBPK7jGT3GQIyqYdC0xXlymmlNuERpccvELn5s8avTY032j+AdhrkNE9uDZkGs9dZod9AmXWeTgjDcBI3k9M2cHTgCJm91XgeBrDb663L2SW/fgENlIV0HEud9W49E66vjT5asXD+36AoAyTOk9S2Wzmzmw+lkbhjbEeM5k/ZyKbJW/uanqr2+yQYHz2iwiTd7b4VUSNWZKL90O2kUwHGBX436OmAayx7SAWObGaj077PMlrmer9q2vEfzPzeGog14YOmNJCjEeHrkQbRuo9DdH5WUvqXI0DHLfpx9PrMFnSbuMLvzx413Yo7DhSgQUl+4xOmXDncugQc7MAe7wcoMf9K4J3DK9RE2HC4fTqoCoCLaRRis5S6aYRDxHCyTulaQs8qmml8/G8spI9tMGNahXFUEjQdhZ8oM6I3is/9LqH185Cp61yCte6rcFE9xEjuS5tLQ+6PZ122OkUPui+XWJOOYbfu2+n5VoPst6+ADjslzcz3hdWe4CKwHU6aV6D/ezMEMd0/RbEBpZnVgA4lVzrLJUYpsIYpc5+N1VOzUB0ej1lv97/lnxonC3Mcj1vA6OT7uRPUiNmi7KVQ2HLd77yiBuAK+P7oQXBpwjWfgZe9yEy9U+7wP8FBfQPNYL9b8EXKfxvg4L64MlwN44CcN3C6LQd3r7y5MSj8oyqE3eJK5CqJAztz6cOOPhVxK44acYCdAk50Act126YuWYN0HtiYvmLKySBGBKWvw9jPHQ5i5CAk99BYbmg8TmcHsY96QjvaqNHdeFvQLQKrW5PJAtBVrCWWbY7pUABzWkKTxU5WJEoBhOxtiTmWhjMTISsik5+dUcnw8bRBiUuj0+N3mSCYAHR8IF0cBpetI2PhWhr7bFSjgszcfhxwgbz47GyGGvM9fg/AAAA//8BAAD//1esdTBPzgEA){height=\"60px\" width=\"240px\"}" ] } ], "metadata": { "jupytext": { "cell_metadata_filter": "id,colab,colab_type,-all", "formats": "ipynb,py:percent", "main_language": "python" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.6" }, "papermill": { "default_parameters": {}, "duration": 22.150892, "end_time": "2021-07-26T21:27:42.273682", "environment_variables": {}, "exception": null, "input_path": "lightning_examples/reinforce-learning-DQN/dqn.ipynb", "output_path": ".notebooks/lightning_examples/reinforce-learning-DQN.ipynb", "parameters": {}, "start_time": "2021-07-26T21:27:20.122790", "version": "2.3.3" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "1c6b56b096d041c0919c5b92c5d97c34": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_7a65c0674f01458586dd48ac6ab27869", "placeholder": "​", "style": "IPY_MODEL_3c7b01fc55e3423b895c4d2536db3ed1", "value": " 13/? [00:00<00:00, 196.93it/s, loss=95.8, v_num=2]" } }, "292eb170164e439ab3c69c33cde369f8": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HBoxModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HBoxView", "box_style": "", "children": [ "IPY_MODEL_ad6c1a4b38634d258a51585b70198b79", "IPY_MODEL_a7aaf59959e642c0a4ae1f7135a7b0fd", "IPY_MODEL_1c6b56b096d041c0919c5b92c5d97c34" ], "layout": "IPY_MODEL_ef6e47263c85484b8b0c4ec4a8bb578a" } }, "2f7877ab98ce43e4b597d36b0a04a87e": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": "2", "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "3c7b01fc55e3423b895c4d2536db3ed1": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "7a65c0674f01458586dd48ac6ab27869": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "a7aaf59959e642c0a4ae1f7135a7b0fd": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "FloatProgressModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_2f7877ab98ce43e4b597d36b0a04a87e", "max": 1.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_ba7905dc03934fb9ba1e0f987957ffff", "value": 1.0 } }, "ad6c1a4b38634d258a51585b70198b79": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "HTMLModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "1.5.0", "_view_name": "HTMLView", "description": "", "description_tooltip": null, "layout": "IPY_MODEL_df04e4b2b00644d0a5f086876becc8ae", "placeholder": "​", "style": "IPY_MODEL_ada489fa15ca472a9286a5baa6783400", "value": "Epoch 199: " } }, "ada489fa15ca472a9286a5baa6783400": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "DescriptionStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "DescriptionStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "description_width": "" } }, "ba7905dc03934fb9ba1e0f987957ffff": { "model_module": "@jupyter-widgets/controls", "model_module_version": "1.5.0", "model_name": "ProgressStyleModel", "state": { "_model_module": "@jupyter-widgets/controls", "_model_module_version": "1.5.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "StyleView", "bar_color": null, "description_width": "" } }, "df04e4b2b00644d0a5f086876becc8ae": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "ef6e47263c85484b8b0c4ec4a8bb578a": { "model_module": "@jupyter-widgets/base", "model_module_version": "1.2.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "1.2.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "1.2.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border": null, "bottom": null, "display": "inline-flex", "flex": null, "flex_flow": "row wrap", "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "overflow_x": null, "overflow_y": null, "padding": null, "right": null, "top": null, "visibility": null, "width": "100%" } } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }