{"cells": [{"cell_type": "markdown", "id": "9c902f0b", "metadata": {"papermill": {"duration": 0.020088, "end_time": "2023-10-04T01:00:26.256659", "exception": false, "start_time": "2023-10-04T01:00:26.236571", "status": "completed"}, "tags": []}, "source": ["\n", "# Fine-Tuning Scheduler\n", "\n", "* **Author:** [Dan Dale](https://github.com/speediedan)\n", "* **License:** CC BY-SA\n", "* **Generated:** 2023-10-04T00:59:41.547882\n", "\n", "This notebook introduces the [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) extension\n", "and demonstrates the use of it to fine-tune a small foundation model on the\n", "[RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task of\n", "[SuperGLUE](https://super.gluebenchmark.com/) with iterative early-stopping defined according to a user-specified\n", "schedule. It uses Hugging Face's ``datasets`` and ``transformers`` libraries to retrieve the relevant benchmark data\n", "and foundation model weights. The required dependencies are installed via the finetuning-scheduler ``[examples]`` extra.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/finetuning-scheduler.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/Lightning-AI/lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)"]}, {"cell_type": "markdown", "id": "c5a929ac", "metadata": {"papermill": {"duration": 0.017506, "end_time": "2023-10-04T01:00:26.292196", "exception": false, "start_time": "2023-10-04T01:00:26.274690", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "2b818c6f", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2023-10-04T01:00:26.321284Z", "iopub.status.busy": "2023-10-04T01:00:26.321092Z", "iopub.status.idle": "2023-10-04T01:00:31.435751Z", "shell.execute_reply": "2023-10-04T01:00:31.434738Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 5.131552, "end_time": "2023-10-04T01:00:31.438692", "exception": false, "start_time": "2023-10-04T01:00:26.307140", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\r\n", "\u001b[0m"]}], "source": ["! pip install --quiet \"finetuning-scheduler[examples]>=2.0.0\" \"setuptools>=68.0.0, <68.3.0\" \"urllib3\" \"torchmetrics>=0.7, <1.3\" \"torch>=1.12.1\" \"matplotlib>=3.0.0, <3.9.0\" \"pytorch-lightning>=1.4, <2.1.0\" \"ipython[notebook]>=8.0.0, <8.17.0\" \"torch>=1.8.1, <2.1.0\""]}, {"cell_type": "markdown", "id": "7d794101", "metadata": {"papermill": {"duration": 0.026176, "end_time": "2023-10-04T01:00:31.494949", "exception": false, "start_time": "2023-10-04T01:00:31.468773", "status": "completed"}, "tags": []}, "source": ["## Scheduled Fine-Tuning with the Fine-Tuning Scheduler Extension\n", "\n", "{height=\"55px\" width=\"401px\"}\n", "\n", "The [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) extension accelerates and enhances model experimentation with flexible fine-tuning schedules.\n", "\n", "Training with the extension is simple and confers a host of benefits:\n", "\n", "- it dramatically increases fine-tuning flexibility\n", "- expedites and facilitates exploration of model tuning dynamics\n", "- enables marginal performance improvements of fine-tuned models\n", "\n", "Setup is straightforward, just install from PyPI! Since this notebook-based example requires a few additional packages (e.g.\n", "``transformers``, ``sentencepiece``), we installed the ``finetuning-scheduler`` package with the ``[examples]`` extra above.\n", "Once the ``finetuning-scheduler`` package is installed, the [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) callback (FTS) is available for use with Lightning.\n", "For additional installation options, please see the Fine-Tuning Scheduler [README](https://github.com/speediedan/finetuning-scheduler/blob/main/README.md).\n", "\n", "\n", "\n", "\n", "\n", "Fundamentally, [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) enables\n", "scheduled, multi-phase, fine-tuning of foundation models. Gradual unfreezing (i.e. thawing) can help maximize\n", "foundation model knowledge retention while allowing (typically upper layers of) the model to\n", "optimally adapt to new tasks during transfer learning [1, 2, 3]\n", "\n", "
\n", "\n", "The [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) callback orchestrates the gradual unfreezing\n", "of models via a fine-tuning schedule that is either implicitly generated (the default) or explicitly provided by the user\n", "(more computationally efficient). Fine-tuning phase transitions are driven by\n", "[FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping)\n", "criteria (a multi-phase extension of ``EarlyStopping`` packaged with FinetuningScheduler), user-specified epoch transitions or a composition of the two (the default mode).\n", "A [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) training session completes when the\n", "final phase of the schedule has its stopping criteria met. See\n", "the [early stopping documentation](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.EarlyStopping.html) for more details on that callback's configuration.\n", "\n", "{height=\"272px\" width=\"376px\"}"]}, {"cell_type": "markdown", "id": "07db61a0", "metadata": {"papermill": {"duration": 0.014737, "end_time": "2023-10-04T01:00:31.527404", "exception": false, "start_time": "2023-10-04T01:00:31.512667", "status": "completed"}, "tags": []}, "source": ["\n", "## Basic Usage\n", "\n", "If no fine-tuning schedule is provided by the user, [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) will generate a\n", "[default schedule](#The-Default-Fine-Tuning-Schedule) and proceed to fine-tune according to the generated schedule,\n", "using default [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) and [FTSCheckpoint](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSCheckpoint) callbacks with ``monitor=val_loss``.\n", "\n", "\n", "\n", "```python\n", "import lightning as L\n", "from finetuning_scheduler import FinetuningScheduler\n", "trainer = L.Trainer(callbacks=[FinetuningScheduler()])\n", "```"]}, {"cell_type": "markdown", "id": "7deee3b5", "metadata": {"papermill": {"duration": 0.013461, "end_time": "2023-10-04T01:00:31.593130", "exception": false, "start_time": "2023-10-04T01:00:31.579669", "status": "completed"}, "tags": []}, "source": ["## The Default Fine-Tuning Schedule\n", "\n", "Schedule definition is facilitated via the [gen_ft_schedule](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.ScheduleImplMixin.gen_ft_schedule) method which dumps a default fine-tuning schedule (by default using a naive, 2-parameters per level heuristic) which can be adjusted as\n", "desired by the user and/or subsequently passed to the callback. Using the default/implicitly generated schedule will likely be less computationally efficient than a user-defined fine-tuning schedule but is useful for exploring a model's fine-tuning behavior and can serve as a good baseline for subsequent explicit schedule refinement.\n", "While the current version of [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) only supports single optimizer and (optional) lr_scheduler configurations, per-phase maximum learning rates can be set as demonstrated in the next section."]}, {"cell_type": "markdown", "id": "bc71b2b5", "metadata": {"papermill": {"duration": 0.012689, "end_time": "2023-10-04T01:00:31.619528", "exception": false, "start_time": "2023-10-04T01:00:31.606839", "status": "completed"}, "tags": []}, "source": ["## Specifying a Fine-Tuning Schedule\n", "\n", "To specify a fine-tuning schedule, it's convenient to first generate the default schedule and then alter the thawed/unfrozen parameter groups associated with each fine-tuning phase as desired. Fine-tuning phases are zero-indexed and executed in ascending order.\n", "\n", "1. First, generate the default schedule to ``Trainer.log_dir``. It will be named after your\n", " ``LightningModule`` subclass with the suffix ``_ft_schedule.yaml``.\n", "\n", "```python\n", " import lightning as L\n", " from finetuning_scheduler import FinetuningScheduler\n", " trainer = L.Trainer(callbacks=[FinetuningScheduler(gen_ft_sched_only=True)])\n", "```\n", "\n", "2. Alter the schedule as desired.\n", "\n", "{height=\"327px\" width=\"800px\"}\n", "\n", "3. Once the fine-tuning schedule has been altered as desired, pass it to\n", " [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) to commence scheduled training:\n", "\n", "```python\n", "import lightning as L\n", "from finetuning_scheduler import FinetuningScheduler\n", "\n", "trainer = L.Trainer(callbacks=[FinetuningScheduler(ft_schedule=\"/path/to/my/schedule/my_schedule.yaml\")])\n", "```"]}, {"cell_type": "markdown", "id": "62af6cce", "metadata": {"papermill": {"duration": 0.012779, "end_time": "2023-10-04T01:00:31.645053", "exception": false, "start_time": "2023-10-04T01:00:31.632274", "status": "completed"}, "tags": []}, "source": ["## Early-Stopping and Epoch-Driven Phase Transition Criteria\n", "\n", "\n", "By default, [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) and epoch-driven\n", "transition criteria are composed. If a ``max_transition_epoch`` is specified for a given phase, the next fine-tuning phase will begin at that epoch unless [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) criteria are met first.\n", "If [FinetuningScheduler.epoch_transitions_only](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler.params.epoch_transitions_only) is ``True``, [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) will not be used\n", "and transitions will be exclusively epoch-driven.\n", "\n", "\n", "\n", "\n", "**Tip:** Use of regex expressions can be convenient for specifying more complex schedules. Also, a per-phase base maximum lr can be specified:\n", "\n", "{height=\"380px\" width=\"800px\"}\n", "\n", "
\n", "\n", "\n", "\n", "The end-to-end example in this notebook ([Scheduled Fine-Tuning For SuperGLUE](#Scheduled-Fine-Tuning-For-SuperGLUE)) uses [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) in explicit mode to fine-tune a small foundation model on the [RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task of [SuperGLUE](https://super.gluebenchmark.com/).\n", "Please see the [official Fine-Tuning Scheduler documentation](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) if you are interested in a similar [CLI-based example](https://finetuning-scheduler.readthedocs.io/en/stable/index.html#example-scheduled-fine-tuning-for-superglue) using the LightningCLI."]}, {"cell_type": "markdown", "id": "06c0d293", "metadata": {"papermill": {"duration": 0.012695, "end_time": "2023-10-04T01:00:31.670485", "exception": false, "start_time": "2023-10-04T01:00:31.657790", "status": "completed"}, "tags": []}, "source": ["## Resuming Scheduled Fine-Tuning Training Sessions\n", "\n", "Resumption of scheduled fine-tuning training is identical to the continuation of\n", "[other training sessions](https://lightning.ai/docs/pytorch/stable/common/trainer.html) with the caveat that the provided checkpoint must have been saved by a [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) session.\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) uses [FTSCheckpoint](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSCheckpoint) (an extension of ``ModelCheckpoint``) to maintain schedule state with special metadata.\n", "\n", "\n", "```python\n", "import lightning as L\n", "from finetuning_scheduler import FinetuningScheduler\n", "trainer = L.Trainer(callbacks=[FinetuningScheduler()])\n", "trainer.ckpt_path=\"some/path/to/my_checkpoint.ckpt\"\n", "trainer.fit(...)\n", "```\n", "\n", "Training will resume at the depth/level of the provided checkpoint according to the specified schedule. Schedules can be altered between training sessions but schedule compatibility is left to the user for maximal flexibility. If executing a user-defined schedule, typically the same schedule should be provided for the original and resumed training sessions.\n", "\n", "By default ([FinetuningScheduler.restore_best](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=restore_best#finetuning_scheduler.fts.FinetuningScheduler.params.restore_best) is ``True``), [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) will attempt to restore the best available checkpoint before fine-tuning depth transitions.\n", "\n", "```python\n", "trainer = L.Trainer(callbacks=[FinetuningScheduler()])\n", "trainer.ckpt_path=\"some/path/to/my_kth_best_checkpoint.ckpt\"\n", "trainer.fit(...)\n", "```\n", "\n", "Note that similar to the behavior of [ModelCheckpoint](https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.ModelCheckpoint.html), when resuming training with a\n", "different [FTSCheckpoint](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSCheckpoint) ``dirpath`` from the provided\n", "checkpoint, the new training session's checkpoint state will be re-initialized at the resumption depth with the provided checkpoint being set as the best checkpoint."]}, {"cell_type": "markdown", "id": "9fd4b1a8", "metadata": {"papermill": {"duration": 0.012728, "end_time": "2023-10-04T01:00:31.696107", "exception": false, "start_time": "2023-10-04T01:00:31.683379", "status": "completed"}, "tags": []}, "source": ["\n", "\n", "**Note:** Currently, [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) supports the following distributed strategy types:\n", "\n", "- ``ddp`` (and aliases ``ddp_find_unused_parameters_false``, ``ddp_find_unused_parameters_true``, ``ddp_spawn``, ``ddp_fork``, ``ddp_notebook``)\n", "- ``fsdp`` (and alias ``fsdp_cpu_offload``)\n", "\n", "Custom or officially unsupported strategies can be used by setting [FinetuningScheduler.allow_untested](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=allow_untested#finetuning_scheduler.fts.FinetuningScheduler.params.allow_untested) to ``True``.\n", "Note that most currently unsupported strategies are so because they require varying degrees of modification to be compatible. For example, ``deepspeed`` will require a [StrategyAdapter](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.strategy_adapters.html#finetuning_scheduler.strategy_adapters.StrategyAdapter) to be written (similar to the one for ``FSDP``, [FSDPStrategyAdapter](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.strategy_adapters.html#finetuning_scheduler.strategy_adapters.FSDPStrategyAdapter)) before support can be added (PRs welcome!),\n", "while ``tpu_spawn`` would require an override of the current broadcast method to include python objects.\n", "
"]}, {"cell_type": "markdown", "id": "e1282094", "metadata": {"papermill": {"duration": 0.012755, "end_time": "2023-10-04T01:00:31.721575", "exception": false, "start_time": "2023-10-04T01:00:31.708820", "status": "completed"}, "tags": []}, "source": ["## Scheduled Fine-Tuning For SuperGLUE\n", "\n", "The following example demonstrates the use of [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) to fine-tune a small foundation model on the [RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task of [SuperGLUE](https://super.gluebenchmark.com/). Iterative early-stopping will be applied according to a user-specified schedule.\n"]}, {"cell_type": "code", "execution_count": 2, "id": "73da1a1d", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:31.749268Z", "iopub.status.busy": "2023-10-04T01:00:31.748867Z", "iopub.status.idle": "2023-10-04T01:00:34.666615Z", "shell.execute_reply": "2023-10-04T01:00:34.665312Z"}, "papermill": {"duration": 2.934229, "end_time": "2023-10-04T01:00:34.668840", "exception": false, "start_time": "2023-10-04T01:00:31.734611", "status": "completed"}, "tags": []}, "outputs": [], "source": ["import os\n", "import warnings\n", "from datetime import datetime\n", "from typing import Any, Dict, Optional\n", "\n", "import datasets\n", "import evaluate"]}, {"cell_type": "code", "execution_count": 3, "id": "55549b1f", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:34.706343Z", "iopub.status.busy": "2023-10-04T01:00:34.705931Z", "iopub.status.idle": "2023-10-04T01:00:36.819413Z", "shell.execute_reply": "2023-10-04T01:00:36.818110Z"}, "papermill": {"duration": 2.134867, "end_time": "2023-10-04T01:00:36.821957", "exception": false, "start_time": "2023-10-04T01:00:34.687090", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Import the `FinetuningScheduler` PyTorch Lightning extension module we want to use. This will import all necessary callbacks.\n", "import finetuning_scheduler as fts # isort: split\n", "\n", "import lightning as L\n", "import sentencepiece as sp # noqa: F401 # isort: split\n", "import torch\n", "from datasets import logging as datasets_logging\n", "from lightning.pytorch.callbacks import EarlyStopping, ModelCheckpoint\n", "from lightning.pytorch.loggers.tensorboard import TensorBoardLogger\n", "from lightning.pytorch.utilities import rank_zero_warn\n", "from torch.optim.adamw import AdamW\n", "from torch.optim.lr_scheduler import CosineAnnealingWarmRestarts\n", "from torch.utils.data import DataLoader\n", "from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer\n", "from transformers import logging as transformers_logging\n", "from transformers.tokenization_utils_base import BatchEncoding\n", "\n", "# set notebook-level variables\n", "TASK_NUM_LABELS = {\"boolq\": 2, \"rte\": 2}\n", "DEFAULT_TASK = \"rte\"\n", "\n", "# reduce hf logging verbosity to focus on tutorial-relevant code/messages\n", "for hflogger in [transformers_logging, datasets_logging]:\n", " hflogger.set_verbosity_error()\n", "# ignore warnings related tokenizers_parallelism/DataLoader parallelism trade-off and\n", "# expected logging behavior\n", "for warnf in [\n", " r\".*does not have many workers.*\",\n", " r\".*The number of training samples.*\",\n", " r\".*converting to a fast.*\",\n", " r\".*number of training batches.*\",\n", "]:\n", " warnings.filterwarnings(\"ignore\", warnf)"]}, {"cell_type": "code", "execution_count": 4, "id": "f6d43eda", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:36.878554Z", "iopub.status.busy": "2023-10-04T01:00:36.877476Z", "iopub.status.idle": "2023-10-04T01:00:36.890455Z", "shell.execute_reply": "2023-10-04T01:00:36.889486Z"}, "papermill": {"duration": 0.041598, "end_time": "2023-10-04T01:00:36.892086", "exception": false, "start_time": "2023-10-04T01:00:36.850488", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class RteBoolqDataModule(L.LightningDataModule):\n", " \"\"\"A ``LightningDataModule`` designed for both the RTE or BoolQ SuperGLUE Hugging Face datasets.\"\"\"\n", "\n", " TASK_TEXT_FIELD_MAP = {\"rte\": (\"premise\", \"hypothesis\"), \"boolq\": (\"question\", \"passage\")}\n", " LOADER_COLUMNS = (\n", " \"datasets_idx\",\n", " \"input_ids\",\n", " \"token_type_ids\",\n", " \"attention_mask\",\n", " \"start_positions\",\n", " \"end_positions\",\n", " \"labels\",\n", " )\n", "\n", " def __init__(\n", " self,\n", " model_name_or_path: str,\n", " task_name: str = DEFAULT_TASK,\n", " max_seq_length: int = 128,\n", " train_batch_size: int = 16,\n", " eval_batch_size: int = 16,\n", " tokenizers_parallelism: bool = True,\n", " **dataloader_kwargs: Any,\n", " ):\n", " r\"\"\"Initialize the ``LightningDataModule`` designed for both the RTE or BoolQ SuperGLUE Hugging Face datasets.\n", "\n", " Args:\n", " model_name_or_path (str):\n", " Can be either:\n", " - A string, the ``model id`` of a pretrained model hosted inside a model repo on huggingface.co.\n", " Valid model ids can be located at the root-level, like ``bert-base-uncased``, or namespaced\n", " under a user or organization name, like ``dbmdz/bert-base-german-cased``.\n", " - A path to a ``directory`` containing model weights saved using\n", " :meth:`~transformers.PreTrainedModel.save_pretrained`, e.g., ``./my_model_directory/``.\n", " task_name (str, optional): Name of the SuperGLUE task to execute. This module supports 'rte' or 'boolq'.\n", " Defaults to DEFAULT_TASK which is 'rte'.\n", " max_seq_length (int, optional): Length to which we will pad sequences or truncate input. Defaults to 128.\n", " train_batch_size (int, optional): Training batch size. Defaults to 16.\n", " eval_batch_size (int, optional): Batch size to use for validation and testing splits. Defaults to 16.\n", " tokenizers_parallelism (bool, optional): Whether to use parallelism in the tokenizer. Defaults to True.\n", " \\**dataloader_kwargs: Arguments passed when initializing the dataloader.\n", " \"\"\"\n", " super().__init__()\n", " task_name = task_name if task_name in TASK_NUM_LABELS.keys() else DEFAULT_TASK\n", " self.text_fields = self.TASK_TEXT_FIELD_MAP[task_name]\n", " self.dataloader_kwargs = {\n", " \"num_workers\": dataloader_kwargs.get(\"num_workers\", 0),\n", " \"pin_memory\": dataloader_kwargs.get(\"pin_memory\", False),\n", " }\n", " self.save_hyperparameters()\n", " os.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\" if self.hparams.tokenizers_parallelism else \"false\"\n", " self.tokenizer = AutoTokenizer.from_pretrained(\n", " self.hparams.model_name_or_path, use_fast=True, local_files_only=False\n", " )\n", "\n", " def prepare_data(self):\n", " \"\"\"Load the SuperGLUE dataset.\"\"\"\n", " # N.B. PL calls prepare_data from a single process (rank 0) so do not use it to assign\n", " # state (e.g. self.x=y)\n", " datasets.load_dataset(\"super_glue\", self.hparams.task_name)\n", "\n", " def setup(self, stage):\n", " \"\"\"Setup our dataset splits for training/validation.\"\"\"\n", " self.dataset = datasets.load_dataset(\"super_glue\", self.hparams.task_name)\n", " for split in self.dataset.keys():\n", " self.dataset[split] = self.dataset[split].map(\n", " self._convert_to_features, batched=True, remove_columns=[\"label\"]\n", " )\n", " self.columns = [c for c in self.dataset[split].column_names if c in self.LOADER_COLUMNS]\n", " self.dataset[split].set_format(type=\"torch\", columns=self.columns)\n", "\n", " self.eval_splits = [x for x in self.dataset.keys() if \"validation\" in x]\n", "\n", " def train_dataloader(self):\n", " return DataLoader(self.dataset[\"train\"], batch_size=self.hparams.train_batch_size, **self.dataloader_kwargs)\n", "\n", " def val_dataloader(self):\n", " return DataLoader(self.dataset[\"validation\"], batch_size=self.hparams.eval_batch_size, **self.dataloader_kwargs)\n", "\n", " def _convert_to_features(self, example_batch: datasets.arrow_dataset.LazyDict) -> BatchEncoding:\n", " \"\"\"Convert raw text examples to a :class:`~transformers.tokenization_utils_base.BatchEncoding` container\n", " (derived from python dict) of features that includes helpful methods for translating between word/character\n", " space and token space.\n", "\n", " Args:\n", " example_batch ([type]): The set of examples to convert to token space.\n", "\n", " Returns:\n", " ``BatchEncoding``: A batch of encoded examples (note default tokenizer batch_size=1000).\n", " \"\"\"\n", " text_pairs = list(zip(example_batch[self.text_fields[0]], example_batch[self.text_fields[1]]))\n", " # Tokenize the text/text pairs\n", " features = self.tokenizer.batch_encode_plus(\n", " text_pairs, max_length=self.hparams.max_seq_length, padding=\"longest\", truncation=True\n", " )\n", " # Rename label to labels to make it easier to pass to model forward\n", " features[\"labels\"] = example_batch[\"label\"]\n", " return features"]}, {"cell_type": "code", "execution_count": 5, "id": "36ebe774", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:36.929207Z", "iopub.status.busy": "2023-10-04T01:00:36.928956Z", "iopub.status.idle": "2023-10-04T01:00:36.945376Z", "shell.execute_reply": "2023-10-04T01:00:36.944252Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.036949, "end_time": "2023-10-04T01:00:36.947067", "exception": false, "start_time": "2023-10-04T01:00:36.910118", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class RteBoolqModule(L.LightningModule):\n", " \"\"\"A ``LightningModule`` that can be used to fine-tune a foundation model on either the RTE or BoolQ SuperGLUE\n", " tasks using Hugging Face implementations of a given model and the `SuperGLUE Hugging Face dataset.\"\"\"\n", "\n", " def __init__(\n", " self,\n", " model_name_or_path: str,\n", " optimizer_init: Dict[str, Any],\n", " lr_scheduler_init: Dict[str, Any],\n", " model_cfg: Optional[Dict[str, Any]] = None,\n", " task_name: str = DEFAULT_TASK,\n", " experiment_tag: str = \"default\",\n", " ):\n", " \"\"\"\n", " Args:\n", " model_name_or_path (str): Path to pretrained model or identifier from https://huggingface.co/models.\n", " optimizer_init (Dict[str, Any]): The desired optimizer configuration.\n", " lr_scheduler_init (Dict[str, Any]): The desired learning rate scheduler config.\n", " model_cfg (Optional[Dict[str, Any]], optional): Defines overrides of the default model config. Defaults to\n", " ``None``.\n", " task_name (str, optional): The SuperGLUE task to execute, one of ``'rte'``, ``'boolq'``. Defaults to \"rte\".\n", " experiment_tag (str, optional): The tag to use for the experiment and tensorboard logs. Defaults to\n", " \"default\".\n", " \"\"\"\n", " super().__init__()\n", " if task_name not in TASK_NUM_LABELS.keys():\n", " rank_zero_warn(f\"Invalid task_name {task_name!r}. Proceeding with the default task: {DEFAULT_TASK!r}\")\n", " task_name = DEFAULT_TASK\n", " self.num_labels = TASK_NUM_LABELS[task_name]\n", " self.model_cfg = model_cfg or {}\n", " conf = AutoConfig.from_pretrained(model_name_or_path, num_labels=self.num_labels, local_files_only=False)\n", " self.model = AutoModelForSequenceClassification.from_pretrained(model_name_or_path, config=conf)\n", " self.model.config.update(self.model_cfg) # apply model config overrides\n", " self.init_hparams = {\n", " \"optimizer_init\": optimizer_init,\n", " \"lr_scheduler_init\": lr_scheduler_init,\n", " \"model_config\": self.model.config,\n", " \"model_name_or_path\": model_name_or_path,\n", " \"task_name\": task_name,\n", " \"experiment_id\": f\"{datetime.now().strftime('%Y%m%d_%H%M%S')}_{experiment_tag}\",\n", " }\n", " self.save_hyperparameters(self.init_hparams)\n", " self.metric = evaluate.load(\"super_glue\", self.hparams.task_name, experiment_id=self.hparams.experiment_id)\n", " self.no_decay = [\"bias\", \"LayerNorm.weight\"]\n", "\n", " @property\n", " def finetuningscheduler_callback(self) -> fts.FinetuningScheduler:\n", " fts_callback = [c for c in self.trainer.callbacks if isinstance(c, fts.FinetuningScheduler)]\n", " return fts_callback[0] if fts_callback else None\n", "\n", " def forward(self, **inputs):\n", " return self.model(**inputs)\n", "\n", " def training_step(self, batch, batch_idx: int):\n", " loss = self(**batch)[0]\n", " self.log(\"train_loss\", loss, prog_bar=True)\n", " return loss\n", "\n", " def on_train_epoch_end(self):\n", " if self.finetuningscheduler_callback:\n", " self.log(\"finetuning_schedule_depth\", float(self.finetuningscheduler_callback.curr_depth))\n", "\n", " def validation_step(self, batch, batch_idx, dataloader_idx=0):\n", " outputs = self(**batch)\n", " val_loss, logits = outputs[:2]\n", " if self.num_labels >= 1:\n", " preds = torch.argmax(logits, axis=1)\n", " elif self.num_labels == 1:\n", " preds = logits.squeeze()\n", " labels = batch[\"labels\"]\n", " self.log(\"val_loss\", val_loss, prog_bar=True)\n", " metric_dict = self.metric.compute(predictions=preds, references=labels)\n", " self.log_dict(metric_dict, prog_bar=True)\n", "\n", " def configure_optimizers(self):\n", " # With FTS >= 2.0, ``FinetuningScheduler`` simplifies initial optimizer configuration by ensuring the optimizer\n", " # configured here will optimize the parameters (and only those parameters) scheduled to be optimized in phase 0\n", " # of the current fine-tuning schedule. This auto-configuration can be disabled if desired by setting\n", " # ``enforce_phase0_params`` to ``False``.\n", " optimizer = AdamW(params=self.model.parameters(), **self.hparams.optimizer_init)\n", " scheduler = {\n", " \"scheduler\": CosineAnnealingWarmRestarts(optimizer, **self.hparams.lr_scheduler_init),\n", " \"interval\": \"epoch\",\n", " }\n", " return [optimizer], [scheduler]"]}, {"cell_type": "markdown", "id": "5bd6e1dc", "metadata": {"papermill": {"duration": 0.017551, "end_time": "2023-10-04T01:00:36.982270", "exception": false, "start_time": "2023-10-04T01:00:36.964719", "status": "completed"}, "tags": []}, "source": ["### Our Training Sessions\n", "\n", "We'll be comparing three different fine-tuning training configurations. Every configuration in this example depends\n", "upon a shared set of defaults, only differing in their respective fine-tuning schedules.\n", "\n", "| Experiment Tag | Training Scenario Description |\n", "|:-----------------:| ---------------------------------------------------------------------- |\n", "| ``fts_explicit`` | Training with a fine-tuning schedule explicitly provided by the user |\n", "| ``nofts_baseline``| A baseline fine-tuning training session (without scheduled fine-tuning) |\n", "| ``fts_implicit`` | Training with an implicitly generated fine-tuning schedule (the default) |\n", "\n", "Let's begin by configuring the ``fts_explicit`` scenario. We'll subsequently run the other two scenarios for\n", "comparison."]}, {"cell_type": "code", "execution_count": 6, "id": "3188c155", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:37.010180Z", "iopub.status.busy": "2023-10-04T01:00:37.009800Z", "iopub.status.idle": "2023-10-04T01:00:37.017080Z", "shell.execute_reply": "2023-10-04T01:00:37.015947Z"}, "papermill": {"duration": 0.022944, "end_time": "2023-10-04T01:00:37.018630", "exception": false, "start_time": "2023-10-04T01:00:36.995686", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Let's create a fine-tuning schedule for our model and run an explicitly scheduled fine-tuning training scenario with it\n", "# Please see the [FinetuningScheduler documentation](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) for a full description of the schedule format\n", "\n", "\n", "ft_schedule_yaml = \"\"\"\n", "0:\n", " params:\n", " - model.classifier.bias\n", " - model.classifier.weight\n", " - model.pooler.dense.bias\n", " - model.pooler.dense.weight\n", " - model.deberta.encoder.LayerNorm.bias\n", " - model.deberta.encoder.LayerNorm.weight\n", " - model.deberta.encoder.rel_embeddings.weight\n", " - model.deberta.encoder.layer.{0,11}.(output|attention|intermediate).*\n", "1:\n", " params:\n", " - model.deberta.embeddings.LayerNorm.bias\n", " - model.deberta.embeddings.LayerNorm.weight\n", "2:\n", " params:\n", " - model.deberta.embeddings.word_embeddings.weight\n", "\"\"\"\n", "ft_schedule_name = \"RteBoolqModule_ft_schedule_deberta_base.yaml\"\n", "# Let's write the schedule to a file so we can simulate loading an explicitly defined fine-tuning\n", "# schedule.\n", "with open(ft_schedule_name, \"w\") as f:\n", " f.write(ft_schedule_yaml)"]}, {"cell_type": "code", "execution_count": 7, "id": "d32e3c9d", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:37.046576Z", "iopub.status.busy": "2023-10-04T01:00:37.045729Z", "iopub.status.idle": "2023-10-04T01:00:38.872986Z", "shell.execute_reply": "2023-10-04T01:00:38.871591Z"}, "papermill": {"duration": 1.843878, "end_time": "2023-10-04T01:00:38.875595", "exception": false, "start_time": "2023-10-04T01:00:37.031717", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "7c6bad07fce443efb64239924c3c67ca", "version_major": 2, "version_minor": 0}, "text/plain": ["Downloading (\u2026)okenizer_config.json: 0%| | 0.00/52.0 [00:00"]}, {"cell_type": "code", "execution_count": 8, "id": "4e3732b0", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:38.967519Z", "iopub.status.busy": "2023-10-04T01:00:38.966692Z", "iopub.status.idle": "2023-10-04T01:00:38.972096Z", "shell.execute_reply": "2023-10-04T01:00:38.971224Z"}, "papermill": {"duration": 0.023705, "end_time": "2023-10-04T01:00:38.973452", "exception": false, "start_time": "2023-10-04T01:00:38.949747", "status": "completed"}, "tags": []}, "outputs": [], "source": ["optimizer_init = {\"weight_decay\": 1e-05, \"eps\": 1e-07, \"lr\": 1e-05}"]}, {"cell_type": "markdown", "id": "225dfe6d", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.013358, "end_time": "2023-10-04T01:00:39.000165", "exception": false, "start_time": "2023-10-04T01:00:38.986807", "status": "completed"}, "tags": []}, "source": ["### LR Scheduler Configuration\n", "\n", "The [CosineAnnealingWarmRestarts scheduler](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.html?highlight=cosineannealingwarm#torch.optim.lr_scheduler.CosineAnnealingWarmRestarts) nicely fits with our iterative fine-tuning since it does not depend upon a global max_epoch\n", "value. The importance of initial warmup is reduced due to the innate warmup effect of Adam bias correction [5]\n", "and the gradual thawing we are performing. Note that commonly used LR schedulers that depend on providing\n", "max_iterations/epochs (e.g. the\n", "[CosineWarmupScheduler](https://github.com/Lightning-AI/tutorials/blob/0c325829101d5a6ebf32ed99bbf5b09badf04a59/course_UvA-DL/05-transformers-and-MH-attention/Transformers_MHAttention.py#L688)\n", "used in other pytorch-lightning tutorials) also work with FinetuningScheduler. Though the LR scheduler is theoretically\n", "justified (Loshchilov & Hutter, 2016), the particular values provided here are primarily empircally driven.\n", "\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) also supports both optimizer and LR scheduler\n", "reinitialization in explicit and implicit finetuning schedule modes. See the advanced usage documentation ([LR scheduler reinitialization](https://finetuning-scheduler.readthedocs.io/en/stable/advanced/lr_scheduler_reinitialization.html), [optimizer reinitialization](https://finetuning-scheduler.readthedocs.io/en/stable/advanced/optimizer_reinitialization.html)) for explanations and demonstration of the extension's support for more complex requirements.\n", ""]}, {"cell_type": "code", "execution_count": 9, "id": "72ba258b", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:39.028023Z", "iopub.status.busy": "2023-10-04T01:00:39.027622Z", "iopub.status.idle": "2023-10-04T01:00:39.032984Z", "shell.execute_reply": "2023-10-04T01:00:39.031883Z"}, "papermill": {"duration": 0.020776, "end_time": "2023-10-04T01:00:39.034353", "exception": false, "start_time": "2023-10-04T01:00:39.013577", "status": "completed"}, "tags": []}, "outputs": [], "source": ["lr_scheduler_init = {\"T_0\": 1, \"T_mult\": 2, \"eta_min\": 1e-07}"]}, {"cell_type": "code", "execution_count": 10, "id": "730e4cfe", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:00:39.063256Z", "iopub.status.busy": "2023-10-04T01:00:39.062452Z", "iopub.status.idle": "2023-10-04T01:00:49.302267Z", "shell.execute_reply": "2023-10-04T01:00:49.301385Z"}, "papermill": {"duration": 10.256623, "end_time": "2023-10-04T01:00:49.304522", "exception": false, "start_time": "2023-10-04T01:00:39.047899", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "473160f448d9410d83df085bd4afb760", "version_major": 2, "version_minor": 0}, "text/plain": ["Downloading pytorch_model.bin: 0%| | 0.00/371M [00:00 None:\n", " trainer = L.Trainer(\n", " enable_progress_bar=enable_progress_bar,\n", " max_epochs=100,\n", " precision=\"16-mixed\",\n", " accelerator=\"auto\",\n", " devices=1,\n", " callbacks=callbacks,\n", " logger=logger,\n", " )\n", " trainer.fit(model, datamodule=dm)\n", "\n", "\n", "print(\n", " \"Note given the computation associated w/ the multiple phases of fine-tuning demonstrated, this notebook is best used with an accelerator\"\n", ")\n", "train()"]}, {"cell_type": "markdown", "id": "cf8a19a2", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.027017, "end_time": "2023-10-04T01:02:20.842272", "exception": false, "start_time": "2023-10-04T01:02:20.815255", "status": "completed"}, "tags": []}, "source": ["### Running the Baseline and Implicit Fine-Tuning Scenarios\n", "\n", "Let's now compare our ``nofts_baseline`` and ``fts_implicit`` scenarios with the ``fts_explicit`` one we just ran.\n", "\n", "We'll need to update our callbacks list, using the core PL ``EarlyStopping`` and ``ModelCheckpoint`` callbacks for the\n", "``nofts_baseline`` (which operate identically to their FTS analogs apart from the recursive training support).\n", "For both core Lightning and user-registered callbacks, we can define our callbacks using a dictionary as we do\n", "with the LightningCLI. This allows us to avoid managing imports and support more complex configuration separated from\n", "code.\n", "\n", "Note that we'll be using identical callback configurations to the ``fts_explicit`` scenario. Keeping [max_depth](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=max_depth#finetuning_scheduler.fts.FinetuningScheduler.params.max_depth) for\n", "the implicit schedule will limit fine-tuning to just the last 4 parameters of the model, which is only a small fraction\n", "of the parameters you'd want to tune for maximum performance. Since the implicit schedule is quite computationally\n", "intensive and most useful for exploring model behavior, leaving [max_depth](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=max_depth#finetuning_scheduler.fts.FinetuningScheduler.params.max_depth) 1 allows us to demo implicit mode\n", "behavior while keeping the computational cost and runtime of this notebook reasonable. To review how a full implicit\n", "mode run compares to the ``nofts_baseline`` and ``fts_explicit`` scenarios, please see the the following\n", "[tensorboard experiment summary](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/)."]}, {"cell_type": "code", "execution_count": 14, "id": "d604905a", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:02:20.879250Z", "iopub.status.busy": "2023-10-04T01:02:20.879059Z", "iopub.status.idle": "2023-10-04T01:02:20.885929Z", "shell.execute_reply": "2023-10-04T01:02:20.884996Z"}, "papermill": {"duration": 0.024311, "end_time": "2023-10-04T01:02:20.887249", "exception": false, "start_time": "2023-10-04T01:02:20.862938", "status": "completed"}, "tags": []}, "outputs": [], "source": ["nofts_callbacks = [EarlyStopping(**earlystopping_kwargs), ModelCheckpoint(**checkpoint_kwargs)]\n", "fts_implicit_callbacks = [\n", " fts.FinetuningScheduler(**fts_kwargs),\n", " fts.FTSEarlyStopping(**earlystopping_kwargs),\n", " fts.FTSCheckpoint(**checkpoint_kwargs),\n", "]\n", "scenario_callbacks = {\"nofts_baseline\": nofts_callbacks, \"fts_implicit\": fts_implicit_callbacks}"]}, {"cell_type": "code", "execution_count": 15, "id": "29ee874b", "metadata": {"execution": {"iopub.execute_input": "2023-10-04T01:02:20.917777Z", "iopub.status.busy": "2023-10-04T01:02:20.917414Z", "iopub.status.idle": "2023-10-04T01:04:10.217300Z", "shell.execute_reply": "2023-10-04T01:04:10.216227Z"}, "papermill": {"duration": 109.31813, "end_time": "2023-10-04T01:04:10.220002", "exception": false, "start_time": "2023-10-04T01:02:20.901872", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Using 16bit Automatic Mixed Precision (AMP)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True (cuda), used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Beginning training the 'nofts_baseline' scenario\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Missing logger folder: lightning_logs/nofts_baseline\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [4,5]\n"]}, {"name": "stderr", "output_type": "stream", "text": ["\n", " | Name | Type | Params\n", "-------------------------------------------------------------\n", "0 | model | DebertaV2ForSequenceClassification | 184 M \n", "-------------------------------------------------------------\n", "184 M Trainable params\n", "0 Non-trainable params\n", "184 M Total params\n", "737.695 Total estimated model params size (MB)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Using 16bit Automatic Mixed Precision (AMP)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True (cuda), used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Beginning training the 'fts_implicit' scenario\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Missing logger folder: lightning_logs/fts_implicit\n"]}, {"name": "stderr", "output_type": "stream", "text": ["fine-tuning schedule dumped to lightning_logs/fts_implicit/version_0/RteBoolqModule_ft_schedule.yaml.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Generated default fine-tuning schedule 'lightning_logs/fts_implicit/version_0/RteBoolqModule_ft_schedule.yaml' for iterative fine-tuning\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [4,5]\n"]}, {"name": "stderr", "output_type": "stream", "text": ["After executing the provided `configure_optimizers` method, the optimizer state differs from the configuration FinetuningScheduler expected at the beginning of scheduled fine-tuning (phase 0).\n", "Since `enforce_phase0_params` is currently set to `True` (the default), FinetuningScheduler has reconfigured the optimizer to optimize the parameters (and only those parameters) scheduled to be optimized in phase 0 of the current fine-tuning schedule.\n", "\n", "\n"]}, {"name": "stderr", "output_type": "stream", "text": ["\n", " | Name | Type | Params\n", "-------------------------------------------------------------\n", "0 | model | DebertaV2ForSequenceClassification | 184 M \n", "-------------------------------------------------------------\n", "1.5 K Trainable params\n", "184 M Non-trainable params\n", "184 M Total params\n", "737.695 Total estimated model params size (MB)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Restoring states from the checkpoint path at lightning_logs/fts_implicit/version_0/checkpoints/epoch=0-step=156.ckpt\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Restored all states from the checkpoint at lightning_logs/fts_implicit/version_0/checkpoints/epoch=0-step=156.ckpt\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Multi-phase fine-tuned training continuing at level 1.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Given the current configuration of `max_depth` (1), this training session will now end when the early stopping conditions are met or `max_epochs` (100) is reached.\n"]}], "source": ["for scenario_name, scenario_callbacks in scenario_callbacks.items():\n", " model = RteBoolqModule(**lightning_module_kwargs, experiment_tag=scenario_name)\n", " logger = TensorBoardLogger(\"lightning_logs\", name=scenario_name)\n", " callbacks = scenario_callbacks\n", " print(f\"Beginning training the '{scenario_name}' scenario\")\n", " train()"]}, {"cell_type": "markdown", "id": "c33eb874", "metadata": {"lines_to_next_cell": 0, "papermill": {"duration": 0.027773, "end_time": "2023-10-04T01:04:10.284669", "exception": false, "start_time": "2023-10-04T01:04:10.256896", "status": "completed"}, "tags": []}, "source": ["### Reviewing the Training Results\n", "\n", "See the [tensorboard experiment summaries](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/) to get a sense\n", "of the relative computational and performance tradeoffs associated with these [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) configurations.\n", "The summary compares a full ``fts_implicit`` execution to ``fts_explicit`` and ``nofts_baseline`` scenarios using DDP\n", "training with 2 GPUs. The full logs/schedules for all three scenarios are available\n", "[here](https://drive.google.com/file/d/1LrUcisRLHeJgh_BDOOD_GUBPp5iHAkoR/view?usp=sharing) and the checkpoints\n", "produced in the scenarios [here](https://drive.google.com/file/d/1t7myBgcqcZ9ax_IT9QVk-vFH_l_o5UXB/view?usp=sharing)\n", "(caution, ~3.5GB).\n", "\n", "[{height=\"315px\" width=\"492px\"}](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/#scalars&_smoothingWeight=0&runSelectionState=eyJmdHNfZXhwbGljaXQiOnRydWUsIm5vZnRzX2Jhc2VsaW5lIjpmYWxzZSwiZnRzX2ltcGxpY2l0IjpmYWxzZX0%3D)\n", "[{height=\"316px\" width=\"505px\"}](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/#scalars&_smoothingWeight=0&runSelectionState=eyJmdHNfZXhwbGljaXQiOmZhbHNlLCJub2Z0c19iYXNlbGluZSI6dHJ1ZSwiZnRzX2ltcGxpY2l0IjpmYWxzZX0%3D)\n", "\n", "Note that given execution context differences, there could be a modest variation in performance from the tensorboard summaries generated by this notebook.\n", "\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) expands the space of possible fine-tuning schedules and the composition of more sophisticated schedules can\n", "yield marginal fine-tuning performance gains. That stated, it should be emphasized the primary utility of [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) is to grant\n", "greater fine-tuning flexibility for model exploration in research. For example, glancing at DeBERTa-v3's implicit training\n", "run, a critical tuning transition point is immediately apparent:\n", "\n", "[{height=\"272px\" width=\"494px\"}](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/#scalars&_smoothingWeight=0&runSelectionState=eyJmdHNfZXhwbGljaXQiOmZhbHNlLCJub2Z0c19iYXNlbGluZSI6ZmFsc2UsImZ0c19pbXBsaWNpdCI6dHJ1ZX0%3D)\n", "\n", "Our `val_loss` begins a precipitous decline at step 3119 which corresponds to phase 17 in the schedule. Referring to our\n", "schedule, in phase 17 we're beginning tuning the attention parameters of our 10th encoder layer (of 11). Interesting!\n", "Though beyond the scope of this tutorial, it might be worth investigating these dynamics further and\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) allows one to do just that quite easily.\n"]}, {"cell_type": "markdown", "id": "0d7af80a", "metadata": {"lines_to_next_cell": 0, "papermill": {"duration": 0.015729, "end_time": "2023-10-04T01:04:10.321609", "exception": false, "start_time": "2023-10-04T01:04:10.305880", "status": "completed"}, "tags": []}, "source": ["\n", "Note that though this example is intended to capture a common usage scenario, substantial variation is expected\n", "among use cases and models.\n", "In summary, [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) provides increased fine-tuning flexibility that can be useful in a variety of\n", "contexts from exploring model tuning behavior to maximizing performance."]}, {"cell_type": "markdown", "id": "78b6eb63", "metadata": {"papermill": {"duration": 0.015777, "end_time": "2023-10-04T01:04:10.353220", "exception": false, "start_time": "2023-10-04T01:04:10.337443", "status": "completed"}, "tags": []}, "source": ["## Footnotes\n", "\n", "- [Howard, J., & Ruder, S. (2018)](https://arxiv.org/pdf/1801.06146.pdf). Fine-tuned Language\n", " Models for Text Classification. ArXiv, abs/1801.06146. [\u21a9](#Scheduled-Fine-Tuning-with-the-Fine-Tuning-Scheduler-Extension)\n", "- [Chronopoulou, A., Baziotis, C., & Potamianos, A. (2019)](https://arxiv.org/pdf/1902.10547.pdf).\n", " An embarrassingly simple approach for transfer learning from pretrained language models. arXiv\n", " preprint arXiv:1902.10547. [\u21a9](#Scheduled-Fine-Tuning-with-the-Fine-Tuning-Scheduler-Extension)\n", "- [Peters, M. E., Ruder, S., & Smith, N. A. (2019)](https://arxiv.org/pdf/1903.05987.pdf). To tune or not to\n", " tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987. [\u21a9](#Scheduled-Fine-Tuning-with-the-Fine-Tuning-Scheduler-Extension)\n", "- [Sivaprasad, P. T., Mai, F., Vogels, T., Jaggi, M., & Fleuret, F. (2020)](https://arxiv.org/pdf/1910.11758.pdf).\n", " Optimizer benchmarking needs to account for hyperparameter tuning. In International Conference on Machine Learning\n", "(pp. 9036-9045). PMLR. [\u21a9](#Optimizer-Configuration)\n", "- [Mosbach, M., Andriushchenko, M., & Klakow, D. (2020)](https://arxiv.org/pdf/2006.04884.pdf). On the stability of\n", "fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884. [\u21a9](#Optimizer-Configuration)\n", "- [Loshchilov, I., & Hutter, F. (2016)](https://arxiv.org/pdf/1608.03983.pdf). Sgdr: Stochastic gradient descent with\n", "warm restarts. arXiv preprint arXiv:1608.03983. [\u21a9](#LR-Scheduler-Configuration)\n"]}, {"cell_type": "markdown", "id": "d6053aea", "metadata": {"papermill": {"duration": 0.015895, "end_time": "2023-10-04T01:04:10.385214", "exception": false, "start_time": "2023-10-04T01:04:10.369319", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/Lightning-AI/lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://www.pytorchlightning.ai/community)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/Lightning-AI/lightning) or [Bolt](https://github.com/Lightning-AI/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/Lightning-AI/lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/Lightning-AI/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "[{height=\"60px\" width=\"240px\"}](https://pytorchlightning.ai)"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Fine-Tuning Scheduler\n", " :card_description: This notebook introduces the [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) extension and demonstrates the use of it to fine-tune a...\n", " :tags: Fine-Tuning,GPU/TPU,Lightning-Examples"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab_type,colab,id,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.12"}, "papermill": {"default_parameters": {}, "duration": 227.374811, "end_time": "2023-10-04T01:04:12.669591", "environment_variables": {}, "exception": null, "input_path": "lightning_examples/finetuning-scheduler/finetuning-scheduler.ipynb", "output_path": ".notebooks/lightning_examples/finetuning-scheduler.ipynb", "parameters": {}, "start_time": "2023-10-04T01:00:25.294780", "version": "2.4.0"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"0045c9e4a65349f4b34b1d3328af2030": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "04559d1a68184a5bb77b963279b18964": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_4ad991ecf5bc4e5a89ca6cd5e38f0702", "placeholder": "\u200b", "style": "IPY_MODEL_a7e70b518cb04a4c8a02b5162e725420", "tabbable": null, "tooltip": null, "value": "Downloading spm.model: 100%"}}, "095b6a1381b64332bc8a4c226327bfba": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "0bf65bc277464cd29651a43f7eadbe32": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_4044f3641689494280f460d5c0c50958", "placeholder": "\u200b", "style": "IPY_MODEL_0045c9e4a65349f4b34b1d3328af2030", "tabbable": null, "tooltip": null, "value": "Downloading (\u2026)lve/main/config.json: 100%"}}, "11fa1564b8fa4190936effa34750e802": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_0bf65bc277464cd29651a43f7eadbe32", "IPY_MODEL_92fd2de9fe234b5b87c21c687cae4dcf", "IPY_MODEL_58d941f718264bbdb144ed2b756e125e"], "layout": "IPY_MODEL_7863820e3d2a42638d05a17ce33d9e23", "tabbable": null, "tooltip": null}}, "12246c0087114d80bbc4c6e2cd18b9f2": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_ade14e595c2a40d58069c3ba20f6704e", "IPY_MODEL_73ff1711ddcd4a1b971a05087255e17b", "IPY_MODEL_1ea95007d81e45aabafe10406d27f494"], "layout": "IPY_MODEL_5bd9e3c200d345ea920cb30cd062b5f7", "tabbable": null, "tooltip": null}}, "1b96320a9b864da58fbce65c67beff12": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "1ea95007d81e45aabafe10406d27f494": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_eef49f2317a54c1ea26e1012d2b80c53", "placeholder": "\u200b", "style": "IPY_MODEL_fb71138dced94a259755cedf48c700ee", "tabbable": null, "tooltip": null, "value": " 3.72k/3.72k [00:00<00:00, 607kB/s]"}}, "300ab00550724d42b403f3086097f349": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_9761eb5388384cddbcb20b28527f7019", "max": 52.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_39f571a38ad045c8940f3dafedd43cd8", "tabbable": null, "tooltip": null, "value": 52.0}}, "302bc617b537419f968fc3a93543d1b8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_b2557bf0946e4b648e5c60ca49fbd09f", "placeholder": "\u200b", "style": "IPY_MODEL_32cb3c775dc5423fb0e908d0302b761d", "tabbable": null, "tooltip": null, "value": "Downloading builder script: 100%"}}, "32cb3c775dc5423fb0e908d0302b761d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "361fbe61b3d04a8ba9061f147cdb9c65": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "39f571a38ad045c8940f3dafedd43cd8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "4044f3641689494280f460d5c0c50958": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "473160f448d9410d83df085bd4afb760": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_cd2097905c974f6bb7701750424dda9b", "IPY_MODEL_cb0f5c72f4aa4005b98e79e9ce7ed7aa", "IPY_MODEL_d6f9e0f2911c408d8db81271d3eba168"], "layout": "IPY_MODEL_e4c840e6b8e54a4e81aca7d66b708ffc", "tabbable": null, "tooltip": null}}, "4ad991ecf5bc4e5a89ca6cd5e38f0702": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "5000b81d9bc649d58ff3fbb098e05ba6": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "51e56e6d98334f97ba5ce0ac4e8b022f": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "525309b45da74927a28f00c6e7c585a2": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "58d941f718264bbdb144ed2b756e125e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_72754e179be04771bce8290dc5cf69b5", "placeholder": "\u200b", "style": "IPY_MODEL_bd0cfbcf38144ab4932009bd001db2b4", "tabbable": null, "tooltip": null, "value": " 579/579 [00:00<00:00, 50.1kB/s]"}}, "5bd9e3c200d345ea920cb30cd062b5f7": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "60af281fbe3a4382827d4589b4c521dd": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "6532ec7c0a6646c892c40a969c1c4281": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "691e6cd7d33b487e9efe0dc09b3afe31": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "714e6201592d499e8c72619afc9b1a1a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_302bc617b537419f968fc3a93543d1b8", "IPY_MODEL_c7b39992515c44edb12e395f5fc825a5", "IPY_MODEL_bd339db0b1944741aad01b778cada398"], "layout": "IPY_MODEL_361fbe61b3d04a8ba9061f147cdb9c65", "tabbable": null, "tooltip": null}}, "72754e179be04771bce8290dc5cf69b5": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "73ff1711ddcd4a1b971a05087255e17b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_ecbe440d001840ffb30a19eb89c19224", "max": 3724.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_9722bae57230451b8e7a2d75c7ecc738", "tabbable": null, "tooltip": null, "value": 3724.0}}, "7863820e3d2a42638d05a17ce33d9e23": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "7c6bad07fce443efb64239924c3c67ca": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_e30bed358a524ff9b847273cfbb02d9f", "IPY_MODEL_300ab00550724d42b403f3086097f349", "IPY_MODEL_a25b673d34a84db39d0d64061265eef9"], "layout": "IPY_MODEL_d649794571c94841b27791adad7a1c27", "tabbable": null, "tooltip": null}}, "803da9ca8e614686b24aa71b2413ddc1": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8043050af1354c9e8904b82579011fd0": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8ba060a6a73440ceac5860a3f512e192": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "8be70ce7e278417ea036cf7a802a9e69": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "92fd2de9fe234b5b87c21c687cae4dcf": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_525309b45da74927a28f00c6e7c585a2", "max": 579.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_bf6d47d177d945dfa01145b3780e1680", "tabbable": null, "tooltip": null, "value": 579.0}}, "9671ec27d8aa45b594f45743e529e204": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_a748911351e243ea8ae3c4be9067ed29", "max": 2464616.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_a5fe4b43bd6a416f959db3b18a14e75a", "tabbable": null, "tooltip": null, "value": 2464616.0}}, "9722bae57230451b8e7a2d75c7ecc738": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "975f9d8c8483412388be188fad051077": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "9761eb5388384cddbcb20b28527f7019": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "a1a56537c1274537a173c82ce76fb004": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "a25b673d34a84db39d0d64061265eef9": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_e38ee12e1d644e61b07d34c2975f75bd", "placeholder": "\u200b", "style": "IPY_MODEL_dd1a41dbc6fb4897bb7f51f50e787b52", "tabbable": null, "tooltip": null, "value": " 52.0/52.0 [00:00<00:00, 4.15kB/s]"}}, "a5fe4b43bd6a416f959db3b18a14e75a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "a748911351e243ea8ae3c4be9067ed29": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "a7e70b518cb04a4c8a02b5162e725420": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "ade14e595c2a40d58069c3ba20f6704e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_691e6cd7d33b487e9efe0dc09b3afe31", "placeholder": "\u200b", "style": "IPY_MODEL_975f9d8c8483412388be188fad051077", "tabbable": null, "tooltip": null, "value": "Downloading extra modules: 100%"}}, "b2557bf0946e4b648e5c60ca49fbd09f": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "b78b2619981447d1bde12251af48aab3": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "bd0cfbcf38144ab4932009bd001db2b4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "bd339db0b1944741aad01b778cada398": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_51e56e6d98334f97ba5ce0ac4e8b022f", "placeholder": "\u200b", "style": "IPY_MODEL_60af281fbe3a4382827d4589b4c521dd", "tabbable": null, "tooltip": null, "value": " 9.64k/9.64k [00:00<00:00, 1.29MB/s]"}}, "bf6d47d177d945dfa01145b3780e1680": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "c7b39992515c44edb12e395f5fc825a5": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_8be70ce7e278417ea036cf7a802a9e69", "max": 9644.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_a1a56537c1274537a173c82ce76fb004", "tabbable": null, "tooltip": null, "value": 9644.0}}, "c7b3fa42d3a1443e9df3c08e90c68ae0": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "cb0f5c72f4aa4005b98e79e9ce7ed7aa": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_6532ec7c0a6646c892c40a969c1c4281", "max": 371146213.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_1b96320a9b864da58fbce65c67beff12", "tabbable": null, "tooltip": null, "value": 371146213.0}}, "cd2097905c974f6bb7701750424dda9b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_8ba060a6a73440ceac5860a3f512e192", "placeholder": "\u200b", "style": "IPY_MODEL_095b6a1381b64332bc8a4c226327bfba", "tabbable": null, "tooltip": null, "value": "Downloading pytorch_model.bin: 100%"}}, "d649794571c94841b27791adad7a1c27": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "d6f9e0f2911c408d8db81271d3eba168": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_8043050af1354c9e8904b82579011fd0", "placeholder": "\u200b", "style": "IPY_MODEL_b78b2619981447d1bde12251af48aab3", "tabbable": null, "tooltip": null, "value": " 371M/371M [00:07<00:00, 48.7MB/s]"}}, "d7138c99622f4e608247e641553235db": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "dd1a41dbc6fb4897bb7f51f50e787b52": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "e30bed358a524ff9b847273cfbb02d9f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_5000b81d9bc649d58ff3fbb098e05ba6", "placeholder": "\u200b", "style": "IPY_MODEL_c7b3fa42d3a1443e9df3c08e90c68ae0", "tabbable": null, "tooltip": null, "value": "Downloading (\u2026)okenizer_config.json: 100%"}}, "e38ee12e1d644e61b07d34c2975f75bd": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "e4c840e6b8e54a4e81aca7d66b708ffc": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "ecbe440d001840ffb30a19eb89c19224": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "edeb549f9e9447f8bb31d9c7838406fe": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "eef49f2317a54c1ea26e1012d2b80c53": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "fb71138dced94a259755cedf48c700ee": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "fb912cd28b2a4cc6b6b40f6d221bd168": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_04559d1a68184a5bb77b963279b18964", "IPY_MODEL_9671ec27d8aa45b594f45743e529e204", "IPY_MODEL_feab3d184baa43d59c96ad9d2bb63681"], "layout": "IPY_MODEL_803da9ca8e614686b24aa71b2413ddc1", "tabbable": null, "tooltip": null}}, "feab3d184baa43d59c96ad9d2bb63681": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_d7138c99622f4e608247e641553235db", "placeholder": "\u200b", "style": "IPY_MODEL_edeb549f9e9447f8bb31d9c7838406fe", "tabbable": null, "tooltip": null, "value": " 2.46M/2.46M [00:00<00:00, 19.5MB/s]"}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}