{"cells": [{"cell_type": "markdown", "id": "43e3a29f", "metadata": {"papermill": {"duration": 0.017122, "end_time": "2023-03-15T10:45:06.228120", "exception": false, "start_time": "2023-03-15T10:45:06.210998", "status": "completed"}, "tags": []}, "source": ["\n", "# Fine-Tuning Scheduler\n", "\n", "* **Author:** [Dan Dale](https://github.com/speediedan)\n", "* **License:** CC BY-SA\n", "* **Generated:** 2023-03-15T10:44:42.950992\n", "\n", "This notebook introduces the [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) extension\n", "and demonstrates the use of it to fine-tune a small foundation model on the\n", "[RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task of\n", "[SuperGLUE](https://super.gluebenchmark.com/) with iterative early-stopping defined according to a user-specified\n", "schedule. It uses Hugging Face's ``datasets`` and ``transformers`` libraries to retrieve the relevant benchmark data\n", "and foundation model weights. The required dependencies are installed via the finetuning-scheduler ``[examples]`` extra.\n", "\n", "\n", "---\n", "Open in [{height=\"20px\" width=\"117px\"}](https://colab.research.google.com/github/PytorchLightning/lightning-tutorials/blob/publication/.notebooks/lightning_examples/finetuning-scheduler.ipynb)\n", "\n", "Give us a \u2b50 [on Github](https://www.github.com/Lightning-AI/lightning/)\n", "| Check out [the documentation](https://pytorch-lightning.readthedocs.io/en/stable/)\n", "| Join us [on Slack](https://www.pytorchlightning.ai/community)"]}, {"cell_type": "markdown", "id": "7bd4e559", "metadata": {"papermill": {"duration": 0.010119, "end_time": "2023-03-15T10:45:06.248897", "exception": false, "start_time": "2023-03-15T10:45:06.238778", "status": "completed"}, "tags": []}, "source": ["## Setup\n", "This notebook requires some packages besides pytorch-lightning."]}, {"cell_type": "code", "execution_count": 1, "id": "a9524551", "metadata": {"colab": {}, "colab_type": "code", "execution": {"iopub.execute_input": "2023-03-15T10:45:06.272368Z", "iopub.status.busy": "2023-03-15T10:45:06.271551Z", "iopub.status.idle": "2023-03-15T10:45:10.556894Z", "shell.execute_reply": "2023-03-15T10:45:10.555484Z"}, "id": "LfrJLKPFyhsK", "lines_to_next_cell": 0, "papermill": {"duration": 4.300809, "end_time": "2023-03-15T10:45:10.559773", "exception": false, "start_time": "2023-03-15T10:45:06.258964", "status": "completed"}, "tags": []}, "outputs": [{"name": "stdout", "output_type": "stream", "text": ["\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\r\n", "\u001b[0m"]}], "source": ["! pip install --quiet \"setuptools==67.4.0\" \"lightning>=2.0.0rc0\" \"ipython[notebook]>=8.0.0, <8.12.0\" \"datasets<2.8.0\" \"torch>=1.8.1, <1.14.0\" \"pytorch-lightning>=1.4, <2.0.0\" \"finetuning-scheduler[examples]>=0.4.0\" \"torchmetrics>=0.7, <0.12\""]}, {"cell_type": "markdown", "id": "464d6be9", "metadata": {"papermill": {"duration": 0.044229, "end_time": "2023-03-15T10:45:10.619512", "exception": false, "start_time": "2023-03-15T10:45:10.575283", "status": "completed"}, "tags": []}, "source": ["## Scheduled Fine-Tuning with the Fine-Tuning Scheduler Extension\n", "\n", "{height=\"55px\" width=\"401px\"}\n", "\n", "The [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) extension accelerates and enhances model experimentation with flexible fine-tuning schedules.\n", "\n", "Training with the extension is simple and confers a host of benefits:\n", "\n", "- it dramatically increases fine-tuning flexibility\n", "- expedites and facilitates exploration of model tuning dynamics\n", "- enables marginal performance improvements of fine-tuned models\n", "\n", "Setup is straightforward, just install from PyPI! Since this notebook-based example requires a few additional packages (e.g.\n", "``transformers``, ``sentencepiece``), we installed the ``finetuning-scheduler`` package with the ``[examples]`` extra above.\n", "Once the ``finetuning-scheduler`` package is installed, the [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) callback is available for use with PyTorch Lightning.\n", "For additional installation options, please see the Fine-Tuning Scheduler [README](https://github.com/speediedan/finetuning-scheduler/blob/main/README.md).\n", "\n", "\n", "\n", "\n", "\n", "Fundamentally, [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) enables\n", "scheduled, multi-phase, fine-tuning of foundation models. Gradual unfreezing (i.e. thawing) can help maximize\n", "foundation model knowledge retention while allowing (typically upper layers of) the model to\n", "optimally adapt to new tasks during transfer learning [1, 2, 3](#f1)\n", "\n", "
\n", "\n", "The [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) callback orchestrates the gradual unfreezing\n", "of models via a fine-tuning schedule that is either implicitly generated (the default) or explicitly provided by the user\n", "(more computationally efficient). Fine-tuning phase transitions are driven by\n", "[FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping)\n", "criteria (a multi-phase extension of ``EarlyStopping`` packaged with FinetuningScheduler), user-specified epoch transitions or a composition of the two (the default mode).\n", "A [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) training session completes when the\n", "final phase of the schedule has its stopping criteria met. See\n", "the [early stopping documentation](https://lightning.ai/docs/pytorch/stable/api/pytorch_lightning.callbacks.EarlyStopping.html) for more details on that callback's configuration.\n", "\n", "{height=\"272px\" width=\"376px\"}"]}, {"cell_type": "markdown", "id": "1e2d7b6d", "metadata": {"papermill": {"duration": 0.010307, "end_time": "2023-03-15T10:45:10.640199", "exception": false, "start_time": "2023-03-15T10:45:10.629892", "status": "completed"}, "tags": []}, "source": ["\n", "## Basic Usage\n", "\n", "\n", "\n", "If no fine-tuning schedule is provided by the user, [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) will generate a\n", "[default schedule](#The-Default-Finetuning-Schedule) and proceed to fine-tune according to the generated schedule,\n", "using default [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) and [FTSCheckpoint](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSCheckpoint) callbacks with ``monitor=val_loss``.\n", "\n", "
\n", "\n", "```python\n", "from pytorch_lightning import Trainer\n", "from finetuning_scheduler import FinetuningScheduler\n", "trainer = Trainer(callbacks=[FinetuningScheduler()])\n", "```"]}, {"cell_type": "markdown", "id": "55c21c88", "metadata": {"papermill": {"duration": 0.010106, "end_time": "2023-03-15T10:45:10.660438", "exception": false, "start_time": "2023-03-15T10:45:10.650332", "status": "completed"}, "tags": []}, "source": ["## The Default Fine-Tuning Schedule\n", "\n", "Schedule definition is facilitated via the [gen_ft_schedule](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.ScheduleImplMixin.gen_ft_schedule) method which dumps a default fine-tuning schedule (by default using a naive, 2-parameters per level heuristic) which can be adjusted as\n", "desired by the user and/or subsequently passed to the callback. Using the default/implicitly generated schedule will likely be less computationally efficient than a user-defined fine-tuning schedule but is useful for exploring a model's fine-tuning behavior and can serve as a good baseline for subsequent explicit schedule refinement.\n", "While the current version of [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) only supports single optimizer and (optional) lr_scheduler configurations, per-phase maximum learning rates can be set as demonstrated in the next section."]}, {"cell_type": "markdown", "id": "dfba32a0", "metadata": {"papermill": {"duration": 0.010112, "end_time": "2023-03-15T10:45:10.680761", "exception": false, "start_time": "2023-03-15T10:45:10.670649", "status": "completed"}, "tags": []}, "source": ["## Specifying a Fine-Tuning Schedule\n", "\n", "To specify a fine-tuning schedule, it's convenient to first generate the default schedule and then alter the thawed/unfrozen parameter groups associated with each fine-tuning phase as desired. Fine-tuning phases are zero-indexed and executed in ascending order.\n", "\n", "1. First, generate the default schedule to ``Trainer.log_dir``. It will be named after your\n", " ``LightningModule`` subclass with the suffix ``_ft_schedule.yaml``.\n", "\n", "```python\n", " from pytorch_lightning import Trainer\n", " from finetuning_scheduler import FinetuningScheduler\n", " trainer = Trainer(callbacks=[FinetuningScheduler(gen_ft_sched_only=True)])\n", "```\n", "\n", "2. Alter the schedule as desired.\n", "\n", "{height=\"327px\" width=\"800px\"}\n", "\n", "3. Once the fine-tuning schedule has been altered as desired, pass it to\n", " [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) to commence scheduled training:\n", "\n", "```python\n", "from pytorch_lightning import Trainer\n", "from finetuning_scheduler import FinetuningScheduler\n", "\n", "trainer = Trainer(callbacks=[FinetuningScheduler(ft_schedule=\"/path/to/my/schedule/my_schedule.yaml\")])\n", "```"]}, {"cell_type": "markdown", "id": "55b53f9f", "metadata": {"papermill": {"duration": 0.010155, "end_time": "2023-03-15T10:45:10.701347", "exception": false, "start_time": "2023-03-15T10:45:10.691192", "status": "completed"}, "tags": []}, "source": ["## Early-Stopping and Epoch-Driven Phase Transition Criteria\n", "\n", "\n", "By default, [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) and epoch-driven\n", "transition criteria are composed. If a ``max_transition_epoch`` is specified for a given phase, the next fine-tuning phase will begin at that epoch unless [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) criteria are met first.\n", "If [FinetuningScheduler.epoch_transitions_only](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler.params.epoch_transitions_only) is ``True``, [FTSEarlyStopping](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSEarlyStopping) will not be used\n", "and transitions will be exclusively epoch-driven.\n", "\n", "\n", "\n", "\n", "**Tip:** Use of regex expressions can be convenient for specifying more complex schedules. Also, a per-phase base maximum lr can be specified:\n", "\n", "{height=\"380px\" width=\"800px\"}\n", "\n", "
\n", "\n", "\n", "\n", "The end-to-end example in this notebook ([Scheduled Fine-Tuning For SuperGLUE](#superglue)) uses [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) in explicit mode to fine-tune a small foundation model on the [RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task of [SuperGLUE](https://super.gluebenchmark.com/).\n", "Please see the [official Fine-Tuning Scheduler documentation](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) if you are interested in a similar [CLI-based example](https://finetuning-scheduler.readthedocs.io/en/stable/index.html#example-scheduled-fine-tuning-for-superglue) using the LightningCLI."]}, {"cell_type": "markdown", "id": "b5193718", "metadata": {"papermill": {"duration": 0.01014, "end_time": "2023-03-15T10:45:10.721696", "exception": false, "start_time": "2023-03-15T10:45:10.711556", "status": "completed"}, "tags": []}, "source": ["## Resuming Scheduled Fine-Tuning Training Sessions\n", "\n", "Resumption of scheduled fine-tuning training is identical to the continuation of\n", "[other training sessions](https://lightning.ai/docs/pytorch/stable/common/trainer.html) with the caveat that the provided checkpoint must have been saved by a [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) session.\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) uses [FTSCheckpoint](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSCheckpoint) (an extension of ``ModelCheckpoint``) to maintain schedule state with special metadata.\n", "\n", "\n", "```python\n", "from pytorch_lightning import Trainer\n", "from finetuning_scheduler import FinetuningScheduler\n", "trainer = Trainer(callbacks=[FinetuningScheduler()])\n", "trainer.fit(..., ckpt_path=\"some/path/to/my_checkpoint.ckpt\")\n", "```\n", "\n", "Training will resume at the depth/level of the provided checkpoint according to the specified schedule. Schedules can be altered between training sessions but schedule compatibility is left to the user for maximal flexibility. If executing a user-defined schedule, typically the same schedule should be provided for the original and resumed training sessions.\n", "\n", "By default ([FinetuningScheduler.restore_best](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=restore_best#finetuning_scheduler.fts.FinetuningScheduler.params.restore_best) is ``True``), [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) will attempt to restore the best available checkpoint before fine-tuning depth transitions.\n", "\n", "```python\n", "trainer = Trainer(callbacks=[FinetuningScheduler()])\n", "trainer.fit(..., ckpt_path=\"some/path/to/my_kth_best_checkpoint.ckpt\")\n", "```\n", "\n", "Note that similar to the behavior of [ModelCheckpoint](https://lightning.ai/docs/pytorch/stable/api/pytorch_lightning.callbacks.ModelCheckpoint.html), (specifically [this PR](https://github.com/Lightning-AI/lightning/pull/12045)),\n", "when resuming training with a different [FTSCheckpoint](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts_supporters.html#finetuning_scheduler.fts_supporters.FTSCheckpoint) ``dirpath`` from the provided\n", "checkpoint, the new training session's checkpoint state will be re-initialized at the resumption depth with the provided checkpoint being set as the best checkpoint."]}, {"cell_type": "markdown", "id": "6165a0a5", "metadata": {"papermill": {"duration": 0.010646, "end_time": "2023-03-15T10:45:10.743715", "exception": false, "start_time": "2023-03-15T10:45:10.733069", "status": "completed"}, "tags": []}, "source": ["\n", "\n", "**Note:** Currently, [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) supports the following strategy types:\n", "\n", "- ``ddp`` (and alias ``ddp_find_unused_parameters_false``)\n", "- ``fsdp_native`` (and alias ``fsdp_native_full_shard_offload``)\n", "- ``ddp_spawn`` (and aliases ``ddp_fork``, ``ddp_notebook``)\n", "- ``dp``\n", "- ``ddp_sharded`` (deprecated, to be removed in 2.0)\n", "- ``ddp_sharded_spawn`` (deprecated, to be removed in 2.0)\n", "\n", "Custom or officially unsupported strategies can be used by setting [FinetuningScheduler.allow_untested](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=allow_untested#finetuning_scheduler.fts.FinetuningScheduler.params.allow_untested) to ``True``.\n", "Note that most currently unsupported strategies are so because they require varying degrees of modification to be compatible. For example, ``deepspeed`` will require a [StrategyAdapter](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.strategy_adapters.html#finetuning_scheduler.strategy_adapters.StrategyAdapter) to be written (similar to the one for ``FSDP``, [FSDPStrategyAdapter](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.strategy_adapters.html#finetuning_scheduler.strategy_adapters.FSDPStrategyAdapter)) before support can be added (PRs welcome!),\n", "while ``tpu_spawn`` would require an override of the current broadcast method to include python objects.\n", "
"]}, {"cell_type": "markdown", "id": "06831f78", "metadata": {"papermill": {"duration": 0.010147, "end_time": "2023-03-15T10:45:10.764104", "exception": false, "start_time": "2023-03-15T10:45:10.753957", "status": "completed"}, "tags": []}, "source": ["\n", "\n", "## Scheduled Fine-Tuning For SuperGLUE\n", "\n", "The following example demonstrates the use of [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) to fine-tune a small foundation model on the [RTE](https://huggingface.co/datasets/viewer/?dataset=super_glue&config=rte) task of [SuperGLUE](https://super.gluebenchmark.com/). Iterative early-stopping will be applied according to a user-specified schedule.\n"]}, {"cell_type": "code", "execution_count": 2, "id": "b0872a60", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:10.786670Z", "iopub.status.busy": "2023-03-15T10:45:10.786298Z", "iopub.status.idle": "2023-03-15T10:45:12.934198Z", "shell.execute_reply": "2023-03-15T10:45:12.932946Z"}, "papermill": {"duration": 2.163052, "end_time": "2023-03-15T10:45:12.937402", "exception": false, "start_time": "2023-03-15T10:45:10.774350", "status": "completed"}, "tags": []}, "outputs": [], "source": ["import os\n", "import warnings\n", "from datetime import datetime\n", "from typing import Any, Dict, List, Optional\n", "\n", "from packaging.version import Version\n", "\n", "import sentencepiece as sp # noqa: F401 # isort: split\n", "import datasets\n", "import evaluate\n", "import pytorch_lightning as pl\n", "import torch\n", "from datasets import logging as datasets_logging\n", "from lightning_fabric.accelerators.cuda import is_cuda_available\n", "from pytorch_lightning.callbacks import EarlyStopping, ModelCheckpoint\n", "from pytorch_lightning.loggers.tensorboard import TensorBoardLogger\n", "from pytorch_lightning.utilities import rank_zero_warn\n", "from torch.optim.lr_scheduler import CosineAnnealingWarmRestarts\n", "from torch.utils.data import DataLoader\n", "from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer\n", "from transformers import logging as transformers_logging\n", "from transformers.tokenization_utils_base import BatchEncoding\n", "\n", "if Version(torch.__version__) == Version(\"1.12.0\") or torch.__version__.startswith(\"1.12.0\"):\n", " # we need to use a patched version of AdamW to fix https://github.com/pytorch/pytorch/issues/80809\n", " # and allow examples to succeed with torch 1.12.0 (this torch bug is fixed in 1.12.1)\n", " from fts_examples.patched_adamw import AdamW\n", "else:\n", " from torch.optim.adamw import AdamW"]}, {"cell_type": "code", "execution_count": 3, "id": "54c49e01", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:12.964936Z", "iopub.status.busy": "2023-03-15T10:45:12.964198Z", "iopub.status.idle": "2023-03-15T10:45:12.977056Z", "shell.execute_reply": "2023-03-15T10:45:12.976248Z"}, "papermill": {"duration": 0.027258, "end_time": "2023-03-15T10:45:12.979548", "exception": false, "start_time": "2023-03-15T10:45:12.952290", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Import the `FinetuningScheduler` PyTorch Lightning extension module we want to use. This will import all necessary callbacks.\n", "import finetuning_scheduler as fts # isort: split\n", "\n", "# set notebook-level variables\n", "TASK_NUM_LABELS = {\"boolq\": 2, \"rte\": 2}\n", "DEFAULT_TASK = \"rte\"\n", "\n", "# reduce hf logging verbosity to focus on tutorial-relevant code/messages\n", "for hflogger in [transformers_logging, datasets_logging]:\n", " hflogger.set_verbosity_error()\n", "# ignore warnings related tokenizers_parallelism/DataLoader parallelism trade-off and\n", "# expected logging behavior\n", "for warnf in [\n", " r\".*does not have many workers.*\",\n", " r\".*The number of training samples.*\",\n", " r\".*converting to a fast.*\",\n", " r\".*number of training batches.*\",\n", "]:\n", " warnings.filterwarnings(\"ignore\", warnf)"]}, {"cell_type": "code", "execution_count": 4, "id": "e0a7f18e", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:13.009256Z", "iopub.status.busy": "2023-03-15T10:45:13.008892Z", "iopub.status.idle": "2023-03-15T10:45:13.027010Z", "shell.execute_reply": "2023-03-15T10:45:13.026226Z"}, "papermill": {"duration": 0.032489, "end_time": "2023-03-15T10:45:13.029400", "exception": false, "start_time": "2023-03-15T10:45:12.996911", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class RteBoolqDataModule(pl.LightningDataModule):\n", " \"\"\"A ``LightningDataModule`` designed for both the RTE or BoolQ SuperGLUE Hugging Face datasets.\"\"\"\n", "\n", " TASK_TEXT_FIELD_MAP = {\"rte\": (\"premise\", \"hypothesis\"), \"boolq\": (\"question\", \"passage\")}\n", " LOADER_COLUMNS = (\n", " \"datasets_idx\",\n", " \"input_ids\",\n", " \"token_type_ids\",\n", " \"attention_mask\",\n", " \"start_positions\",\n", " \"end_positions\",\n", " \"labels\",\n", " )\n", "\n", " def __init__(\n", " self,\n", " model_name_or_path: str,\n", " task_name: str = DEFAULT_TASK,\n", " max_seq_length: int = 128,\n", " train_batch_size: int = 16,\n", " eval_batch_size: int = 16,\n", " tokenizers_parallelism: bool = True,\n", " **dataloader_kwargs: Any,\n", " ):\n", " r\"\"\"Initialize the ``LightningDataModule`` designed for both the RTE or BoolQ SuperGLUE Hugging Face\n", " datasets.\n", "\n", " Args:\n", " model_name_or_path (str):\n", " Can be either:\n", " - A string, the ``model id`` of a pretrained model hosted inside a model repo on huggingface.co.\n", " Valid model ids can be located at the root-level, like ``bert-base-uncased``, or namespaced\n", " under a user or organization name, like ``dbmdz/bert-base-german-cased``.\n", " - A path to a ``directory`` containing model weights saved using\n", " :meth:`~transformers.PreTrainedModel.save_pretrained`, e.g., ``./my_model_directory/``.\n", " task_name (str, optional): Name of the SuperGLUE task to execute. This module supports 'rte' or 'boolq'.\n", " Defaults to DEFAULT_TASK which is 'rte'.\n", " max_seq_length (int, optional): Length to which we will pad sequences or truncate input. Defaults to 128.\n", " train_batch_size (int, optional): Training batch size. Defaults to 16.\n", " eval_batch_size (int, optional): Batch size to use for validation and testing splits. Defaults to 16.\n", " tokenizers_parallelism (bool, optional): Whether to use parallelism in the tokenizer. Defaults to True.\n", " \\**dataloader_kwargs: Arguments passed when initializing the dataloader.\n", " \"\"\"\n", " super().__init__()\n", " task_name = task_name if task_name in TASK_NUM_LABELS.keys() else DEFAULT_TASK\n", " self.text_fields = self.TASK_TEXT_FIELD_MAP[task_name]\n", " self.dataloader_kwargs = {\n", " \"num_workers\": dataloader_kwargs.get(\"num_workers\", 0),\n", " \"pin_memory\": dataloader_kwargs.get(\"pin_memory\", False),\n", " }\n", " self.save_hyperparameters()\n", " os.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\" if self.hparams.tokenizers_parallelism else \"false\"\n", " self.tokenizer = AutoTokenizer.from_pretrained(\n", " self.hparams.model_name_or_path, use_fast=True, local_files_only=False\n", " )\n", "\n", " def prepare_data(self):\n", " \"\"\"Load the SuperGLUE dataset.\"\"\"\n", " # N.B. PL calls prepare_data from a single process (rank 0) so do not use it to assign\n", " # state (e.g. self.x=y)\n", " datasets.load_dataset(\"super_glue\", self.hparams.task_name)\n", "\n", " def setup(self, stage):\n", " \"\"\"Setup our dataset splits for training/validation.\"\"\"\n", " self.dataset = datasets.load_dataset(\"super_glue\", self.hparams.task_name)\n", " for split in self.dataset.keys():\n", " self.dataset[split] = self.dataset[split].map(\n", " self._convert_to_features, batched=True, remove_columns=[\"label\"]\n", " )\n", " self.columns = [c for c in self.dataset[split].column_names if c in self.LOADER_COLUMNS]\n", " self.dataset[split].set_format(type=\"torch\", columns=self.columns)\n", "\n", " self.eval_splits = [x for x in self.dataset.keys() if \"validation\" in x]\n", "\n", " def train_dataloader(self):\n", " return DataLoader(self.dataset[\"train\"], batch_size=self.hparams.train_batch_size, **self.dataloader_kwargs)\n", "\n", " def val_dataloader(self):\n", " return DataLoader(self.dataset[\"validation\"], batch_size=self.hparams.eval_batch_size, **self.dataloader_kwargs)\n", "\n", " def _convert_to_features(self, example_batch: datasets.arrow_dataset.LazyDict) -> BatchEncoding:\n", " \"\"\"Convert raw text examples to a :class:`~transformers.tokenization_utils_base.BatchEncoding` container\n", " (derived from python dict) of features that includes helpful methods for translating between word/character\n", " space and token space.\n", "\n", " Args:\n", " example_batch ([type]): The set of examples to convert to token space.\n", "\n", " Returns:\n", " ``BatchEncoding``: A batch of encoded examples (note default tokenizer batch_size=1000).\n", " \"\"\"\n", " text_pairs = list(zip(example_batch[self.text_fields[0]], example_batch[self.text_fields[1]]))\n", " # Tokenize the text/text pairs\n", " features = self.tokenizer.batch_encode_plus(\n", " text_pairs, max_length=self.hparams.max_seq_length, padding=\"longest\", truncation=True\n", " )\n", " # Rename label to labels to make it easier to pass to model forward\n", " features[\"labels\"] = example_batch[\"label\"]\n", " return features"]}, {"cell_type": "code", "execution_count": 5, "id": "59715fbd", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:13.057782Z", "iopub.status.busy": "2023-03-15T10:45:13.057417Z", "iopub.status.idle": "2023-03-15T10:45:13.080805Z", "shell.execute_reply": "2023-03-15T10:45:13.079966Z"}, "lines_to_next_cell": 2, "papermill": {"duration": 0.03754, "end_time": "2023-03-15T10:45:13.083222", "exception": false, "start_time": "2023-03-15T10:45:13.045682", "status": "completed"}, "tags": []}, "outputs": [], "source": ["class RteBoolqModule(pl.LightningModule):\n", " \"\"\"A ``LightningModule`` that can be used to fine-tune a foundation model on either the RTE or BoolQ SuperGLUE\n", " tasks using Hugging Face implementations of a given model and the `SuperGLUE Hugging Face dataset.\"\"\"\n", "\n", " def __init__(\n", " self,\n", " model_name_or_path: str,\n", " optimizer_init: Dict[str, Any],\n", " lr_scheduler_init: Dict[str, Any],\n", " model_cfg: Optional[Dict[str, Any]] = None,\n", " task_name: str = DEFAULT_TASK,\n", " experiment_tag: str = \"default\",\n", " ):\n", " \"\"\"\n", " Args:\n", " model_name_or_path (str): Path to pretrained model or identifier from https://huggingface.co/models.\n", " optimizer_init (Dict[str, Any]): The desired optimizer configuration.\n", " lr_scheduler_init (Dict[str, Any]): The desired learning rate scheduler config.\n", " model_cfg (Optional[Dict[str, Any]], optional): Defines overrides of the default model config. Defaults to\n", " ``None``.\n", " task_name (str, optional): The SuperGLUE task to execute, one of ``'rte'``, ``'boolq'``. Defaults to \"rte\".\n", " experiment_tag (str, optional): The tag to use for the experiment and tensorboard logs. Defaults to\n", " \"default\".\n", " \"\"\"\n", " super().__init__()\n", " if task_name not in TASK_NUM_LABELS.keys():\n", " rank_zero_warn(f\"Invalid task_name {task_name!r}. Proceeding with the default task: {DEFAULT_TASK!r}\")\n", " task_name = DEFAULT_TASK\n", " self.num_labels = TASK_NUM_LABELS[task_name]\n", " self.model_cfg = model_cfg or {}\n", " conf = AutoConfig.from_pretrained(model_name_or_path, num_labels=self.num_labels, local_files_only=False)\n", " self.model = AutoModelForSequenceClassification.from_pretrained(model_name_or_path, config=conf)\n", " self.model.config.update(self.model_cfg) # apply model config overrides\n", " self.init_hparams = {\n", " \"optimizer_init\": optimizer_init,\n", " \"lr_scheduler_init\": lr_scheduler_init,\n", " \"model_config\": self.model.config,\n", " \"model_name_or_path\": model_name_or_path,\n", " \"task_name\": task_name,\n", " \"experiment_id\": f\"{datetime.now().strftime('%Y%m%d_%H%M%S')}_{experiment_tag}\",\n", " }\n", " self.save_hyperparameters(self.init_hparams)\n", " self.metric = evaluate.load(\"super_glue\", self.hparams.task_name, experiment_id=self.hparams.experiment_id)\n", " self.no_decay = [\"bias\", \"LayerNorm.weight\"]\n", "\n", " @property\n", " def finetuningscheduler_callback(self) -> fts.FinetuningScheduler:\n", " fts_callback = [c for c in self.trainer.callbacks if isinstance(c, fts.FinetuningScheduler)]\n", " return fts_callback[0] if fts_callback else None\n", "\n", " def forward(self, **inputs):\n", " return self.model(**inputs)\n", "\n", " def training_step(self, batch, batch_idx):\n", " outputs = self(**batch)\n", " loss = outputs[0]\n", " self.log(\"train_loss\", loss)\n", " return loss\n", "\n", " def training_epoch_end(self, outputs: List[Any]) -> None:\n", " if self.finetuningscheduler_callback:\n", " self.log(\"finetuning_schedule_depth\", float(self.finetuningscheduler_callback.curr_depth))\n", "\n", " def validation_step(self, batch, batch_idx, dataloader_idx=0):\n", " outputs = self(**batch)\n", " val_loss, logits = outputs[:2]\n", " if self.num_labels >= 1:\n", " preds = torch.argmax(logits, axis=1)\n", " elif self.num_labels == 1:\n", " preds = logits.squeeze()\n", " labels = batch[\"labels\"]\n", " self.log(\"val_loss\", val_loss, prog_bar=True)\n", " metric_dict = self.metric.compute(predictions=preds, references=labels)\n", " self.log_dict(metric_dict, prog_bar=True)\n", "\n", " def _init_param_groups(self) -> List[Dict]:\n", " \"\"\"Initialize the parameter groups. Used to ensure weight_decay is not applied to our specified bias\n", " parameters when we initialize the optimizer.\n", "\n", " Returns:\n", " List[Dict]: A list of parameter group dictionaries.\n", " \"\"\"\n", " return [\n", " {\n", " \"params\": [\n", " p\n", " for n, p in self.model.named_parameters()\n", " if not any(nd in n for nd in self.no_decay) and p.requires_grad\n", " ],\n", " \"weight_decay\": self.hparams.optimizer_init[\"weight_decay\"],\n", " },\n", " {\n", " \"params\": [\n", " p\n", " for n, p in self.model.named_parameters()\n", " if any(nd in n for nd in self.no_decay) and p.requires_grad\n", " ],\n", " \"weight_decay\": 0.0,\n", " },\n", " ]\n", "\n", " def configure_optimizers(self):\n", " # the phase 0 parameters will have been set to require gradients during setup\n", " # you can initialize the optimizer with a simple requires.grad filter as is often done,\n", " # but in this case we pass a list of parameter groups to ensure weight_decay is\n", " # not applied to the bias parameter (for completeness, in this case it won't make much\n", " # performance difference)\n", " optimizer = AdamW(params=self._init_param_groups(), **self.hparams.optimizer_init)\n", " scheduler = {\n", " \"scheduler\": CosineAnnealingWarmRestarts(optimizer, **self.hparams.lr_scheduler_init),\n", " \"interval\": \"epoch\",\n", " }\n", " return [optimizer], [scheduler]"]}, {"cell_type": "markdown", "id": "43f3c183", "metadata": {"papermill": {"duration": 0.010529, "end_time": "2023-03-15T10:45:13.108991", "exception": false, "start_time": "2023-03-15T10:45:13.098462", "status": "completed"}, "tags": []}, "source": ["### Our Training Sessions\n", "\n", "We'll be comparing three different fine-tuning training configurations. Every configuration in this example depends\n", "upon a shared set of defaults, only differing in their respective fine-tuning schedules.\n", "\n", "| Experiment Tag | Training Scenario Description |\n", "|:-----------------:| ---------------------------------------------------------------------- |\n", "| ``fts_explicit`` | Training with a fine-tuning schedule explicitly provided by the user |\n", "| ``nofts_baseline``| A baseline fine-tuning training session (without scheduled fine-tuning) |\n", "| ``fts_implicit`` | Training with an implicitly generated fine-tuning schedule (the default) |\n", "\n", "Let's begin by configuring the ``fts_explicit`` scenario. We'll subsequently run the other two scenarios for\n", "comparison."]}, {"cell_type": "code", "execution_count": 6, "id": "b0760dcc", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:13.131530Z", "iopub.status.busy": "2023-03-15T10:45:13.131168Z", "iopub.status.idle": "2023-03-15T10:45:13.137808Z", "shell.execute_reply": "2023-03-15T10:45:13.136927Z"}, "papermill": {"duration": 0.01974, "end_time": "2023-03-15T10:45:13.139205", "exception": false, "start_time": "2023-03-15T10:45:13.119465", "status": "completed"}, "tags": []}, "outputs": [], "source": ["# Let's create a fine-tuning schedule for our model and run an explicitly scheduled fine-tuning training scenario with it\n", "# Please see the [FinetuningScheduler documentation](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) for a full description of the schedule format\n", "\n", "\n", "ft_schedule_yaml = \"\"\"\n", "0:\n", " params:\n", " - model.classifier.bias\n", " - model.classifier.weight\n", " - model.pooler.dense.bias\n", " - model.pooler.dense.weight\n", " - model.deberta.encoder.LayerNorm.bias\n", " - model.deberta.encoder.LayerNorm.weight\n", " - model.deberta.encoder.rel_embeddings.weight\n", " - model.deberta.encoder.layer.{0,11}.(output|attention|intermediate).*\n", "1:\n", " params:\n", " - model.deberta.embeddings.LayerNorm.bias\n", " - model.deberta.embeddings.LayerNorm.weight\n", "2:\n", " params:\n", " - model.deberta.embeddings.word_embeddings.weight\n", "\"\"\"\n", "ft_schedule_name = \"RteBoolqModule_ft_schedule_deberta_base.yaml\"\n", "# Let's write the schedule to a file so we can simulate loading an explicitly defined fine-tuning\n", "# schedule.\n", "with open(ft_schedule_name, \"w\") as f:\n", " f.write(ft_schedule_yaml)"]}, {"cell_type": "code", "execution_count": 7, "id": "4fe20839", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:13.168684Z", "iopub.status.busy": "2023-03-15T10:45:13.168339Z", "iopub.status.idle": "2023-03-15T10:45:15.388633Z", "shell.execute_reply": "2023-03-15T10:45:15.387243Z"}, "papermill": {"duration": 2.234534, "end_time": "2023-03-15T10:45:15.391358", "exception": false, "start_time": "2023-03-15T10:45:13.156824", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Global seed set to 42\n"]}, {"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "9ae7d01f948e4c9b9844735d5d60818e", "version_major": 2, "version_minor": 0}, "text/plain": ["Downloading (\u2026)okenizer_config.json: 0%| | 0.00/52.0 [00:00\n", "\n", "Though other optimizers can arguably yield some marginal advantage contingent on the context,\n", "the Adam optimizer (and the [AdamW version](https://pytorch.org/docs/stable/_modules/torch/optim/adamw.html#AdamW) which\n", "implements decoupled weight decay) remains robust to hyperparameter choices and is commonly used for fine-tuning\n", "foundation language models. See [(Sivaprasad et al., 2020)](#f2) and [(Mosbach, Andriushchenko & Klakow, 2020)](#f3) for theoretical and systematic empirical justifications of Adam and its use in fine-tuning\n", "large transformer-based language models. The values used here have some justification\n", "in the referenced literature but have been largely empirically determined and while a good\n", "starting point could be could be further tuned.\n", "\n", ""]}, {"cell_type": "code", "execution_count": 8, "id": "d47b9398", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:15.443196Z", "iopub.status.busy": "2023-03-15T10:45:15.442447Z", "iopub.status.idle": "2023-03-15T10:45:15.448651Z", "shell.execute_reply": "2023-03-15T10:45:15.447796Z"}, "papermill": {"duration": 0.020819, "end_time": "2023-03-15T10:45:15.450403", "exception": false, "start_time": "2023-03-15T10:45:15.429584", "status": "completed"}, "tags": []}, "outputs": [], "source": ["optimizer_init = {\"weight_decay\": 1e-05, \"eps\": 1e-07, \"lr\": 1e-05}"]}, {"cell_type": "markdown", "id": "adf9f7f1", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.010814, "end_time": "2023-03-15T10:45:15.474924", "exception": false, "start_time": "2023-03-15T10:45:15.464110", "status": "completed"}, "tags": []}, "source": ["### LR Scheduler Configuration\n", "\n", "\n", "\n", "The [CosineAnnealingWarmRestarts scheduler](https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.CosineAnnealingWarmRestarts.html?highlight=cosineannealingwarm#torch.optim.lr_scheduler.CosineAnnealingWarmRestarts) nicely fits with our iterative fine-tuning since it does not depend upon a global max_epoch\n", "value. The importance of initial warmup is reduced due to the innate warmup effect of Adam bias correction [[5]](#f3)\n", "and the gradual thawing we are performing. Note that commonly used LR schedulers that depend on providing\n", "max_iterations/epochs (e.g. the\n", "[CosineWarmupScheduler](https://github.com/Lightning-AI/tutorials/blob/0c325829101d5a6ebf32ed99bbf5b09badf04a59/course_UvA-DL/05-transformers-and-MH-attention/Transformers_MHAttention.py#L688)\n", "used in other pytorch-lightning tutorials) also work with FinetuningScheduler. Though the LR scheduler is theoretically\n", "justified [(Loshchilov & Hutter, 2016)](#f4), the particular values provided here are primarily empircally driven.\n", "\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) also supports LR scheduler\n", "reinitialization in both explicit and implicit finetuning schedule modes. See the [advanced usage documentation](https://finetuning-scheduler.readthedocs.io/en/stable/advanced/lr_scheduler_reinitialization.html) for explanations and demonstration of the extension's support for more complex requirements.\n", "
"]}, {"cell_type": "code", "execution_count": 9, "id": "d93aa0cf", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:15.498469Z", "iopub.status.busy": "2023-03-15T10:45:15.497918Z", "iopub.status.idle": "2023-03-15T10:45:15.501435Z", "shell.execute_reply": "2023-03-15T10:45:15.500842Z"}, "papermill": {"duration": 0.016518, "end_time": "2023-03-15T10:45:15.502746", "exception": false, "start_time": "2023-03-15T10:45:15.486228", "status": "completed"}, "tags": []}, "outputs": [], "source": ["lr_scheduler_init = {\"T_0\": 1, \"T_mult\": 2, \"eta_min\": 1e-07}"]}, {"cell_type": "code", "execution_count": 10, "id": "e9c7bfba", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:45:15.526666Z", "iopub.status.busy": "2023-03-15T10:45:15.525886Z", "iopub.status.idle": "2023-03-15T10:45:38.338960Z", "shell.execute_reply": "2023-03-15T10:45:38.337595Z"}, "papermill": {"duration": 22.826703, "end_time": "2023-03-15T10:45:38.340453", "exception": false, "start_time": "2023-03-15T10:45:15.513750", "status": "completed"}, "tags": []}, "outputs": [{"data": {"application/vnd.jupyter.widget-view+json": {"model_id": "1bc4ba08f03b44798a72b2dc7c37a40c", "version_major": 2, "version_minor": 0}, "text/plain": ["Downloading pytorch_model.bin: 0%| | 0.00/371M [00:00 None:\n", " trainer = pl.Trainer(\n", " enable_progress_bar=enable_progress_bar,\n", " max_epochs=100,\n", " precision=16,\n", " accelerator=\"auto\",\n", " devices=1 if is_cuda_available() else None,\n", " callbacks=callbacks,\n", " logger=logger,\n", " )\n", " trainer.fit(model, datamodule=dm)\n", "\n", "\n", "print(\n", " \"Note given the computation associated w/ the multiple phases of fine-tuning demonstrated, this notebook is best used with an accelerator\"\n", ")\n", "train()"]}, {"cell_type": "markdown", "id": "63dd27ce", "metadata": {"lines_to_next_cell": 2, "papermill": {"duration": 0.011953, "end_time": "2023-03-15T10:47:21.013707", "exception": false, "start_time": "2023-03-15T10:47:21.001754", "status": "completed"}, "tags": []}, "source": ["### Running the Baseline and Implicit Fine-Tuning Scenarios\n", "\n", "Let's now compare our ``nofts_baseline`` and ``fts_implicit`` scenarios with the ``fts_explicit`` one we just ran.\n", "\n", "We'll need to update our callbacks list, using the core PL ``EarlyStopping`` and ``ModelCheckpoint`` callbacks for the\n", "``nofts_baseline`` (which operate identically to their FTS analogs apart from the recursive training support).\n", "For both core PyTorch Lightning and user-registered callbacks, we can define our callbacks using a dictionary as we do\n", "with the LightningCLI. This allows us to avoid managing imports and support more complex configuration separated from\n", "code.\n", "\n", "Note that we'll be using identical callback configurations to the ``fts_explicit`` scenario. Keeping [max_depth](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=max_depth#finetuning_scheduler.fts.FinetuningScheduler.params.max_depth) for\n", "the implicit schedule will limit fine-tuning to just the last 4 parameters of the model, which is only a small fraction\n", "of the parameters you'd want to tune for maximum performance. Since the implicit schedule is quite computationally\n", "intensive and most useful for exploring model behavior, leaving [max_depth](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html?highlight=max_depth#finetuning_scheduler.fts.FinetuningScheduler.params.max_depth) 1 allows us to demo implicit mode\n", "behavior while keeping the computational cost and runtime of this notebook reasonable. To review how a full implicit\n", "mode run compares to the ``nofts_baseline`` and ``fts_explicit`` scenarios, please see the the following\n", "[tensorboard experiment summary](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/)."]}, {"cell_type": "code", "execution_count": 14, "id": "43950509", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:47:21.039108Z", "iopub.status.busy": "2023-03-15T10:47:21.038895Z", "iopub.status.idle": "2023-03-15T10:47:21.045537Z", "shell.execute_reply": "2023-03-15T10:47:21.044692Z"}, "papermill": {"duration": 0.021135, "end_time": "2023-03-15T10:47:21.046875", "exception": false, "start_time": "2023-03-15T10:47:21.025740", "status": "completed"}, "tags": []}, "outputs": [], "source": ["nofts_callbacks = [EarlyStopping(**earlystopping_kwargs), ModelCheckpoint(**checkpoint_kwargs)]\n", "fts_implicit_callbacks = [\n", " fts.FinetuningScheduler(**fts_kwargs),\n", " fts.FTSEarlyStopping(**earlystopping_kwargs),\n", " fts.FTSCheckpoint(**checkpoint_kwargs),\n", "]\n", "scenario_callbacks = {\"nofts_baseline\": nofts_callbacks, \"fts_implicit\": fts_implicit_callbacks}"]}, {"cell_type": "code", "execution_count": 15, "id": "20f142e0", "metadata": {"execution": {"iopub.execute_input": "2023-03-15T10:47:21.078597Z", "iopub.status.busy": "2023-03-15T10:47:21.078367Z", "iopub.status.idle": "2023-03-15T10:49:13.570131Z", "shell.execute_reply": "2023-03-15T10:49:13.568853Z"}, "papermill": {"duration": 112.507956, "end_time": "2023-03-15T10:49:13.573482", "exception": false, "start_time": "2023-03-15T10:47:21.065526", "status": "completed"}, "tags": []}, "outputs": [{"name": "stderr", "output_type": "stream", "text": ["Using 16bit None Automatic Mixed Precision (AMP)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True (cuda), used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Beginning training the 'nofts_baseline' scenario\n"]}, {"name": "stderr", "output_type": "stream", "text": ["You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Missing logger folder: lightning_logs/nofts_baseline\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"name": "stderr", "output_type": "stream", "text": ["\n", " | Name | Type | Params\n", "-------------------------------------------------------------\n", "0 | model | DebertaV2ForSequenceClassification | 184 M \n", "-------------------------------------------------------------\n", "184 M Trainable params\n", "0 Non-trainable params\n", "184 M Total params\n", "368.847 Total estimated model params size (MB)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Using 16bit None Automatic Mixed Precision (AMP)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["GPU available: True (cuda), used: True\n"]}, {"name": "stderr", "output_type": "stream", "text": ["TPU available: False, using: 0 TPU cores\n"]}, {"name": "stderr", "output_type": "stream", "text": ["IPU available: False, using: 0 IPUs\n"]}, {"name": "stderr", "output_type": "stream", "text": ["HPU available: False, using: 0 HPUs\n"]}, {"name": "stdout", "output_type": "stream", "text": ["Beginning training the 'fts_implicit' scenario\n"]}, {"name": "stderr", "output_type": "stream", "text": ["You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Missing logger folder: lightning_logs/fts_implicit\n"]}, {"name": "stderr", "output_type": "stream", "text": ["fine-tuning schedule dumped to lightning_logs/fts_implicit/version_0/RteBoolqModule_ft_schedule.yaml.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Generated default fine-tuning schedule 'lightning_logs/fts_implicit/version_0/RteBoolqModule_ft_schedule.yaml' for iterative fine-tuning\n"]}, {"name": "stderr", "output_type": "stream", "text": ["LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]\n"]}, {"name": "stderr", "output_type": "stream", "text": ["\n", " | Name | Type | Params\n", "-------------------------------------------------------------\n", "0 | model | DebertaV2ForSequenceClassification | 184 M \n", "-------------------------------------------------------------\n", "1.5 K Trainable params\n", "184 M Non-trainable params\n", "184 M Total params\n", "368.847 Total estimated model params size (MB)\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Restoring states from the checkpoint path at lightning_logs/fts_implicit/version_0/checkpoints/epoch=0-step=156.ckpt\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Restored all states from the checkpoint file at lightning_logs/fts_implicit/version_0/checkpoints/epoch=0-step=156.ckpt\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Multi-phase fine-tuned training continuing at level 1.\n"]}, {"name": "stderr", "output_type": "stream", "text": ["Given the current configuration of `max_depth` (1), this training session will now end when the early stopping conditions are met or `max_epochs` (100) is reached.\n"]}], "source": ["for scenario_name, scenario_callbacks in scenario_callbacks.items():\n", " model = RteBoolqModule(**lightning_module_kwargs, experiment_tag=scenario_name)\n", " logger = TensorBoardLogger(\"lightning_logs\", name=scenario_name)\n", " callbacks = scenario_callbacks\n", " print(f\"Beginning training the '{scenario_name}' scenario\")\n", " train()"]}, {"cell_type": "markdown", "id": "bbb8962f", "metadata": {"lines_to_next_cell": 0, "papermill": {"duration": 0.013533, "end_time": "2023-03-15T10:49:13.604866", "exception": false, "start_time": "2023-03-15T10:49:13.591333", "status": "completed"}, "tags": []}, "source": ["### Reviewing the Training Results\n", "\n", "See the [tensorboard experiment summaries](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/) to get a sense\n", "of the relative computational and performance tradeoffs associated with these [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) configurations.\n", "The summary compares a full ``fts_implicit`` execution to ``fts_explicit`` and ``nofts_baseline`` scenarios using DDP\n", "training with 2 GPUs. The full logs/schedules and detailed system configuration used for all three scenarios are available\n", "[here](https://drive.google.com/file/d/1LrUcisRLHeJgh_BDOOD_GUBPp5iHAkoR/view?usp=sharing) and the checkpoints\n", "produced in the scenarios [here](https://drive.google.com/file/d/1t7myBgcqcZ9ax_IT9QVk-vFH_l_o5UXB/view?usp=sharing)\n", "(caution, ~3.5GB).\n", "\n", "[{height=\"315px\" width=\"492px\"}](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/#scalars&_smoothingWeight=0&runSelectionState=eyJmdHNfZXhwbGljaXQiOnRydWUsIm5vZnRzX2Jhc2VsaW5lIjpmYWxzZSwiZnRzX2ltcGxpY2l0IjpmYWxzZX0%3D)\n", "[{height=\"316px\" width=\"505px\"}](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/#scalars&_smoothingWeight=0&runSelectionState=eyJmdHNfZXhwbGljaXQiOmZhbHNlLCJub2Z0c19iYXNlbGluZSI6dHJ1ZSwiZnRzX2ltcGxpY2l0IjpmYWxzZX0%3D)\n", "\n", "Note that the results above may vary to a small degree from the tensorboard summaries generated by this notebook\n", "which used DP, 1 GPU and likely when you're running this, different versions of certain software components (e.g. pytorch, transformers).\n", "\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) expands the space of possible fine-tuning schedules and the composition of more sophisticated schedules can\n", "yield marginal fine-tuning performance gains. That stated, it should be emphasized the primary utility of [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) is to grant\n", "greater fine-tuning flexibility for model exploration in research. For example, glancing at DeBERTa-v3's implicit training\n", "run, a critical tuning transition point is immediately apparent:\n", "\n", "[{height=\"272px\" width=\"494px\"}](https://tensorboard.dev/experiment/n7U8XhrzRbmvVzC4SQSpWw/#scalars&_smoothingWeight=0&runSelectionState=eyJmdHNfZXhwbGljaXQiOmZhbHNlLCJub2Z0c19iYXNlbGluZSI6ZmFsc2UsImZ0c19pbXBsaWNpdCI6dHJ1ZX0%3D)\n", "\n", "Our `val_loss` begins a precipitous decline at step 3119 which corresponds to phase 17 in the schedule. Referring to our\n", "schedule, in phase 17 we're beginning tuning the attention parameters of our 10th encoder layer (of 11). Interesting!\n", "Though beyond the scope of this tutorial, it might be worth investigating these dynamics further and\n", "[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) allows one to do just that quite easily.\n"]}, {"cell_type": "markdown", "id": "2d1bdf3e", "metadata": {"lines_to_next_cell": 0, "papermill": {"duration": 0.013185, "end_time": "2023-03-15T10:49:13.631655", "exception": false, "start_time": "2023-03-15T10:49:13.618470", "status": "completed"}, "tags": []}, "source": ["\n", "Note that though this example is intended to capture a common usage scenario, substantial variation is expected\n", "among use cases and models.\n", "In summary, [FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) provides increased fine-tuning flexibility that can be useful in a variety of\n", "contexts from exploring model tuning behavior to maximizing performance."]}, {"cell_type": "markdown", "id": "335cb6aa", "metadata": {"papermill": {"duration": 0.01323, "end_time": "2023-03-15T10:49:13.658205", "exception": false, "start_time": "2023-03-15T10:49:13.644975", "status": "completed"}, "tags": []}, "source": ["## Footnotes\n", "\n", "\n", "- \n", "\n", "[Howard, J., & Ruder, S. (2018)](https://arxiv.org/pdf/1801.06146.pdf). Fine-tuned Language\n", " Models for Text Classification. ArXiv, abs/1801.06146. [\u21a9](#a1)\n", "\n", "
\n", "- \n", "\n", "[Chronopoulou, A., Baziotis, C., & Potamianos, A. (2019)](https://arxiv.org/pdf/1902.10547.pdf).\n", " An embarrassingly simple approach for transfer learning from pretrained language models. arXiv\n", " preprint arXiv:1902.10547. [\u21a9](#a1)\n", "\n", "
\n", "- \n", "\n", "[Peters, M. E., Ruder, S., & Smith, N. A. (2019)](https://arxiv.org/pdf/1903.05987.pdf). To tune or not to\n", " tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987. [\u21a9](#a1)\n", "\n", "
\n", "- \n", "\n", "[Sivaprasad, P. T., Mai, F., Vogels, T., Jaggi, M., & Fleuret, F. (2020)](https://arxiv.org/pdf/1910.11758.pdf).\n", " Optimizer benchmarking needs to account for hyperparameter tuning. In International Conference on Machine Learning\n", "(pp. 9036-9045). PMLR. [\u21a9](#a2)\n", "\n", "
\n", "- \n", "\n", "[Mosbach, M., Andriushchenko, M., & Klakow, D. (2020)](https://arxiv.org/pdf/2006.04884.pdf). On the stability of\n", "fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884. [\u21a9](#a2)\n", "\n", "
\n", "- \n", "\n", "[Loshchilov, I., & Hutter, F. (2016)](https://arxiv.org/pdf/1608.03983.pdf). Sgdr: Stochastic gradient descent with\n", "warm restarts. arXiv preprint arXiv:1608.03983. [\u21a9](#a3)\n", "\n", "
\n", "\n", "
"]}, {"cell_type": "markdown", "id": "3a15461a", "metadata": {"papermill": {"duration": 0.021571, "end_time": "2023-03-15T10:49:13.693130", "exception": false, "start_time": "2023-03-15T10:49:13.671559", "status": "completed"}, "tags": []}, "source": []}, {"cell_type": "markdown", "id": "0f3c2b81", "metadata": {"papermill": {"duration": 0.014034, "end_time": "2023-03-15T10:49:13.720830", "exception": false, "start_time": "2023-03-15T10:49:13.706796", "status": "completed"}, "tags": []}, "source": ["## Congratulations - Time to Join the Community!\n", "\n", "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the Lightning\n", "movement, you can do so in the following ways!\n", "\n", "### Star [Lightning](https://github.com/Lightning-AI/lightning) on GitHub\n", "The easiest way to help our community is just by starring the GitHub repos! This helps raise awareness of the cool\n", "tools we're building.\n", "\n", "### Join our [Slack](https://www.pytorchlightning.ai/community)!\n", "The best way to keep up to date on the latest advancements is to join our community! Make sure to introduce yourself\n", "and share your interests in `#general` channel\n", "\n", "\n", "### Contributions !\n", "The best way to contribute to our community is to become a code contributor! At any time you can go to\n", "[Lightning](https://github.com/Lightning-AI/lightning) or [Bolt](https://github.com/Lightning-AI/lightning-bolts)\n", "GitHub Issues page and filter for \"good first issue\".\n", "\n", "* [Lightning good first issue](https://github.com/Lightning-AI/lightning/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* [Bolt good first issue](https://github.com/Lightning-AI/lightning-bolts/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)\n", "* You can also contribute your own notebooks with useful examples !\n", "\n", "### Great thanks from the entire Pytorch Lightning Team for your interest !\n", "\n", "[{height=\"60px\" width=\"240px\"}](https://pytorchlightning.ai)"]}, {"cell_type": "raw", "metadata": {"raw_mimetype": "text/restructuredtext"}, "source": [".. customcarditem::\n", " :header: Fine-Tuning Scheduler\n", " :card_description: This notebook introduces the [Fine-Tuning Scheduler](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) extension and demonstrates the use of it to fine-tune a...\n", " :tags: Fine-Tuning,GPU/TPU,Lightning-Examples"]}], "metadata": {"jupytext": {"cell_metadata_filter": "colab,id,colab_type,-all", "formats": "ipynb,py:percent", "main_language": "python"}, "language_info": {"codemirror_mode": {"name": "ipython", "version": 3}, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.16"}, "papermill": {"default_parameters": {}, "duration": 250.277442, "end_time": "2023-03-15T10:49:15.158131", "environment_variables": {}, "exception": null, "input_path": "lightning_examples/finetuning-scheduler/finetuning-scheduler.ipynb", "output_path": ".notebooks/lightning_examples/finetuning-scheduler.ipynb", "parameters": {}, "start_time": "2023-03-15T10:45:04.880689", "version": "2.4.0"}, "widgets": {"application/vnd.jupyter.widget-state+json": {"state": {"060543ec56fd4b828d700285151d5a4a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "068c14cadba84ca8a4cec3222cd46545": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "07c62a88260346c5b537899846c172fc": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_21a3c1ce9acb4398bc7fc3eeb6c758b7", "placeholder": "\u200b", "style": "IPY_MODEL_3b6d938e2f2b4eaba93d75812e92e8bc", "tabbable": null, "tooltip": null, "value": "Downloading builder script: 100%"}}, "0f10fb70358d46518c7e1ed5db48b80f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "130f59dbef914688b51284902dd18a51": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "19973fa79757454ca943f0110f6c7f3b": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "1bc4ba08f03b44798a72b2dc7c37a40c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_ae83574e6e824515ba929cb1680cbf8a", "IPY_MODEL_43150130155f41bf883a62d1a0567c71", "IPY_MODEL_b6dac7810883443b993f087601ddca97"], "layout": "IPY_MODEL_626f46e3454d4dfdb6d09c4326f679ae", "tabbable": null, "tooltip": null}}, "21a3c1ce9acb4398bc7fc3eeb6c758b7": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "257db3e497c94c13a9af122aab526caf": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_bbb079d553724977b64e5cd1ba5c7397", "placeholder": "\u200b", "style": "IPY_MODEL_fa63d860c2e5478ca2a384444cca0373", "tabbable": null, "tooltip": null, "value": " 9.64k/9.64k [00:00<00:00, 616kB/s]"}}, "290b572c47fb493aba1be3380ec36ccf": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "2bf5630438e546079961fa3e08c4d7b7": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "3b6d938e2f2b4eaba93d75812e92e8bc": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "43150130155f41bf883a62d1a0567c71": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_4650e5527d2048a6b38af9450c6f83e4", "max": 371146213.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_93e121ae022b4f68ba6bd3db35eff33b", "tabbable": null, "tooltip": null, "value": 371146213.0}}, "44468cc6fb7b41c9b44a5c6ea65b91c5": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "4650e5527d2048a6b38af9450c6f83e4": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "51c8a4698f524400a1ede06914722f99": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "547f957f43644a90b273d89d992a9828": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_757ee9230d514d25b6674164b50e330c", "placeholder": "\u200b", "style": "IPY_MODEL_a37e9fb6019441e3b7e13ddbcf7b25e8", "tabbable": null, "tooltip": null, "value": " 579/579 [00:00<00:00, 35.4kB/s]"}}, "5f1f6fa5e0684105b89ede5dcd23a760": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_07c62a88260346c5b537899846c172fc", "IPY_MODEL_d2cae59307bc43c9843fb879e3ab815c", "IPY_MODEL_257db3e497c94c13a9af122aab526caf"], "layout": "IPY_MODEL_d5730589980e4fe0971728f0da84cbeb", "tabbable": null, "tooltip": null}}, "626f46e3454d4dfdb6d09c4326f679ae": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "70330594da8e4884be9d3e5ac09b578f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "74f94d8672c14d999d88129f25e5469a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "757ee9230d514d25b6674164b50e330c": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "79f7f0736e2a45d9a027c3317e4c6217": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "7cb3651d9b3043788c0f7b272a4df895": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "7e1ed84b677a44b8a35aa5e0e7219eae": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "7f858d428f8048f1836c5078a40b2e7e": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "87d48442d901407dae211d4a4ce145df": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_e1143c636b0347f8a7f25b6352f9db1e", "max": 579.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_b09f478ff3cb4e17a5bdd3bffea868a5", "tabbable": null, "tooltip": null, "value": 579.0}}, "8841dbf887304cbcb9c72b51d166812f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_130f59dbef914688b51284902dd18a51", "placeholder": "\u200b", "style": "IPY_MODEL_7cb3651d9b3043788c0f7b272a4df895", "tabbable": null, "tooltip": null, "value": " 52.0/52.0 [00:00<00:00, 2.23kB/s]"}}, "8e971c21e1004125af27a00745357f65": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "93e121ae022b4f68ba6bd3db35eff33b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "9789f86034484a529c12cbd9ca3c7b7d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "9ad11ff696d0477aae759cd129dcd41d": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "9ae7d01f948e4c9b9844735d5d60818e": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_f485fe7fb6ba4724b74d64adbd1ad6e3", "IPY_MODEL_c7cc4c9ab42e46a893acb8aac9a72f8d", "IPY_MODEL_8841dbf887304cbcb9c72b51d166812f"], "layout": "IPY_MODEL_aa12cc0c28bf40acaf3dda7a76bf9469", "tabbable": null, "tooltip": null}}, "9de327ed63c14575bd99b7c5b83e38ee": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "a2fc67a090ca4635bf05e8be2a0eadaa": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_cc4677395bbf4407be654e21d7e997ba", "placeholder": "\u200b", "style": "IPY_MODEL_79f7f0736e2a45d9a027c3317e4c6217", "tabbable": null, "tooltip": null, "value": "Downloading spm.model: 100%"}}, "a37e9fb6019441e3b7e13ddbcf7b25e8": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}, "a87828b815b64baa92446753ddeec775": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "aa12cc0c28bf40acaf3dda7a76bf9469": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "ae83574e6e824515ba929cb1680cbf8a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_290b572c47fb493aba1be3380ec36ccf", "placeholder": "\u200b", "style": "IPY_MODEL_70330594da8e4884be9d3e5ac09b578f", "tabbable": null, "tooltip": null, "value": "Downloading pytorch_model.bin: 100%"}}, "aec4167583fa44869111da8ea73f63ba": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_aee2c98769054cfab83f8b249dfaf068", "placeholder": "\u200b", "style": "IPY_MODEL_9de327ed63c14575bd99b7c5b83e38ee", "tabbable": null, "tooltip": null, "value": " 3.72k/3.72k [00:00<00:00, 277kB/s]"}}, "aee2c98769054cfab83f8b249dfaf068": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "afe56dc5a4da44e5915d6265b848565b": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_dd61523841c6413fb161614a9d7d3b7d", "IPY_MODEL_bf83aaeda7cc42298f559e815c30d3ec", "IPY_MODEL_aec4167583fa44869111da8ea73f63ba"], "layout": "IPY_MODEL_b3dbe80284834656a3a42c7b0274da79", "tabbable": null, "tooltip": null}}, "b09f478ff3cb4e17a5bdd3bffea868a5": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "b3dbe80284834656a3a42c7b0274da79": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "b6dac7810883443b993f087601ddca97": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_19973fa79757454ca943f0110f6c7f3b", "placeholder": "\u200b", "style": "IPY_MODEL_74f94d8672c14d999d88129f25e5469a", "tabbable": null, "tooltip": null, "value": " 371M/371M [00:20<00:00, 10.8MB/s]"}}, "bbb079d553724977b64e5cd1ba5c7397": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "bf83aaeda7cc42298f559e815c30d3ec": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_e15d6603a2b544aa942ec526bb4b3901", "max": 3724.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_c983ddb0c6314cb49af479232c3c862a", "tabbable": null, "tooltip": null, "value": 3724.0}}, "c28337719d504620835a5e95062c1c19": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "c7cc4c9ab42e46a893acb8aac9a72f8d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_eb3fa43091b947379ebc58bbc5cf9e42", "max": 52.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_0f10fb70358d46518c7e1ed5db48b80f", "tabbable": null, "tooltip": null, "value": 52.0}}, "c983ddb0c6314cb49af479232c3c862a": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "ProgressStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "ProgressStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "bar_color": null, "description_width": ""}}, "cc4677395bbf4407be654e21d7e997ba": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "cd06c0e821ad43a983766851a735ca9f": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_a87828b815b64baa92446753ddeec775", "placeholder": "\u200b", "style": "IPY_MODEL_7e1ed84b677a44b8a35aa5e0e7219eae", "tabbable": null, "tooltip": null, "value": " 2.46M/2.46M [00:00<00:00, 16.8MB/s]"}}, "d0111fe65f12406d85e4891270c270f5": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "d245e2e996c042179e9cccf59ba20d83": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_d0111fe65f12406d85e4891270c270f5", "placeholder": "\u200b", "style": "IPY_MODEL_2bf5630438e546079961fa3e08c4d7b7", "tabbable": null, "tooltip": null, "value": "Downloading (\u2026)lve/main/config.json: 100%"}}, "d2cae59307bc43c9843fb879e3ab815c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_7f858d428f8048f1836c5078a40b2e7e", "max": 9644.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_51c8a4698f524400a1ede06914722f99", "tabbable": null, "tooltip": null, "value": 9644.0}}, "d5730589980e4fe0971728f0da84cbeb": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "d72fa1c3f657445abf3144ebafbe8ec7": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "dcf37b9923bf47379711bfb7da94d70d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_a2fc67a090ca4635bf05e8be2a0eadaa", "IPY_MODEL_e5eb88c2443247ddae5982dae7a060f4", "IPY_MODEL_cd06c0e821ad43a983766851a735ca9f"], "layout": "IPY_MODEL_44468cc6fb7b41c9b44a5c6ea65b91c5", "tabbable": null, "tooltip": null}}, "dd61523841c6413fb161614a9d7d3b7d": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_068c14cadba84ca8a4cec3222cd46545", "placeholder": "\u200b", "style": "IPY_MODEL_060543ec56fd4b828d700285151d5a4a", "tabbable": null, "tooltip": null, "value": "Downloading extra modules: 100%"}}, "e1143c636b0347f8a7f25b6352f9db1e": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "e15d6603a2b544aa942ec526bb4b3901": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "e5eb88c2443247ddae5982dae7a060f4": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "FloatProgressModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "FloatProgressModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "ProgressView", "bar_style": "success", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_c28337719d504620835a5e95062c1c19", "max": 2464616.0, "min": 0.0, "orientation": "horizontal", "style": "IPY_MODEL_8e971c21e1004125af27a00745357f65", "tabbable": null, "tooltip": null, "value": 2464616.0}}, "eb3fa43091b947379ebc58bbc5cf9e42": {"model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": {"_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null}}, "f3e1d2ba29514070b8fdd697af0ced4c": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HBoxModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HBoxModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HBoxView", "box_style": "", "children": ["IPY_MODEL_d245e2e996c042179e9cccf59ba20d83", "IPY_MODEL_87d48442d901407dae211d4a4ce145df", "IPY_MODEL_547f957f43644a90b273d89d992a9828"], "layout": "IPY_MODEL_d72fa1c3f657445abf3144ebafbe8ec7", "tabbable": null, "tooltip": null}}, "f485fe7fb6ba4724b74d64adbd1ad6e3": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLModel", "state": {"_dom_classes": [], "_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLModel", "_view_count": null, "_view_module": "@jupyter-widgets/controls", "_view_module_version": "2.0.0", "_view_name": "HTMLView", "description": "", "description_allow_html": false, "layout": "IPY_MODEL_9ad11ff696d0477aae759cd129dcd41d", "placeholder": "\u200b", "style": "IPY_MODEL_9789f86034484a529c12cbd9ca3c7b7d", "tabbable": null, "tooltip": null, "value": "Downloading (\u2026)okenizer_config.json: 100%"}}, "fa63d860c2e5478ca2a384444cca0373": {"model_module": "@jupyter-widgets/controls", "model_module_version": "2.0.0", "model_name": "HTMLStyleModel", "state": {"_model_module": "@jupyter-widgets/controls", "_model_module_version": "2.0.0", "_model_name": "HTMLStyleModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "StyleView", "background": null, "description_width": "", "font_size": null, "text_color": null}}}, "version_major": 2, "version_minor": 0}}}, "nbformat": 4, "nbformat_minor": 5}