• Docs >
  • Lightning Transformers

Lightning Transformers

Lightning Transformers offers a flexible interface for training and fine-tuning SOTA Transformer models using the PyTorch Lightning Trainer.

pip install lightning-transformers

In Lightning Transformers, we offer the following benefits:

  • Powered by PyTorch Lightning - Accelerators, custom Callbacks, Loggers, and high performance scaling with minimal changes.

  • Backed by HuggingFace Transformers models and datasets, spanning multiple modalities and tasks within NLP/Audio and Vision.

  • Task Abstraction for Rapid Research & Experimentation - Build your own custom transformer tasks across all modalities with little friction.

  • Powerful config composition backed by Hydra - simply swap out models, optimizers, schedulers task, and many more configurations without touching the code.

  • Seamless Memory and Speed Optimizations - Out-of-the-box training optimizations such as DeepSpeed ZeRO or FairScale Sharded Training with no code changes.

Using Lightning-Transformers

Lightning Transformers has a collection of tasks for common NLP problems such as language_modeling, translation and more. To use, simply:

  1. Pick a task to train (passed to train.py as task=)

  2. Pick a dataset (passed to train.py as dataset=)

  3. Customize the backbone, optimizer, or any component within the config

  4. Add any Lightning supported parameters and optimizations

python train.py \
    task=<TASK> \
    backbone.pretrained_model_name_or_path=<BACKBONE> # Optionally change the HF backbone
    optimizer=<OPTIMIZER> # Optionally specify optimizer (Default AdamW)
    trainer.<ANY_TRAINER_FLAGS> # Optionally specify Lightning trainer arguments

To learn more about Lightning Transformers, please refer to the Lightning Transformers documentation.