Get Started
Blog
Docs
PyTorch Lightning Lightning TorchMetrics Lightning Flash Lightning Bolts
Previous Versions
GitHub
Lightning AI

Table of Contents

2.0.0

Get Started

Lightning in 15 minutes
Installation
Guide how to upgrade to the 2.0 version

Level Up

Basic skills
Intermediate skills
Advanced skills
Expert skills

Core API

LightningModule
Trainer

API Reference

accelerators
callbacks
cli
core
loggers
profiler
trainer
strategies
tuner
utilities

Common Workflows

Avoid overfitting
Build a Model
Configure hyperparameters from the CLI
Customize the progress bar
Deploy models into production
Effective Training Techniques
Find bottlenecks in your code
Manage experiments
Organize existing PyTorch into Lightning
Run on an on-prem cluster
Save and load model progress
Save memory with half-precision
Train 1 trillion+ parameter models
Train on single or multiple GPUs
Train on single or multiple HPUs
Train on single or multiple IPUs
Train on single or multiple TPUs
Train on MPS
Use a pretrained model
Complex data uses
Use a pure PyTorch training loop

Glossary

Accelerators
Callback
Checkpointing
Cluster
Cloud checkpoint
Console Logging
Debugging
Early stopping
Experiment manager (Logger)
Finetuning
Flash
GPU
Half precision
HPU
Inference
IPU
Lightning CLI
LightningDataModule
LightningModule
Log
TPU
Metrics
Model
Model Parallel
Plugins
Progress bar
Production
Predict
Pretrained models
Profiler
Pruning and Quantization
Remote filesystem and FSSPEC
Strategy
Strategy registry
Style guide
SWA
SLURM
Transfer learning
Trainer
Torch distributed

Hands-on Examples

Tutorial 1: Introduction to PyTorch
Tutorial 2: Activation Functions
Tutorial 3: Initialization and Optimization
Tutorial 4: Inception, ResNet and DenseNet
Tutorial 5: Transformers and Multi-Head Attention
Tutorial 6: Basics of Graph Neural Networks
Tutorial 7: Deep Energy-Based Generative Models
Tutorial 8: Deep Autoencoders
Tutorial 9: Normalizing Flows for Image Modeling
Tutorial 10: Autoregressive Image Modeling
Tutorial 11: Vision Transformers
Tutorial 12: Meta-Learning - Learning to Learn
Tutorial 13: Self-Supervised Contrastive Learning with SimCLR
GPU and batched data augmentation with Kornia and PyTorch-Lightning
Barlow Twins Tutorial
PyTorch Lightning Basic GAN Tutorial
PyTorch Lightning CIFAR10 ~94% Baseline Tutorial
PyTorch Lightning DataModules
Fine-Tuning Scheduler
Introduction to PyTorch Lightning
TPU training with PyTorch Lightning
How to train a Deep Q Network
Finetune Transformers Models with PyTorch Lightning
Multi-agent Reinforcement Learning With WarpDrive

Community

Contributor Covenant Code of Conduct
Contributing
How to Become a Core Contributor
Lightning Governance
Versioning Policy
Past PyTorch Lightning versions
Changelog

Docs >
accelerators >
distributed

Shortcuts

distributed¶

Utilities that can be used with distributed training.

Built with Sphinx using a theme provided by Read the Docs.

distributed

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. Read PyTorch Lightning's Privacy Policy.

Get Started
Blog
Ecosystem

PyTorch Lightning
Lightning Fabric
TorchMetrics
Fabric

Github
Lightning.ai