Models Agents Infrastructure Pricing Docs Blog Security

Models Agents Infrastructure Pricing Docs Blog Security

PyTorch Lightning

Platform

Lightning AIDevelop, train and deploy on Lightning AI Python SDKProgrammatic access to the Lightning AI platform CLIAccess the Lightning AI platform from the command line

Open Source Libraries

PyTorch LightningFinetune and pretrain AI models. Focus on science, not engineering.Lightning FabricScale foundation models to 1000s of GPUs with expert-level control TorchMetrics90+ PyTorch metrics, optimized for distributed training ThunderCompiler for PyTorch: Make PyTorch models faster!LitLoggerExperiment tracking built into Lightning AI LitDataStream and process datasets at scale LitServeEasily serve AI models — Lightning fast LitAILLM router and minimal agent framework in one LitGPT20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

Stable (2.6.6)

Home

Lightning in 15 minutes Install

2.0 Upgrade Guide

2.0.x 1.9.x Upgrade from 1.8 to the 2.0 Upgrade from 1.7 to the 2.0 Upgrade from 1.6 to the 2.0 Upgrade from 1.5 to the 2.0 Upgrade from 1.4 to the 2.0

Level Up

Level 2: Add a validation and test set Level 5: Debug, visualize and find performance bottlenecks Level 6: Predict with your model

Intermediate skills

Level 7: Hardware acceleration Level 8: Modularize your projects Level 9: Understand your model Level 10: Explore SOTA scaling techniques Level 11: Deploy your models Level 12: Optimize training speed Level 13: Run on a multi-node cluster

Advanced skills

Level 14: Customize configs to run in production Level 15: Customize the trainer Level 16: Own the training loop Level 17: Enable advanced checkpointing Level 18: Master TPUS Level 19: Train models with billions of parameters

Level 21: Extend the Lightning CLI Level 22: Integrate a custom cluster Level 24: Add a new accelerator or Strategy

Core API

LightningModule Trainer

Optional API

Accelerator CPUAccelerator CUDAAccelerator XLAAccelerator BackboneFinetuning BaseFinetuning BasePredictionWriter BatchSizeFinder Callback DeviceStatsMonitor EarlyStopping GradientAccumulationScheduler LambdaCallback LearningRateFinder LearningRateMonitor ModelCheckpoint ModelPruning ModelSummary OnExceptionCheckpoint ProgressBar RichModelSummary RichProgressBar SpikeDetection StochasticWeightAveraging ThroughputMonitor Timer TQDMProgressBar WeightAveraging LightningArgumentParser LightningCLI SaveConfigCallback CheckpointHooks DataHooks ModelHooks LightningDataModule LightningModule HyperparametersMixin LightningOptimizer comet csv_logs litlogger logger mlflow tensorboard wandb ClusterEnvironment KubeflowEnvironment LightningEnvironment LSFEnvironment MPIEnvironment SLURMEnvironment TorchElasticEnvironment XLAEnvironment AsyncCheckpointIO CheckpointIO TorchCheckpointIO XLACheckpointIO LayerSync BitsandbytesPrecision DeepSpeedPrecision DoublePrecision FSDPPrecision HalfPrecision MixedPrecision Precision TransformerEnginePrecision XLAPrecision TorchSyncBatchNorm AdvancedProfiler PassThroughProfiler Profiler PyTorchProfiler SimpleProfiler XLAProfiler DDPStrategy DeepSpeedStrategy FSDPStrategy ModelParallelStrategy ParallelStrategy SingleDeviceStrategy SingleDeviceXLAStrategy Strategy XLAStrategy Trainer Tuner combined_loader data deepspeed memory parsing seed

More

Examples Community Glossary How to

Reference

Hardware agnostic training (preparation)GPU GPU training (Advanced)GPU training (Basic)Expert GPU training (FAQ)GPU training (Intermediate)Train on MPS MPS training (basic)TPU TPU training (Advanced)TPU training (Basic)TPU training (FAQ)TPU training (Intermediate)

Compile DDP Optimizations Finetuning Efficient initialization

DeepSpeed FSDP Tensor Parallel 2D Parallelism

Post-training Quantization Pretrained models Pruning and Quantization Speed Up Model Training Strategy registry Weight averaging Transfer learning Warnings

Lightning CLI Configure hyperparameters from the CLI (Advanced)Configure hyperparameters from the CLI (Advanced)Configure hyperparameters from the CLI (Advanced)Configure hyperparameters from the CLI (Expert)Frequently asked questions for LightningCLI Configure hyperparameters from the CLI (Intermediate)Configure hyperparameters from the CLI (Intermediate)

Cluster SLURM Run on an on-prem cluster (expert)Run on an on-prem cluster (intermediate)TorchRun (TorchElastic)Run single or multi-node on Lightning Studios

Checkpointing Cloud checkpoint Saving and loading checkpoints (basic)Distributed Checkpoints Customize checkpointing behavior (intermediate)Upgrading checkpoints (intermediate)Console Logging Early stopping Add validation and test datasets Validate and test a model (basic)Validate and test a model (intermediate)Hooks Configure hyperparameters from the CLI Interactive Notebooks (Jupyter, Colab, Kaggle)Optimization Half precision N-Bit Precision (Basic)N-Bit Precision (Expert)N-Bit Precision (Intermediate)Progress bar Remote filesystem and FSSPEC Truncated Backpropagation Through Time (TBPTT)

Common Workflows

Lightning Governance

Accessing DataLoaders Using 3rd Party Data Iterables Complex data uses LightningDataModule Arbitrary iterable support

Debugging Debug your model (advanced)Debug your model (basic)Debug your model (intermediate)

Deploy models into production Production Deploy models into production (advanced)Predict Inference

Accelerators Callback Save Callback state Save DataModule state Logging Plugins Strategy

CometLogger CSVLogger LitLogger MLFlowLogger TensorBoardLogger WandbLogger

Strategy integrations

Level 3: Visualize training progress

Model Own your loop (advanced)Supercharge training (intermediate)Manual Optimization Use a pure PyTorch training loop Train a model (basic)

PyTorch Lightning Tutorials Past PyTorch Lightning versions

Organize existing PyTorch into Lightning Style guide

Profiler Find bottlenecks in your code (advanced)Find bottlenecks in your code (basic)Find bottlenecks in your code (expert)Find bottlenecks in your code (intermediate)

Versioning Policy

Experiment manager (Logger)Log Track and Visualize Experiments (advanced)Track and Visualize Experiments (basic)Track and Visualize Experiments (expert)Manage experiments

Docs/Stable (2.6.6)/Level Up/Advanced skills/Level 19: Train models with billions of parameters

Level 19: Train models with billions of parameters

Scale to billions of parameters with multiple distributed strategies.

Scale with distributed strategies — Learn about different distributed strategies to reach bigger model parameter sizes.
Train models with billions of parameters — Scale to billions of params on GPUs with FSDP, TP or Deepspeed.

Was this page helpful?

Thanks for the feedback.

What went wrong?

Inaccurate or out of dateHard to understandMissing informationHard to find things / poorly organizedCode sample didn't workSomething is broken (link, image, layout)

Share with AI

Open in ChatGPT Open in Claude

Was this helpful?

Thanks for the feedback.

What went wrong?

Inaccurate or out of dateHard to understandMissing informationHard to find things / poorly organizedCode sample didn't workSomething is broken (link, image, layout)

↑↓ Navigate↵ Open⌘↵ New tabesc Close