How-tos

Basics

• Convert to Fabric in 5 minutes — Learn how to add Fabric to your PyTorch code

• Scale your model with Accelerators — Take advantage of your hardware with a switch of a flag

• Structure your Fabric code — Best practices for setting up your training script with Fabric

• Launch distributed training — Launch a Python script on multiple devices and machines

• Launch Fabric in a notebook — Launch on multiple devices from within a Jupyter notebook

• Improve performance with Mixed-Precision training — Save memory and speed up training using mixed precision

Build your own Trainer

• Organize your model code with LightningModule — Organize your code in a LightningModule and use it with Fabric

• Encapsulate code into Callbacks — Make use of the Callback system in Fabric

• Track and visualize experiments — Learn how Fabric helps you remove boilerplate code for tracking metrics with a logger

• Save and load model progress — Efficient saving and loading of model weights, training state, hyperparameters and more.

• Build your own Trainer — Take our Fabric Trainer template and customize it for your needs

Advanced Topics

• Use efficient gradient accumulation — Learn how to perform efficient gradient accumulation in distributed settings

• Distribute communication — Learn all about communication primitives for distributed operation. Gather, reduce, broadcast, etc.

• Use multiple models and optimizers — See how flexible Fabric is to work with multiple models and optimizers!

• Speed up models by compiling them — Use torch.compile to speed up models on modern hardware

• Train models with billions of parameters — Train the largest models with FSDP/TP across multiple GPUs and machines

• Initialize models efficiently — Reduce the time and peak memory usage for model initialization

• Save and load very large models — Save and load very large models efficiently with distributed checkpoints