Advanced skills

Use efficient gradient accumulation — Learn how to perform efficient gradient accumulation in distributed settings

Distribute communication — Learn all about communication primitives for distributed operation. Gather, reduce, broadcast, etc.

Use multiple models and optimizers — See how flexible Fabric is to work with multiple models and optimizers!

Speed up models by compiling them — Use torch.compile to speed up models on modern hardware

Train models with billions of parameters — Train the largest models with FSDP/TP across multiple GPUs and machines

Save and load very large models — Save and load very large models efficiently with distributed checkpoints