10.2 Fabric – Scaling PyTorch Models without Boilerplate Code

References

• Official Fabric documentation

Code

• Part 2, Training an LLM with PyTorch and Fabric: 10.2-fabric/

What we covered in this video lecture

In this lecture, we learned about Fabric, an open-source library designed for scaling PyTorch models without requiring a significant rewriting of code.

Fabric integrates with PyTorch, allowing the implementation of features such as mixed precision training and multi-GPU training, typically found in the Lightning Trainer, without the need to restructure our code around a Lightning Module, which can be useful if we adopt or port existing PyTorch code.

Additional resources if you want to learn more

Fabric supports many advanced use cases, including loggers and callback, which we didn’t cover in this lecture. To learn more about those, I recommend checking out the official Fabric documentation.

Moreover, as a small fun-fact: The Lit-LLaMA repository for implementing state-of-the-art LLM techniques (such as distributed training and efficient finetuning) makes use of Fabric as well!

Watch Video 1 Mark complete and go to Unit 10.3 →