{"id":5648737,"date":"2023-09-07T07:12:57","date_gmt":"2023-09-07T11:12:57","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5648737"},"modified":"2023-09-07T16:06:10","modified_gmt":"2023-09-07T20:06:10","slug":"pytorch-lightning-and-fabric","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/blog\/pytorch-lightning-and-fabric\/","title":{"rendered":"When to Use PyTorch Lightning or Lightning Fabric"},"content":{"rendered":"

Takeaways<\/h3> Discover how to use PyTorch Lightning or Lightning Fabric to pretrain your custom model checkpoints or finetune LLMs like <\/span>CodeLLaMA<\/span><\/a> and <\/span>LLama 2<\/span><\/a>.<\/span> <\/div>\n

Introduction<\/h2>\n

PyTorch Lightning<\/a> and Lightning Fabric<\/a> enable researchers and machine learning engineers to train PyTorch models at scale. Both frameworks do the heavy lifting for you and orchestrate training across multi-GPU and multi-Node environments. All you need to bring is a PyTorch module! And maybe a GPU \ud83d\ude06. So, why are there two frameworks?<\/p>\n

Short answer: varying levels of abstraction and control provide the versatility you need for a great development experience.<\/p>\n

In other words \u2013 we\u2019ve built some really useful features for you so that you don\u2019t have to. Depending on your need, you can choose either framework based on the amount of abstraction and control that you need or want. This is all about placing the power in your hands to get the job done in the way that works best for you.<\/p>\n

\"\"<\/p>\n

Similarities of PyTorch Lightning and Lightning Fabric<\/h2>\n

Becoming familiar with the shared features is a good starting point for understanding why we have two frameworks that focus on model training. Why do they share features? Because they are both built with the same tasks in mind, and we want to make the development experience as seamless as possible for you.<\/p>\n

A common benefit of PyTorch Lightning and Lightning Fabric is that both frameworks enable researchers and machine learning engineers to train in multi-device and multi-node environments with common flags such as devices, num_nodes, and strategy. Where devices is either the number of devices (CPUs, GPUs, TPUs or others) to train on, or the specific devices to train on given a multi-device environment. The num_nodes flag is used to set the number of clusters in a multi-node environment. And strategy is used to select the distributed training strategy i.e. DDP, FSDP, or DeepSpeed.<\/p>\n

use the accelerator flag along with devices to properly set the device type i.e. CPU, GPU, or TPU.
\n<\/p><\/blockquote><\/div>\n

Why is this a good thing? The ecosystem has a standard naming convention so that we can easily port concepts from one framework to the other. Meaning, we won\u2019t have to learn new argument names to do the same task.<\/p>\n