{"id":5649138,"date":"2023-10-31T20:41:18","date_gmt":"2023-11-01T00:41:18","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5649138"},"modified":"2023-11-14T11:54:02","modified_gmt":"2023-11-14T16:54:02","slug":"doubling-neural-network-finetuning-efficiency-with-16-bit-precision-techniques","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/community\/tutorial\/doubling-neural-network-finetuning-efficiency-with-16-bit-precision-techniques\/","title":{"rendered":"Doubling Neural Network Finetuning Efficiency with 16-bit Precision Techniques"},"content":{"rendered":"

Takeaways<\/h3>This guide targets PyTorch model training, illustrating how you can adjust the floating point precision to drastically enhance training speed and halve memory consumption, all without compromising the prediction accuracy.<\/div>\n
\"\"

An excerpt of the improvements we can gain from leveraging the techniques introduced in this article.<\/p><\/div>\n

 <\/p>\n

In this article, we will work with a vision transformer from PyTorch’s <\/span>Torchvision<\/span><\/a> library, providing simple code examples that you can execute on your own machine without the need to download and install numerous code and dataset dependencies. The self-contained baseline training script comprises approximately 100 lines of code, excluding whitespace and code comments.<\/span><\/p>\n

All benchmarks were executed using the open source <\/span>Lightning 2.1<\/span><\/a> package with PyTorch 2.1 and CUDA 12.1 on a single A100 GPU. You can find all code examples <\/span>here on GitHub<\/span><\/a>.<\/span><\/p>\n

 <\/p>\n

Finetuning a Vision Transformer on a Single GPU<\/span><\/h2>\n

While we are working with a vision transformer here (the ViT-L-16 model from the paper\u00a0<\/span><\/p>\n

An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale<\/span><\/a>), all the techniques used in this article transfer to other models as well: Convolutional networks, large language models (LLMs), and others.<\/span><\/p>\n

Note that we are finetuning the model for classification instead of training it from scratch to optimize predictive performance.<\/span><\/p>\n

Let’s begin with a simple baseline in PyTorch. The complete code is available <\/span>here on GitHub<\/span><\/a>, which implements and finetunes a vision transformer:<\/span><\/p>\n

The core code for implementing this vision transformers is as follows:<\/span><\/p>\n

# Import torchvision\r\nfrom torchvision.models import vit_l_16\r\nfrom torchvision.models import ViT_L_16_Weights\r\n\r\n# Initialize pretrained vision transformer\r\nmodel = vit_l_16(weights=ViT_L_16_Weights.IMAGENET1K_V1)\r\n\r\n# replace output layer\r\nmodel.heads.head = torch.nn.Linear(in_features=1024, out_features=10)\r\n\r\n# finetune model<\/pre>\n
The relevant benchmark numbers for this baseline are as follows:<\/span><\/p>\n