{"id":5649209,"date":"2023-11-15T16:50:11","date_gmt":"2023-11-15T21:50:11","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5649209"},"modified":"2023-11-15T16:50:11","modified_gmt":"2023-11-15T21:50:11","slug":"8-bit-quantization-with-lightning-fabric","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/blog\/8-bit-quantization-with-lightning-fabric\/","title":{"rendered":"8-bit Quantization with Lightning Fabric"},"content":{"rendered":"

Takeaways<\/h3> Readers will learn the basics of Lightning Fabric\u2019s plugin for 8-bit quantization. <\/div>\n

\"\"<\/p>\n

Introduction<\/h2>\n

The aim of 8-bit quantization is to reduce the memory usage of the model parameters by using lower precision types than full (float32) or half (bfloat16) precision. Meaning \u2013 8-bit quantization compresses models that have billions of parameters like Llama 2 or SDXL and makes them require less memory.<\/p>\n

Thankfully, Lightning Fabric<\/a> makes quantization as easy as setting a mode flag in a plugin!<\/p>\n

8-bit Quantization<\/h2>\n

8-bit quantization is discussed in the popular paper 8-bit Optimizers via Block-wise Quantization<\/a> and was introduced in FP8 Formats for Deep Learning<\/a>.<\/p>\n

As stated in the original paper, 8-bit quantization was the natural progression after 16-bit precision. Although it was the natural progression, the implementation was not as simple as moving from FP32 to FP16 \u2013 as those two floating point types share the same representation scheme and 8-bit does not.<\/p>\n

8-bit quantization requires a new representation scheme, and this new scheme allows for fewer numbers to be represented than FP16 or FP32. This means model performance may be affected when using quantization, so it is good to be aware of this trade-off. Additionally, model performance should be evaluated in its quantized form if the weights will be used on an edge device that requires quantization.<\/p>\n

Lightning Fabric can use 8-bit quantization by setting the mode<\/code> flag to int8<\/code> for inference.<\/p>\n

from lightning.fabric import Fabric\r\nfrom lightning.fabric.plugins import BitsandbytesPrecision\r\n\r\n# available 8-bit quantization modes\r\n# (\"int8\")\r\n\r\nmode = \"int8\"\r\nplugin = BitsandbytesPrecision(mode=mode)\r\nfabric = Fabric(plugins=plugin)\r\n\r\nmodel = CustomModule() # your PyTorch model\r\nmodel = fabric.setup_module(model) # quantizes the layers<\/code>