{"id":5649055,"date":"2023-10-12T21:58:40","date_gmt":"2023-10-13T01:58:40","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5649055"},"modified":"2023-10-16T15:07:34","modified_gmt":"2023-10-16T19:07:34","slug":"lora-insights","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/community\/lora-insights\/","title":{"rendered":"Finetuning LLMs with LoRA and QLoRA: Insights from Hundreds of Experiments"},"content":{"rendered":"

Takeaways<\/h3>LoRA is one of the most widely used, parameter-efficient finetuning techniques for training custom LLMs. From saving memory with QLoRA to selecting the optimal LoRA settings, this article provides practical insights for those interested in applying it.<\/div>\n

 <\/p>\n

Introduction: Getting the Most out of LoRA<\/span><\/h2>\n

I’ve run hundreds, if not thousands, of experiments involving LoRA over the past few months. A few weeks ago, I took the time to delve deeper into some of the hyperparameter choices.<\/span><\/p>\n

This is more of an experimental diary presented in sequential order. I hope it proves useful to some. Specifically, I aim to address questions about the value of QLoRA, whether to replace AdamW with SGD, the potential use of a scheduler, and how to adjust the LoRA hyperparameters.<\/span><\/p>\n

There’s a lot to discuss on the experimental side, so I’ll keep the introduction to LoRA brief.<\/span><\/p>\n

In short, LoRA, short for Low-Rank Adaptation (<\/span>Hu et al 2021<\/span><\/a>), adds a small number of trainable parameters to the model while the original model parameters remain frozen.<\/span><\/p>\n

LoRA decomposes a weight matrix into two smaller weight matrices, as illustrated below, to approximate full supervised finetuning in a more parameter-efficient manner.<\/span><\/p>\n

\"\"<\/p>\n

For more details about LoRA, please see my in-depth article <\/span>Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA)<\/span><\/a>.<\/span><\/p>\n

The topics we are going to cover in this article as organized as follows:<\/p>\n

1. Evaluation Tasks and Dataset
\n2. Code Framework
\n3. Choosing a Good Base Model
\n4. Evaluating the LoRA Defaults
\n5. Memory Savings with QLoRA
\n6. Learning Rate Schedulers and SGD
\n7. Iterating Over the Dataset Multiple Times
\n8. LoRA Hyperparameter Tuning Part 1: LoRA for All Layers
\n9. LoRA Hyperparameter Tuning Part 2: Increasing R
\n10. LoRA Hyperparameter Tuning Part 3: Changing Alpha
\n11. LoRA Hyperparameter Tuning Part 3: Very Large R
\n12. Leaderboard Submission
\n13. Conclusion<\/p>\n

 <\/p>\n

Evaluation Tasks and Dataset<\/h2>\n

The focus of this article is on selecting the optimal settings. To stay within a reasonable scope, I’m keeping the dataset fixed and focusing solely on supervised instruction-finetuning of LLMs. (Modifications to the dataset or finetuning for classification might be addressed in future articles.)<\/span><\/p>\n

For the model evaluation, I selected a small subset of tasks from Eleuther AI’s <\/span>Evaluation Harness<\/span><\/a>, including <\/span>TruthfulQA<\/span><\/a>, <\/span>BLiMP Causative,<\/span><\/a> MMLU Global Facts<\/span><\/a>, and simple arithmetic tasks with two (arithmetic 2ds) and four digits (arithmetic 4ds).<\/span><\/p>\n

In each benchmark, the model performance score is normalized between 0 and 1, where 1 is a perfect score. TruthfulQA reports two scores, which are defined as follows:<\/span><\/p>\n