{"id":5647794,"date":"2023-04-26T08:51:09","date_gmt":"2023-04-26T12:51:09","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5647794"},"modified":"2023-06-22T13:13:15","modified_gmt":"2023-06-22T17:13:15","slug":"lora-llm","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/community\/tutorial\/lora-llm\/","title":{"rendered":"Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA)"},"content":{"rendered":"

Key takeaway<\/h3> In the rapidly evolving field of AI, using large language models in an efficient and effective manner is becoming more and more important. In this article, you will learn how to tune an LLM with Low-Rank Adaptation (LoRA) in a computationally efficient manner! <\/div>\n

 <\/p>\n

Why Finetuning?<\/strong><\/h2>\n

Pretrained large language models are often referred to as foundation models for a good reason: they perform well on various tasks, and we can use them as a foundation for finetuning on a target task. As discussed in our previous article (Understanding Parameter-Efficient Finetuning of Large Language Models: From Prefix Tuning to LLaMA-Adapters<\/a>), we discussed finetuning allows us to adapt a model to a target domain and target task. Still, it can be computationally very costly — the larger the model, the more expensive it is to update its layers.<\/p>\n

As an alternative to updating all layers, parameter-efficient methods such as prefix tuning and adapters have been developed — for a detailed review, please see our previous post<\/a>. Now, there is one more popular parameter-efficient finetuning technique: Low-rank adaptation (LoRA) by Hu et al<\/a>. What is LoRA? How does it work? And how does it compare to the other popular finetuning approaches? Let’s answer all these questions in this article!<\/p>\n

 <\/p>\n

\"PCA<\/div>\n

 <\/p>\n

 <\/p>\n

Making Weight Updates More Efficient<\/strong><\/h2>\n

Building on this idea outlined above, the paper\u00a0LoRA: Low-Rank Adaptation of Large Language Models<\/a>\u00a0proposes to decompose the weight changes,\u00a0\u0394W<\/em>, into a lower-rank representation. (To be technically correct, LoRA does not decompose the matrices directly, but it learns the decomposed matrices via backpropagation \u2014 this is a nitpicky detail that will make sense later).<\/p>\n

Before we take a closer look at LoRA, let\u2019s briefly explain the training procedure during regular finetuning. So, what are the weight changes\u00a0\u0394W<\/em>? Suppose\u00a0W<\/em>\u00a0represents the weight matrix in a given neural network layer. Then, using regular backpropagation, we can obtain the weight update\u00a0\u0394W<\/em>, which is typically calculated as a negative gradient of the loss times the learning rate:<\/p>\n

\u0394W<\/em> = \u03b1<\/em> ( -\u2207 LW<\/sub>).<\/p>\n

Then, when we have \u0394W<\/em>, we can update the original weights as follows: W<\/em>‘ = W<\/em> + \u0394W<\/em>. This is illustrated in the figure below (bias vectors are omitted for simplicity):<\/p>\n

Alternatively, we can keep the weight update matrix separate and compute the outputs as follows: h = W x + \u0394W x<\/em>,<\/p>\n

 <\/p>\n

\"Regular<\/div>\n

 <\/p>\n

where x<\/span> represents the inputs, as illustrated below.<\/p>\n

 <\/p>\n

\"\"<\/div>\n

 <\/p>\n

Why would we do this? For now, this alternative formulation serves a pedagogical goal to illustrate LoRA, but we will come back to it.<\/p>\n

So, when we train fully connected (i.e., “dense”) layers in a neural network, as shown above, the weight matrices usually have full rank, which is a technical term meaning that a matrix does not have any linearly dependent (i.e., “redundant”) rows or columns. In contrast, to full rank, low rank means that the matrix has redundant rows or columns.<\/p>\n

So, while the weights of a pretrained model have full rank on the pretrained tasks, the LoRA authors point out that pretrained large language models have a low “intrinsic dimension” when they are adapted to a new task, according to\u00a0Aghajanyan et al.<\/a>\u00a0(2020).<\/p>\n

A low intrinsic dimension means the data can be effectively represented or approximated by a lower-dimensional space while retaining most of its essential information or structure. In other words, this means we can decompose the new weight matrix for the adapted task into lower-dimensional (smaller) matrices without losing too much important information.<\/p>\n

For example, suppose \u0394W<\/em> is the weight update for an A \u00d7 B<\/em> weight matrix. Then, we can decompose the weight update matrix into two smaller matrices: \u0394W = WA<\/sub> WB<\/sub><\/em>, where WA<\/sub><\/em> is an an A \u00d7 r<\/em>-dimensional matrix, and WB<\/sub><\/em> is an an r \u00d7 B<\/em>-dimensional matrix. Here, we keep the original weight W<\/em> frozen and only train the new matrices WA<\/sub><\/em> and WB<\/sub><\/em>. This, in a nutshell, is the LoRA method, which is illustrated in the figure below.<\/p>\n

 <\/p>\n

 <\/p>\n

\"\"<\/div>\n

 <\/p>\n

 
\nChoosing the rank<\/strong><\/p>\n

Note that r<\/em>, in the figure above, is a hyperparameter here that we can use to specify the rank of the low-rank matrices used for adaptation. A smaller r<\/em> leads to a simpler low-rank matrix, which results in fewer parameters to learn during adaptation. This can lead to faster training and potentially reduced computational requirements. However, with a smaller r<\/em>, the capacity of the low-rank matrix to capture task-specific information decreases. This may result in lower adaptation quality, and the model might not perform as well on the new task compared to a higher r<\/em>. In summary, choosing a smaller r<\/em> in LoRA has a trade-off between model complexity, adaptation capacity, and the risk of underfitting or overfitting. It\u2019s thus important to experiment with different r<\/em> values to find the right balance to achieve the desired performance on the new task.<\/p>\n

 
\nImplementing LoRA<\/strong><\/p>\n

The implementation of LoRA is relatively straight-forward. We can think of it as a modified forward pass for the fully connected layers in an LLM. In pseudo-code, this looks like as follows:<\/p>\n


\ninput_dim = 768  # e.g., the hidden size of the pre-trained model
\noutput_dim = 768  # e.g., the output size of the layer
\nrank = 8  # The rank 'r' for the low-rank adaptation\n\nW = ... # from pretrained network with shape input_dim x output_dim\n\nW_A = nn.Parameter(torch.empty(input_dim, rank)) # LoRA weight A
\nW_B = nn.Parameter(torch.empty(rank, output_dim)) # LoRA weight B\n\n# Initialization of LoRA weights
\nnn.init.kaiming_uniform_(W_A, a=math.sqrt(5))
\nnn.init.zeros_(W_B)\n\ndef regular_forward_matmul(x, W):
\n    h = x @ W
\nreturn h\n\ndef lora_forward_matmul(x, W, W_A, W_B):
\n    h = x @ W  # regular matrix multiplication
\n    h += x @ (W_A @ W_B)*alpha # use scaled LoRA weights
\nreturn h
\n<\/code>