{"id":5648406,"date":"2023-07-22T19:54:32","date_gmt":"2023-07-22T23:54:32","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5648406"},"modified":"2024-07-02T13:42:59","modified_gmt":"2024-07-02T17:42:59","slug":"how-to-build-a-chatbot-using-open-source-llms-like-llama-2-and-falcon","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/community\/tutorial\/how-to-build-a-chatbot-using-open-source-llms-like-llama-2-and-falcon\/","title":{"rendered":"How to build a chatbot using open-source LLMs like Llama 2 and Falcon"},"content":{"rendered":"
\n

Takeaways<\/h3>In this blog, we’ll learn how to build a chatbot using open-source LLMs. We will be using Lit-GPT<\/a> and LangChain. Lit-GPT is an optimized collection of open-source LLMs for finetuning and inference. It supports – Falcon, Llama 2, Vicuna, LongChat, and other top-performing open-source large language models.<\/div>\n<\/header>\n

 <\/p>\n

This article is outdated with respect to the latest LitGPT version. Please check the official repo<\/a> for latest code samples.<\/p>\n

 <\/p><\/blockquote>\n

\n

Advancements of LLMs in Chatbot Development<\/h2>\n

Chatbots have become integral to many industries, including e-commerce, customer service, and healthcare. With advancements in large language models, building a chatbot has become easier than ever. Chatbots can find relevant information from a large knowledge base and present it to the user. It can connect to a database, query from a vector database, and answer questions based on documents.<\/p>\n

LLMs have the ability to generate answers based on the given prompt. They can now even consume a whole document and respond to queries based on the document. In-context learning provides a way where LLM learns how to solve a new task at inference time without any change in the model weights. The prompt contains an example of the task that model is expected to do. The prompt engineering discussion can go much longer but we will be focusing on how to build your first MVP chatbot.<\/p>\n

Making conversation with an LLM<\/h2>\n

Foundational large language models (LLMs) are trained to predict the next word. These LLMs are then finetuned on an instruction dataset, which consists of user input<\/em>, context<\/em>, and expected output<\/em>, to follow human instruction and generate relevant responses. An example of an instruction prompt to make the LLM behave like a chatbot is provided below.<\/p>\n

A chat between a curious user and an artificial intelligence assistant.\r\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\r\n\r\nUSER: {input}\r\nASSISTANT:<\/code><\/pre>\n
The first two lines serve as instructions to the LLM, directing it to provide helpful, detailed, and polite responses as a chatbot assistant. The third line, USER: {input}<\/code>, represents the user input, where {input}<\/code> will be replaced by the user’s query. The LLM will then start predicting the next words to produce a response to the user’s prompt.<\/p>\n
In this tutorial, we will use an instruction-tuned model and provide the user input as a prompt. Let’s create our first chatbot by using the prompt defined above. We will use LongChat, which is a LLaMA-like model trained on a chat dataset with a context length of 16K (~12K words<\/a>).<\/p>\n
We will define our prompt template as longchat_prompt_template<\/code>, and for each user query, we will format the string and feed it to the model.<\/p>\n
longchat_template = \"\"\"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.\r\nUSER: {input}\r\nASSISTANT:\"\"\"\r\n\r\noutput = longchat_template.format(input=\"My name is Aniket?\")<\/code><\/pre>\n
We will be using the llm-inference<\/code> library, which is just a wrapper over Lit-GPT to provide an API interface for loading the model and using the generation method. First, we load our model with 4-bit quantization using bnb.nf4<\/code> that will require about 6GB GPU memory.<\/p>\n
# pip install llm-inference\r\nfrom llm_inference import LLMInference, prepare_weights\r\nfrom rich import print\r\n\r\npath = str(prepare_weights(\"lmsys\/longchat-7b-16k\"))\r\nmodel = LLMInference(checkpoint_dir=path, quantize=\"bnb.nf4\")<\/code><\/pre>\n
Next, use the model.chat<\/code><\/a> method, an API interface for the Lit-GPT chat script<\/a>, to generate the response from the formatted template with user input.<\/p>\n
# pip install llm-inference\r\nfrom llm_inference import LLMInference, prepare_weights\r\nfrom rich import print\r\n\r\npath = str(prepare_weights(\"lmsys\/longchat-7b-16k\"))\r\nmodel = LLMInference(checkpoint_dir=path, quantize=\"bnb.nf4\")\r\n\r\nlongchat_template = \"\"\"A chat between a curious user and an artificial intelligence assistant.\r\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\r\nUSER: {input}\r\nASSISTANT:\"\"\"\r\n\r\noutput = model.chat(longchat_template.format(input=\"My name is Aniket?\"))\r\nprint(output)<\/code><\/pre>\n
\"\"<\/figure>\n
Assistant doesn\u2019t have a memory<\/h2>\n
Our chatbot works great, but there is an issue: our assistant is stateless and forgets the previous interactions. As you can see below, the user prompts the bot with their name and then, in the second interaction, asks “What is my name?” However, the assistant is not able to provide the requested information due to its lack of memory.<\/p>\n
\"\"<\/figure>\n
Fixing this issue is not too difficult. We can provide the context to the LLM through the prompt and it should be able to look it up. We can store our conversation history and inject it into the prompt like the example below where the user provides their name as context, and the model was able to answer their name in a follow-up question.<\/p>\n
\"\"<\/figure>\n
Memory and prompting – Maintaining a continued conversation<\/h2>\n
LLMs are stateless interfaces that provide language understanding and retrieval capabilities. However, LLMs do not have memory, so subsequent queries are not affected by earlier calls. Therefore, if you provide your name in one query and expect the LLM to remember it for the next query, it will not happen.<\/p>\n
To give LLMs the ability to remember previous interactions, we can store the conversation history in the prompt template as context for the model. We create the following prompt template –<\/p>\n
A chat between a curious user and an artificial intelligence assistant.\r\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\r\nContext:\r\n{history}\r\nUSER: {input}\r\nASSISTANT:<\/code><\/pre>\n
{history}<\/code> is replaced with the previous conversations, and {input}<\/code> is replaced by the current query from the user. We keep updating the history<\/code> and input<\/code> for each interaction. LangChain<\/code> provides some useful classes for formatting prompts and updating the context using more advanced ways, like looking up context from a vector database<\/a>.<\/p>\n
We will use the PromptTemplate<\/code> class with history<\/code> and input<\/code> as variables.<\/p>\n
from langchain.prompts import PromptTemplate\r\n\r\nlongchat_template = \"\"\"A chat between a curious user and an artificial intelligence assistant.\r\nThe assistant gives helpful, detailed, and polite answers to the user's questions.\r\nContext:\r\n{history}\r\nUSER: {input}\r\nASSISTANT:\"\"\"\r\n\r\nlongchat_prompt_template = PromptTemplate(\r\n    input_variables=[\"input\", \"history\"], template=longchat_template\r\n)\r\nprint(longchat_prompt_template.format(\r\n    input = \"What is my name?\",\r\n    history =\"USER: Hi, I am Aniket!\\nAssistant: How can I help you Aniket?\"\r\n))<\/code><\/pre>\n
You can format the longchat_prompt_template<\/code> using longchat_prompt_template.format<\/code> method by providing input<\/code> and history<\/code> .<\/p>\n
\"\"<\/figure>\n
Next, we create a conversation chain using the ConversationChain<\/code> class. It takes the LLM, prompt template, and a memory manager object as input. We will use ConversationBufferMemory<\/code>, which stores all our conversations and update the prompt template history<\/code> on each interaction.<\/p>\n
from langchain.chains import ConversationChain\r\nfrom llm_chain import LitGPTLLM\r\nfrom llm_inference import LLMInference\r\n\r\npath = \"checkpoints\/lmsys\/longchat-7b-16k\"\r\nmodel = LLMInference(checkpoint_dir=path, quantize=\"bnb.nf4\")\r\n\r\n\r\nllm = LitGPTLLM(model=model)\r\nconversation = ConversationChain(\r\n    llm=llm,\r\n    memory=ConversationBufferMemory(ai_prefix=\"Assistant\", human_prefix=\"User\"),\r\n    prompt=longchat_prompt_template,\r\n)\r\n\r\nconversation(\"hi, I am Aniket\")[\"response\"]<\/code><\/pre>\n
\"\"<\/figure>\n
You can access the memory using the conversation<\/code> object and manipulate it as well. To print the current conversation we can run print(conversation.memory.buffer)<\/code><\/p>\n
\"\"<\/figure>\n
This memory is updated into the prompt as context after each conversation.<\/p>\n
QA over Documents as context<\/h2>\n
The chatbots can be further extended to do more complex tasks like extracting documents from a database and answering questions based on the given document as context. You can build a document QnA bot with this technique. We won\u2019t go deep here in this blog, but if you\u2019re curious, you can replace the model memory with another LLM that searches for relevant documents and update the context in the prompt template with the extracted document. You can read more about it on the LangChain example here<\/a>.<\/p>\n
Conclusion<\/h2>\n
\n
In conclusion, building a chatbot using open-source LLMs has become easier than ever with advancements in large language models. The ability of LLMs to generate answers based on the given prompt, even from a whole document, has made them an integral part of many industries. By using instruction-tuned models and providing user input as a prompt, we can create chatbots that provide helpful, detailed, and polite responses. While chatbots are stateless, we can use chat history as context for the model to remember previous interactions. Finally, the chatbots can be further extended to do more complex tasks like extracting documents from a database and answering questions based on the given document as context.<\/p>\n<\/div>\n
Resources<\/h2>\n