The Ultimate Battle of Language Models: Lit-LLaMA vs GPT3.5 vs Bloom vs …

Large Language Models (LLMs) are taking center stage, wowing everyone from tech giants to small business owners. These powerful models can understand complex information and provide human-like responses to a wide range of questions.

But with so many options out there, how do you know which one is the real deal?

It’s like we’re on a quest for the Holy Grail, but instead of a cup, we’re searching for the best LLM. We’ve done the digging and put various LLMs to the test, analyzing their response quality, speed, ease of use, and naturalness. Keep reading to see what we discovered!

The TLDR

GPT-3 and GPT-4 were a clear cut above the rest, but are a little harder to access given you need to pay for them and you’ll be sharing your data with OpenAI.

Flan-t5 (11b) and Lit-LLaMA (7b) answered all of our questions accurately and they’re publicly available. They’ll hold up in an interrogation even though they don’t really have a sense of humor.

Putting the “know-it-all” models to the test

We set out on a mission to put different LLMs to the test by asking them questions about topics they hadn’t encountered before. Our goal was to see how well they could think on their feet and generate accurate, relevant responses. To make things even more interesting, we chose a diverse range of models from the four main categories available on the market:

All these models are basically transformer designs with their own special sauce. – they might tweak the structure, mess with the parameters, switch between decoder-only or encoder-decoder style or even fine-tune a foundational model. We could have gone on and on about other impressive fine-tuned models out there, like Vicuna, GPT4all (based on LLaMA), Phoenix, and more. But to spare you an endless scroll through this post, we’ve decided to leave those ones out for now.

We decided to test the limits of LLMs by challenging them to answer questions about a recent event they couldn’t be trained on: Elon Musk’s Twitter username troubles. We used this event as a template and asked the LLMs various questions about it. You can see our code in this repo!

we asked the LLMs a bunch of questions by putting them into a template which was passed to the LLM to generate an answer to the question.

Once we had everything ready to go, we started bombarding the models with questions – some were easy-peasy, and some were more complicated:

1. What’s Elon’s new Twitter username?
2. Why is it funny that he cannot change it?
3. Make a joke about this.
4. How did this get started?

Open source models intended for commercial use did well with the straightforward questions

Turns out, these models did pretty well with the straightforward stuff. However, none of these models seemed to have a sense of humor! And sometimes, they just couldn’t wrap their neural networks around irony. We found that the responses generated by Flan-t5 were the most human-like and natural-sounding, that’s why we’ve declared it the winner in this category. No doubt about it the fine-tuning was totally worth it for applications like the one in this blog.

Open-source models intended for research were better, but still left you wanting for a more natural response

These models did a better job than the commercial open source models, and they definitely understood there were some funny elements to the article:

It is funny because it is ironic. Musk is trying to change the world, but he cannot change his own name.

But none mentioned how ironic is that Elon is the CEO of Twitter

Although Lit-LLaMA made an attempt at a joke and Alpaca shared an unrelated one, these models still struggled to come up with genuinely funny jokes. Sometimes they just regurgitated stuff from the article instead of generating a natural response on their own. Additionally, Alpaca had trouble providing Elon’s new username, which was a bit disappointing

Oh, and get this, Lit-LLaMA has a bit of a quirky side. Sometimes it can’t help itself and spits out answers in LaTeX format!

For example, when asked How did this get started? it responded:

\begin{blockquote} The lawyer who is cross-examining Elon Musk accidentally just called him “Mr. Tweet” instead of “Mr. Musk.” Elon says “Mr. Tweet” is all good. “That’s probably an accurate description,” \end{blockquote}

We used the Lit-LLaMA repository and the weights from LLaMA, which can only be used for research, and using them in other ways will violate the license. Lit-LLaMA with enterprise weights has yet to be made available. You can sign up here to get access as soon they are available.

Private models stood out above the rest, they even made us chuckle

No doubt about it, these models lived up to the hype. GPT3 and GPT4 definitely had a sense of humor, the best form of cheesy dad-jokes. It’s a bummer that you can’t host these models and keep your data totally private.

Wrapping up the takeaways

• Open-source models, especially Flan-t5, are pretty good at answering simple questions, but they can struggle a bit when things get more complex or if you want some humor.
• Fine-tuning those models can totally level up their performance game! Just check out how much better flan-t5 does compare to regular ol’ t5.
• Models under OpenRail License are worth checking out, although they tend to give short answers. But keep in mind that figuring out the license’s use restrictions might not be so straightforward.
• LLaMA 7B was decent at explaining stuff, but it relied on quoting the article a lot for context and wasn’t great at creating any original jokes. Plus, it seemed to have a bit of a thing for using LaTeX format for some reason!
• Alpaca had a few jokes ready to go, but when it came to answering questions about the article, it seemed to struggle a bit.
• The private models like GPT3 and GPT4 were amazing, giving us super-detailed summaries and even cracking some jokes. But, they are pricey, and definitely not the best choice if you’re working with sensitive information.
• Keep in mind that the performance of these models can vary depending on the context, and how you phrase your questions can make a big difference in the quality of the answers you get.
• Bottom line: these language models are seriously cool, and as they keep getting better, they’re going to become an essential tools for all kinds of different applications.

Note: Some of these models weren’t publicly available yet. But we couldn’t leave them out of the comparison, so we used this convenient tool,