FAQ

LitServe vs XYZ

LitServe is built on FastAPI but optimized for AI workloads with specialized multi-process handling. It is designed to be flexible first, without compromising on performance. Flexibility easily enables things like composite AI systems, RAG systems, etc...

For a vllm, ollama alternative, you are looking for LitGPT serve.

LitServe vs Ollama, vLLM

LitServe is one level of abstraction below Ollama and vLLM. We designed LitServe so you could build something like vLLM and Ollama yourself for bespoke use cases. To build something like vLLM you would take LitServe, your LLM inference code and add custom kernels, and optimizations like KV-caching.

For a vllm, ollama alternative, you are looking for LitGPT serve.

LitServe vs FastAPI

LitServe is built on top of FastAPI but optimized for AI model serving. When you write your own Fast API servers there is a lot of code that you still have to write yourself (streaming, batching, etc...) that's usually not as efficient or well tested as our implementations. A custom Fast API server is also less reusable, so you'd have to implement streaming AGAIN for every single custom server you deploy.

In addition, LitServe benefits from the large Lightning AI open-source community that contributes fixes, optimizations and improvements to make LitServe enterprise grade and high throughput at scale.

TorchServe vs TFServe

TorchServe is the traditional server that's been used to deploy PyTorch models. TorchServe is built with a Java runtime and has different trade-offs. LitServe has been built in full stack Python with similar performance (in cases at least 10% faster) but it's framework agnostic (supports Jax, PyTorch, Tensorflow). We encourage you to try the different solutions yourself and choose what works best for your setting!

Is LitServe for LLMs

LitServe (at this stage) has not been optimized for fast LLM serving. It does a good job at serving LLMs that are used by a few users or internally at companies. Other solutions such as VLLM are more optimized for LLM serving because of custom kernels, kv-caching and other optimizations overfit to LLMs. These are optimizations you can find in LitGPT and do yourself.

However, vLLM and similar frameworks only work with LLMs, whereas LitServe can serve ANY AI model such as vision models, audio, BERT (nlp, text), video, tabular models, random forests, etc...