Multiple API endpoints on one port

Hosting each AI model on a separate server increases infrastructure cost and complicates scaling. LitServe solves this by allowing you to serve multiple machine learning models on a single server, each with its own API endpoint, all under the same HTTP port - like :8000. This setup simplifies deployment, reduces compute expenses, and supports fast delivery of features in production environments.

That means you can handle tasks like text classification, embeddings, and streaming text generation, without spinning up a new server for each. It keeps the deployment infrastructure lean and clean.

Background

What is an endpoint?

An endpoint is a unique URL path that routes requests to a specific model. For instance, you might have /embed for text embeddings and /generate for text generation, allowing different models to serve distinct tasks on the same server.

Applications

• Serve different model versions to different users.
• Reduce costs through resource sharing.
• Handle diverse tasks like image processing, OCR, text generation, and text embedding on the same server.

Multiple model endpoints under the same port

LitServe allows us to expose multiple model APIs through different endpoints - all served on the same HTTP port (such as 8000). This avoids the need to allocate and manage separate ports for each model, simplifying deployment in cloud environments, containers, and load balancers.

Each API can define its own route using the api_path argument, and LitServe automatically handles routing based on the incoming request path. This architecture supports clear separation of concerns while keeping infrastructure minimal.

For example:

• POST /classify-text → Sentiment analysis API
• POST /v1/chat/completions → Text generation API

To serve multiple APIs on a single server, pass a list of LitAPI implementations, each configured with its own endpoint path and optional parameters such as batch size or batch timeout. The example below demonstrates how to serve two different models - sentiment analysis and text generation, using a single server instance.

from transformers import pipeline
import litserve as ls

class SentimentAnalysisAPI(ls.LitAPI):
    def setup(self, device):
        self.model = pipeline("sentiment-analysis", model="stevhliu/my_awesome_model", device=device)

    def decode_request(self, request: dict):
        return request["text"]

    def predict(self, text):
        return self.model(text)[0]

class TextGenerationAPI(ls.LitAPI):
    def setup(self, device):
        self.generator = pipeline("text-generation", model="gpt2", device=device)

    def decode_request(self, request: dict):
        return request["prompt"]

    def predict(self, prompt):
        return self.generator(prompt)[0]["generated_text"]

if __name__ == "__main__":
    sentiment_api = SentimentAnalysisAPI(api_path="/classify-text")
    chat_api = TextGenerationAPI(api_path="/v1/chat/completions")

    server = ls.LitServer([sentiment_api, chat_api])
    server.run(port=8000)

Advanced

Streaming and non-streaming APIs

Many real-world applications require serving both streaming and non-streaming models—such as combining a text generation model that streams output (like OpenAI’s API) with a classification model that returns the full result at once. LitServe makes it easy to run both types of APIs side by side on a single server.

For example, we can serve a streaming LLM for text generation (using LitServe's OpenAISpec) and a standard classification model together:

from transformers import pipeline
import litserve as ls

class SentimentAnalysisAPI(ls.LitAPI):
    def setup(self, device):
        self.model = pipeline("sentiment-analysis", model="stevhliu/my_awesome_model", device=device)

    def decode_request(self, request: dict):
        return request["text"]

    def predict(self, text):
        return self.model(text)[0]

class TextGenerationAPI(ls.LitAPI):
    def setup(self, device):
        self.generator = pipeline("text-generation", model="gpt2", device=device)

    async def predict(self, prompt):
        for output in self.generator(prompt[-1]["content"]):
            yield output["generated_text"]


if __name__ == "__main__":
    sentiment_api = SentimentAnalysisAPI()
    chat_api = TextGenerationAPI(spec=ls.OpenAISpec())

    server = ls.LitServer([sentiment_api, chat_api])
    server.run(port=8000)

With this setup, a single LitServe instance can handle both streaming and non-streaming endpoints, making it easier to deliver rich, multi-modal AI services without extra infrastructure.

Conclusion

Multiple model serving with LitServe offers flexibility in deployment architecture. The load-all-together approach provides a simple starting point for serving multiple models, while the planned multiple routes feature will enable more complex deployment patterns. Choose the approach that best matches your application's needs for resource utilization, maintainability, and scalability.