Prototype server APIs

Studios can expose arbitrary APIs and apps running on servers. These exposed servers also support serverless where the Studio will turn off if the app is not being used and will turn on again if it is being used.

Expose servers

This section describes how to expose servers that do not have web interfaces, ie: APIs.

Use the API builder

To expose a server via a public API, install the API builder plugin, create a new API and run your server on the Studio.

Now access the API from any other machine with an internet connection.

Add security

Restrict access to your endpoint by enabling security. There are 2 ways of securing an endpoint in the API builder.

Token authentication

Token-based authentication is a protocol that generates encrypted security tokens. It enables users to verify their identity to websites, which then generates a unique encrypted authentication token.

Basic authentication

Basic Authentication is a method for an HTTP user agent (e.g., a web browser) to provide a username and password when making a request. When employing Basic Authentication, users include an encoded string in the Authorization header of each request they make.

Save money with serverless

Serverless is the ability for Studios to sleep when inactive and auto-wake up when active. This saves you money and resources because the Studio only runs when it needs to.

Multiple servers in one Studio

Studios can host multiple servers at the same time. For example, you can deploy 5 different model APIs and configure serverless for each. Whenever any of the apps are used by a user, the Studio will wake up and render the page. If multiple apps are used, the Studio will route the requests accordingly. In this case, think of a Studio as a load balancer.

Examples

Model API

This is a simple, hello world model server using our serving library litserve

# server.py
import litserve as ls

# STEP 1: DEFINE YOUR MODEL API
class SimpleLitAPI(ls.LitAPI):
    def setup(self, device):
        # Setup the model so it can be called in `predict`.
        self.model = lambda x: x**2

    def decode_request(self, request):
        # Convert the request payload to your model input.
        return request["input"]

    def predict(self, x):
        # Run the model on the input and return the output.
        return self.model(x)

    def encode_response(self, output):
        # Convert the model output to a response payload.
        return {"output": output}


# STEP 2: START THE SERVER
if __name__ == "__main__":
    api = SimpleLitAPI()
    server = ls.LitServer(api, accelerator="auto")
    server.run(port=8000)

Here are other more advanced examples

Serve Hugging Face models

Use your preferred serving framework to deploy any Hugging Face model:

import litserve as ls
from transformers import pipeline

class HuggingFaceLitAPI(ls.LitAPI):
    def setup(self, device):
        model_name = "distilbert-base-uncased-finetuned-sst-2-english"
        self.pipeline = pipeline("text-classification", model=model_name, device=0 if device=="gpu" else -1)

    def decode_request(self, request):
        return request["text"]

    def predict(self, text):
        return self.pipeline(text)

    def encode_response(self, output):
        return {"label": output[0]["label"], "score": output[0]["score"]}

if __name__ == "__main__":
    api = HuggingFaceLitAPI()
    server = ls.LitServer(api, accelerator="gpu", devices=4, workers_per_device=2)
    server.run(port=8000)

Here are more examples:

FastAPI endpoint

Studios support any kind of server as long as it exposes a port. For example this studio shows how to run a fastAPI server that deploys an image classification model.

Generic servers

Any server that exposes a port can run in serverless mode on a Studio. Visit our full collection of Studios for serving here.