{"id":5647232,"date":"2023-02-08T17:56:47","date_gmt":"2023-02-08T22:56:47","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5647232"},"modified":"2023-07-28T11:26:50","modified_gmt":"2023-07-28T15:26:50","slug":"optimize-inference-scheduler","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/community\/optimize-inference-scheduler\/","title":{"rendered":"Accelerate Serving Stable Diffusion by Optimizing the Inference Scheduler"},"content":{"rendered":"

Key Takeaway<\/h3>Learn how to accelerate serving models with sequential inference steps, like Stable Diffusion.<\/div>\n

In this blog post, we demonstrate how we accelerated our serving of diffusion models by up to 18% for higher batch sizes and cover how to leverage expressive Lightning systems to design a new serving strategy for diffusion models.<\/p>\n

 <\/p>\n

What are diffusion models?<\/h2>\n

In Denoising Diffusion Probabilistic Models <\/em><\/a>(2020), Jonathan Ho et al. introduce a new diffusion probabilistic model with a sampling strategy called DDPM. During training, random Gaussian<\/span> noise is added gradually to the input image (the signal is destroyed) and the model is trained to predict the noise added to the image to recover the original image.\"\"<\/p>\n

During inference, starting from random Gaussian noise, a sampler uses the trained model to sequentially predict and apply modifications to the noise to uncover the hidden image. In other words, the model is given multiple opportunities to progressively improve upon its prediction. The total number of attempts it is given to improve upon its prediction is a hyperparameter referred to as the number of inference steps. In practice, we’ve found that on average 30 is a good number of inference steps.<\/span><\/p>\n

Here are the results where we used the same random seed but varied the number of model attempts from 1 to 30 steps with the following prompt: astronaut riding a horse, digital art, epic lighting, highly-detailed masterpiece trending HQ<\/span>.<\/em><\/p>\n

\"\"<\/p>\n

As you can see, the more attempts, the better the final prediction is.<\/p>\n

If you are interested in learning more about the theory behind diffusion models, we recommend The Illustrated Stable Diffusion<\/a> by Jay Alammar and How diffusion models work: the math from scratch<\/a> by\u00a0Sergios Karagiannakos and<\/a> Nikolas Adaloglou<\/a>.<\/p>\n

 <\/p>\n

Traditional inference method<\/h1>\n

When receiving multiple user requests, the current approach to serving is to group the prompts into a single input called \u201ca batch\u201d and run inference through the entire model using that same batch. Usually, the bigger the batch, the faster the inference per input is.<\/p>\n

Here is the pseudo-code associated with diffusion model inference. We pass the prompts to a text encoder, then run the encoded text and noisy images through the sample for the given number of inference steps.<\/p>\n

\n\nimgs = ... # batch of random noise images
\ntext_conditions = model.text_encoder([prompt_1, prompt_2, ...])\n\nfor _ in range(steps):
\nimgs = sampler.step(imgs, text_conditions, ...)\n\nfinal_imgs = imgs\n\n<\/code>