At June’s inaugural Lightning Developer Conference, Lightning AI’s Staff Product Manager Eden Afek explored how you can bring your machine learning models to life through AI-powered Lightning Apps.
You can watch her presentation below, or continue reading to learn how Lightning AI simplifies this process, allowing you to create a fully interactive application in a matter of days — not months.
Open Source Makes Everything Easier
One of Lightning AI‘s core experiences is our gallery of open-source Lightning Apps and Components that you can use to extend your research and build it into an end-to-end ML system in minutes. Because they’re all open source, you can download the code and modify it in whatever way you need to fit your unique use case.
These constantly growing App and Component galleries allow you to piece together whatever you need to create a machine learning system. This is possible whether you’re doing research, model building and training, pushing to production, deploying to a public cloud, or even keeping your Lightning App private on your own cluster. Whatever you need is at your fingertips!
To get a better sense of how Lightning accelerates machine learning development, let’s walk through a few use cases.
Jane, Academic Researcher
Jane is a researcher working on computer vision for robotic. Recently, she began writing a paper for an upcoming conference. Despite her rapidly approaching deadline, she hasn’t been able to find the right hyperparameters to train her model on, and has no idea how to share her work with others in her lab. These roadblocks risk jeopardizing her ability to finish the paper in time, and her two options don’t seem like they will allow her to meet her deadline:
- She could train on a shared university cluster, but that might take months.
- She could train on the cloud, but Jane isn’t a cloud expert and doesn’t have the time or resources to learn how to get started or set up an environment.
Lightning AI, however, offers an ideal third option.
Jane can quickly browse the Lightning App gallery for something she can use as a jumping-off point. For example, Jane can use either the Collaborate App (to train with her lab partners) or the Lightning HPO Component (to help select her hyperparameters):
- Collaborate: A Lightning App that can train a model at scale, collaboratively, with colleagues or the AI community. This app allows you to train a model across machines spread over the internet. You don’t have to worry about connecting mixed sets of distributed devices with different GPUs. It’s as simple as beginning your own training, and others can join by clicking a link you provide. The app also handles connecting, updating, and monitoring your training all through the UI!
- Lightning HPO: A component that provides a pythonic implementation of Scalable Hyperparameter Tuning.
Most Lightning Apps give you the option to Launch it immediately, so you can see the app in action:
Once Jane finds a Lightning App she wants to use, she can click Clone & Run. This will copy an instance of this Lightning App into Jane’s account, and, in a few moments, she’ll have her first App running on the Lightning Cloud. It’s that easy!
If Jane wants to make any changes to her App, all she needs to do is to click the Code button to download its code.
Remember, the code is open source and allows you to change it!
The code downloads as an app.py file.
Now, Jane just changes the model, finds the hyperparameters she wants to sweep on, and with a single CLI command can deploy to the cloud.
Once the App is running, Jane can go to the admin dashboard where she can see all the components that are a part of her Lightning App, view what trainings are running or stopped, and any other details important to her.
When she is ready to share, Jane can click the Share button (next to the Code button at the top right). This will copy the link to her clipboard, and she can then share it with her collaborators.
Juan, Machine Learning Engineer
Juan is an ML engineer at a news company. He is tasked with improving the recommendations given to readers.
Although Juan has some experience with machine learning and models, he is unsure how to set up an end-to-end ML recommendation system that would work in production.
In the past, this work would have required an expansion of the ML team to handle different parts of the system, which would require resources and time that are unavailable to Juan’s company. If Juan were to handle building the system on his own, he would have to spend months researching, trying out available tools, and integrating them.
Juan doesn’t have the time for these options so, like Jane, he turns to Lightning AI to see if it can give him what he needs. Looking through the App Gallery, he finds the AI-powered HackerNews App:
The AI-powered HackerNews Lightning App recommends HackerNews stories to users based on their favorite HackerNews articles. The underlying magic powering this Lightning App is the recommendation system. This App serves to illustrate an end-to-end MLOps workflow for enterprise recommendation systems — for both news articles and beyond.
Juan can download the code right away, or Clone & Run to install an instance in his own Lightning account before downloading and modifying the code.
There is one complication that concerns Juan: the data he is training on is private and he therefore can’t run the App on a public cloud.
Lightning AI easily solves this by allowing Juan to add his own cluster, and then simply and securely run the data on his own infrastructure:
A Quick Wrap-up on Lightning AI
With Lightning AI, research and ML systems can be easily run locally, on the cloud, or on your own private infrastructure. Each user has their own portfolio of Lightning Apps and Components that can be shared publicly. This enables collaboration with other team or community members to improve and advance upon.
The open source Lightning Apps and Components provided on Lightning AI are meant to help you discover something new, find and offer inspiration, and build the next wave of AI-driven applications that you can share with the world or keep private for clients and customers.
