Artifacts

In ML engineering, managing and accessing generated files, known as artifacts, is a critical task. Lightning AI simplifies this process by providing a unified filesystem across Studios, the teamspace drive, for easy artifact management.

What is an artifact

At its core, an artifact is a generated file crucial in AI development projects. Examples of artifacts include:

Simply put, an artifact is a file. In the context of AI development, an artifact can be:

• Model checkpoints
• Generated images or text
• Various script-generated files

Artifact sharing is at the core of the development and deployment process in AI projects.

Share artifacts between Studios

Within Lightning AI, all Studios share a filesystem: the teamspace drive. This setup facilitates the seamless transfer of artifacts between Studios. Let's say you finetune a model on one Studio and want to deploy that model on a different Studio. To do this, you'd need to figure out how to get the model checkpoint to the deployment Studio.

In Lightning AI, transferring artifacts is as simple as it is on your laptop. Simply copy the file over or symlink it.

Download artifacts

For those integrating Lightning Studios with external production pipelines or simply needing to transfer files out, the teamspace drive offers a straightforward download capability mimicking a typical desktop experience.

Automate file downloads with the SDK

# export LIGHTNING_USER_ID=000000-000000-000000-00000-000000000
# export LIGHTNING_API_KEY=111111-111111-111111-11111-111111111

from lightning_sdk import Studio

studio = Studio("studio-2", teamspace="research", org="lightning-ai")

studio.download_file("ptl_default.ckpt")

Export artifacts to S3

You might need artifacts in other S3 buckets that are not connected to Lightning AI. To move them there, use the AWS CLI:

View and explore artifacts

The teamspace drive provides a robust platform to explore artifacts. Drive is optimized to display massive, multi-terabyte datasets without problems. Drive outperforms VSCode when you need to explore large datasets.