Job

class lightning_sdk.Job(name, teamspace=None, org=None, user=None, *, _fetch_job=True, _num_machines=1)

Bases: object

Submit and manage jobs on the Lightning AI Platform.

Fetch already existing jobs.

Parameters:

name (str) – the name of the job.
teamspace (Union[str, Teamspace, None]) – the teamspace the job is part of.
org (Union[str, Organization, None]) – the name of the organization owning the teamspace in case it is owned by an org. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
user (Union[str, User, None]) – the name of the user owning the teamspace in case it is owned directly by a user instead of an org. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
name (str)
teamspace (str | Teamspace | None)
org (str | Organization | None)
user (str | User | None)
_fetch_job (bool)
_num_machines (int)

Raises:

ValueError – If the teamspace cannot be resolved from the provided arguments, or if the job is not found when _fetch_job=True.
PermissionError – If the user does not have access to jobs in the given teamspace.

classmethod run(name, machine, cloud=None, command=None, studio=None, image=None, teamspace=None, org=None, user=None, env=None, interruptible=False, image_credentials=None, cloud_account_auth=False, entrypoint=None, path_mappings=None, max_runtime=None, max_run_attempts=None, reuse_snapshot=True, scratch_disks=None, placement_group_id=None, num_machines=1, tags=None, keep_minutes=None)

Run async workloads using a docker image or a compute environment from your studio.

Parameters:

name (str) – The name of the job. Needs to be unique within the teamspace.
machine (Union[Machine, str]) – The machine type to run the job on.
num_machines (int) – The number of machines to run on. Defaults to one.
command (Optional[str]) – The command to run inside your job. Required if using a studio. Optional if using an image. If not provided for images, will run the container entrypoint and default command. Studio jobs run the command as a bash script with set -e, so the first command that fails ends the job with its exit code. That includes cleanup commands and EXIT traps that run after exit 0.
studio (Union[Studio, str, None]) – The studio env to run the job with. Mutually exclusive with image. If both studio and image are left unset and this code is running inside a Studio (detected via the LIGHTNING_CLOUD_SPACE_ID env var), defaults to that Studio, provided its teamspace matches the resolved teamspace.
image (Optional[str]) – The docker image to run the job with. Mutually exclusive with studio.
teamspace (Union[str, Teamspace, None]) – The teamspace the job should be associated with. Defaults to the current teamspace. Accepts a bare name or an owner/teamspace slug.
org (Union[str, Organization, None]) – The organization owning the teamspace, if any. Defaults to the current organization. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
user (Union[str, User, None]) – The user owning the teamspace, if any. Defaults to the current user. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
cloud (Union[CloudProvider, str, None]) – Cloud provider or cloud account to run the job on.
env (Optional[Dict[str, str]]) – Environment variables to set inside the job.
interruptible (bool) – Whether the job should run on interruptible instances. Cheaper but can be preempted.
image_credentials (Optional[str]) – Credentials secret name used to pull a private image.
cloud_account_auth (bool) – Whether to authenticate with the cloud account to pull the image. Required if the registry is part of a cloud provider, such as ECR.
entrypoint (Optional[str]) – The entrypoint of your docker container. Defaults to sh -c. Set to an empty string to use the image’s pre-defined entrypoint with a command. Only applicable when submitting docker jobs.
path_mappings (Optional[Dict[str, str]]) – Maps container paths to data-connection paths in the form {"<CONTAINER_PATH>": "<CONNECTION_NAME>:<PATH>"} or {"<CONTAINER_PATH>": "<CONNECTION_NAME>"} for the root of a connection. Only applicable when submitting docker jobs.
max_runtime (Optional[int]) – How long, in seconds, to reserve a DWS (Dynamic Workload Scheduler) machine, such as some top-end GCP GPUs or Lightning baremetal. On GCP the job is stopped when the reservation ends. On Lightning baremetal the job keeps running afterwards and the machine becomes interruptible. For a hard time limit, wrap your command, e.g. timeout 7200 python train.py. Has no effect on interruptible (spot) machines. None means no reservation is requested.
max_run_attempts (Optional[int]) – Max number of run attempts for this job. None or 0 means unset (backend default). 1 means a single attempt (no retries). N > 1 allows up to N attempts. For multi-machine jobs this is set at the multi-machine job level (not the per-machine JobSpec). When greater than 1 for a multi-machine job, the RECREATE_ALL_NODES fault tolerance strategy is set automatically on the multi-machine job body.
reuse_snapshot (bool) – Whether to reuse a Studio snapshot when multiple jobs for the same Studio are submitted. Turning this off may result in longer startup times. Defaults to True.
scratch_disks (Optional[Dict[str, int]]) – Optional mapping of scratch-disk mount paths to their sizes in GiB.
placement_group_id (Optional[str]) – Optional placement group identifier for colocating the job.
tags (Optional[List[str]]) – Teamspace tag names to apply to the job. Tags that don’t exist in the teamspace yet are created, which requires permission to create tags. Names are normalised by the platform (lowercased, whitespace collapsed), so job.tags may read back slightly differently from what was passed in.
keep_minutes (Optional[int]) – Minutes to keep the machine after the job stops. None or 0 releases the machine immediately.
name (str)
machine (Machine | str)
cloud (CloudProvider | str | None)
command (str | None)
studio (Studio | str | None)
image (str | None)
teamspace (str | Teamspace | None)
org (str | Organization | None)
user (str | User | None)
env (Dict[str, str] | None)
interruptible (bool)
image_credentials (str | None)
cloud_account_auth (bool)
entrypoint (str | None)
path_mappings (Dict[str, str] | None)
max_runtime (int | None)
max_run_attempts (int | None)
reuse_snapshot (bool)
scratch_disks (Dict[str, int] | None)
placement_group_id (str | None)
num_machines (int)
tags (List[str] | None)
keep_minutes (int | None)

Returns:

The newly submitted Job instance.

Return type:

Job

Raises:

ValueError – If required arguments are missing or mutually exclusive arguments are both provided.
RuntimeError – If image and studio are both provided.

download_artifacts(target_dir='.', path='')

Download what this job wrote to a local directory.

Parameters:

target_dir (Union[str, Path]) – Local directory to download into. Created if it does not exist.
path (str) – Subfolder of the job’s artifacts to download. Defaults to all of them.
target_dir (str | Path)
path (str)

Raises:

RuntimeError – If the job keeps no artifacts. A container job only keeps them when it was launched with artifacts_destination.

Return type:

None

download_logs(*, timestamps=False)

Download the job’s stored logs to a file and return the written path.

Reads every page of the job’s history from the logs API. A multi-machine job downloads the merged timeline of all its machines, each line prefixed with its machine name. A finished job whose lines are not in the logs API yet raises RuntimeError, as logs does.

Return type:: str
Parameters:: timestamps (bool)

list_artifacts(path='', recursive=False)

List what this job wrote to the teamspace drive.

Parameters:

path (str) – Subfolder of the job’s artifacts to list. Defaults to all of them.
recursive (bool) – When True, descend into subfolders.
path (str)
recursive (bool)

Returns:

One entry per file or folder, empty when the job keeps no artifacts. For a multi-machine job every machine is listed, and each path is prefixed with the machine’s name.

Return type:

List[FileEntry]

rename(new_name)

Rename this job.

Only standalone jobs (not launched from a deployment, MMT, or pipeline) can be renamed. An empty name or a name that matches the current name raises a ValueError.

Parameters:

new_name (str) – The new name for the job.
new_name (str)

Raises:

ValueError – If new_name is empty or matches the current name.

Return type:

None

set_tags(tags)

Replace this job’s tag set with the given tag names.

Tags that don’t exist in the teamspace yet are created, which requires permission to create tags. Pass an empty list to remove all tags.

Parameters:

tags (List[str]) – The list of tag names to set on the job.
tags (List[str])

Return type:

None

property artifacts_uri: str | None

The lit:// address of this job’s artifacts, for lightning ls and lightning cp.

None when the job keeps no artifacts, which is the case for a container job launched without artifacts_destination, and for a multi-machine job, whose machines each have their own — iterate over job.machines.

property current_run_attempt: int | None

Current run attempt for this job, or None if unset.

For multi-machine jobs this reads the multi-machine job body; for standalone jobs it reads the per-job spec.

property id: str | None: The job’s unique identifier.

property is_multi_machine: bool: Whether this object represents a multi-machine parent job.

property keep_minutes: int | None: Minutes the machine is kept after the job stops, or None if unset.

property logs: _Logs

The job’s logs.

Use it as a value for a snapshot of the logs up to now:

print(job.logs)

or call it to pass options and/or follow the logs live:

recent = job.logs(tail=100)            # snapshot of the last 100 lines
for line in job.logs(follow=True):     # stream new lines as they arrive
    print(line)

Options:

follow: Keep the stream open and yield new lines as they are produced. Returns an iterator of lines instead of a string.
tail: Only include the last N lines.
rank: Machine (0-based) to read in a multi-machine job. The logs API has no rank parameter, so the machine is resolved client-side to its own job and read from the same logs API; equivalent to job.machines[rank].logs.
timestamps: Prepend each line with its ISO-8601 timestamp.
since/until: Only include lines within this RFC3339 time range.
query: Only include lines containing every whitespace-separated term.
severity: Only include lines at or above this level (error, warning, info or debug).

since, until, query and severity are applied by the server, to both the saved logs and the live stream.

A finished job whose lines are not in the logs API yet is still being migrated from older storage, and raises RuntimeError; reading it again shortly after succeeds. A running one with no history in the logs API has its live stream tailed briefly instead.

property machines: Tuple[Job, ...]: The rank-ordered machines in this job.

property max_run_attempts: int | None

Max number of run attempts for this job, or None if unset.

For multi-machine jobs this reads the multi-machine job body; for standalone jobs it reads the per-job spec.

property num_machines: int: The number of machines allocated to this job.

property started_at: datetime | None: When the job started running, or None if it hasn’t started yet.

property stopped_at: datetime | None: When the job stopped running, or None if it hasn’t stopped yet.

property studio_name: str | None: The name of the studio this job runs in, without instantiating the Studio.

property tags: Tuple[str, ...]: The teamspace tags applied to this job, in the order the platform returns them.

Was this page helpful?