Multi-Machine Training

class lightning_sdk.MMT(name, teamspace=None, org=None, user=None, *, _fetch_job=True, _num_machines=2)

Bases: Job

Compatibility interface for multi-machine jobs.

Multi-machine functionality is implemented by lightning_sdk.job.Job. MMT is deprecated; use Job instead.

Fetch already existing jobs.

Parameters:

name (str) – the name of the job.
teamspace (Union[str, Teamspace, None]) – the teamspace the job is part of.
org (Union[str, Organization, None]) – the name of the organization owning the teamspace in case it is owned by an org. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
user (Union[str, User, None]) – the name of the user owning the teamspace in case it is owned directly by a user instead of an org. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
name (str)
teamspace (str | Teamspace | None)
org (str | Organization | None)
user (str | User | None)
_fetch_job (bool)
_num_machines (int)

Raises:

ValueError – If the teamspace cannot be resolved from the provided arguments, or if the job is not found when _fetch_job=True.
PermissionError – If the user does not have access to jobs in the given teamspace.

classmethod run(name, num_machines, machine, cloud=None, command=None, studio=None, image=None, teamspace=None, org=None, user=None, env=None, interruptible=False, image_credentials=None, cloud_account_auth=False, entrypoint=None, path_mappings=None, max_runtime=None, max_run_attempts=None, reuse_snapshot=True, placement_group_id=None, tags=None, keep_minutes=None)

Run async workloads using a docker image or a compute environment from your studio.

Parameters:

name (str) – The name of the job. Needs to be unique within the teamspace.
machine (Union[Machine, str]) – The machine type to run the job on.
num_machines (int) – The number of machines to run on. Defaults to one.
command (Optional[str]) – The command to run inside your job. Required if using a studio. Optional if using an image. If not provided for images, will run the container entrypoint and default command. Studio jobs run the command as a bash script with set -e, so the first command that fails ends the job with its exit code. That includes cleanup commands and EXIT traps that run after exit 0.
studio (Union[Studio, str, None]) – The studio env to run the job with. Mutually exclusive with image. If both studio and image are left unset and this code is running inside a Studio (detected via the LIGHTNING_CLOUD_SPACE_ID env var), defaults to that Studio, provided its teamspace matches the resolved teamspace.
image (Optional[str]) – The docker image to run the job with. Mutually exclusive with studio.
teamspace (Union[str, Teamspace, None]) – The teamspace the job should be associated with. Defaults to the current teamspace. Accepts a bare name or an owner/teamspace slug.
org (Union[str, Organization, None]) – The organization owning the teamspace, if any. Defaults to the current organization. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
user (Union[str, User, None]) – The user owning the teamspace, if any. Defaults to the current user. Deprecated — pass the owner as part of teamspace instead, e.g. teamspace="owner/teamspace".
cloud (Union[CloudProvider, str, None]) – Cloud provider or cloud account to run the job on.
env (Optional[Dict[str, str]]) – Environment variables to set inside the job.
interruptible (bool) – Whether the job should run on interruptible instances. Cheaper but can be preempted.
image_credentials (Optional[str]) – Credentials secret name used to pull a private image.
cloud_account_auth (bool) – Whether to authenticate with the cloud account to pull the image. Required if the registry is part of a cloud provider, such as ECR.
entrypoint (Optional[str]) – The entrypoint of your docker container. Defaults to sh -c. Set to an empty string to use the image’s pre-defined entrypoint with a command. Only applicable when submitting docker jobs.
path_mappings (Optional[Dict[str, str]]) – Maps container paths to data-connection paths in the form {"<CONTAINER_PATH>": "<CONNECTION_NAME>:<PATH>"} or {"<CONTAINER_PATH>": "<CONNECTION_NAME>"} for the root of a connection. Only applicable when submitting docker jobs.
max_runtime (Optional[int]) – How long, in seconds, to reserve a DWS (Dynamic Workload Scheduler) machine, such as some top-end GCP GPUs or Lightning baremetal. On GCP the job is stopped when the reservation ends. On Lightning baremetal the job keeps running afterwards and the machine becomes interruptible. For a hard time limit, wrap your command, e.g. timeout 7200 python train.py. Has no effect on interruptible (spot) machines. None means no reservation is requested.
max_run_attempts (Optional[int]) – Max number of run attempts for this job. None or 0 means unset (backend default). 1 means a single attempt (no retries). N > 1 allows up to N attempts. For multi-machine jobs this is set at the multi-machine job level (not the per-machine JobSpec). When greater than 1 for a multi-machine job, the RECREATE_ALL_NODES fault tolerance strategy is set automatically on the multi-machine job body.
reuse_snapshot (bool) – Whether to reuse a Studio snapshot when multiple jobs for the same Studio are submitted. Turning this off may result in longer startup times. Defaults to True.
scratch_disks – Optional mapping of scratch-disk mount paths to their sizes in GiB.
placement_group_id (Optional[str]) – Optional placement group identifier for colocating the job.
tags (Optional[List[str]]) – Teamspace tag names to apply to the job. Tags that don’t exist in the teamspace yet are created, which requires permission to create tags. Names are normalised by the platform (lowercased, whitespace collapsed), so job.tags may read back slightly differently from what was passed in.
keep_minutes (Optional[int]) – Minutes to keep the machine after the job stops. None or 0 releases the machine immediately.
name (str)
num_machines (int)
machine (Machine | str)
cloud (CloudProvider | str | None)
command (str | None)
studio (Studio | str | None)
image (str | None)
teamspace (str | Teamspace | None)
org (str | Organization | None)
user (str | User | None)
env (Dict[str, str] | None)
interruptible (bool)
image_credentials (str | None)
cloud_account_auth (bool)
entrypoint (str | None)
path_mappings (Dict[str, str] | None)
max_runtime (int | None)
max_run_attempts (int | None)
reuse_snapshot (bool)
placement_group_id (str | None)
tags (List[str] | None)
keep_minutes (int | None)

Returns:

The newly submitted Job instance.

Return type:

Job

Raises:

ValueError – If required arguments are missing or mutually exclusive arguments are both provided.
RuntimeError – If image and studio are both provided.

class lightning_sdk.mmt.MMTMachine(*args, **kwargs)

Bases: Protocol

A single machine in a multi-machine job.

Was this page helpful?