Multi-Machine Training

Multi-node distributed training.

Usage:

lightning mmt [OPTIONS] COMMAND [ARGS]...

Commands

Command

Description

delete

Delete a multi-machine job.

inspect

Inspect a multi-machine job for further details as JSON.

list

List multi-machine jobs for a given teamspace.

logs

View the logs for a multi-machine job.

run

Run async workloads on multiple machines using a docker image.

ssh

SSH into a running multi-machine job.

stop

Stop a multi-machine job.

lightning mmt delete

Delete a multi-machine job.

Usage:

lightning mmt delete [OPTIONS] NAME

Arguments

Name

Type

Required

Description

NAME

text

yes

Multi-machine training run name to delete.

Options

Option

Type

Required

Default

Description

--teamspace

TEXT

no

Sentinel.UNSET

Override default teamspace (format: owner/teamspace).

--yes, -y

—

no

—

Delete without prompting for confirmation.

lightning mmt inspect

Inspect a multi-machine job for further details as JSON.

Usage:

lightning mmt inspect [OPTIONS]

Options

Option

Type

Required

Default

Description

--name

TEXT

no

—

The multi-machine job name. Required.

--teamspace

TEXT

no

—

the name of the teamspace the job lives in. Should be specified as {teamspace_owner}/{teamspace_name} (e.g my-org/my-teamspace). If not specified, uses the configured default teamspace.

--json

—

no

—

Output as JSON (inspect always emits JSON).

lightning mmt list

List multi-machine jobs for a given teamspace.

Usage:

lightning mmt list [OPTIONS]

Options

Option

Type

Required

Default

Description

--teamspace

TEXT

no

—

the teamspace to list multi-machine jobs from. Should be specified as {owner}/{name}. Defaults to the configured teamspace.

--all

—

no

—

if teamspace is not provided, list all multi-machine jobs in all teamspaces.

--sort-by, --sort_by

[name|teamspace|studio|image|status|machine|cloud-account]

no

—

the attribute to sort the multi-machine jobs by.

--json

—

no

—

Output as JSON.

lightning mmt logs

View the logs for a multi-machine job.

Usage:

lightning mmt logs [OPTIONS] [COMMAND] [ARGS]...

Commands

Command

Description

download

Download the complete logs for a multi-machine job.

Options

Option

Type

Required

Default

Description

--teamspace

TEXT

no

—

Teamspace owner/name. Uses the configured default teamspace when omitted.

--follow, -f

—

no

—

Stream new log lines as they are produced.

--tail

INTEGER

no

—

Only show the last N lines.

--timestamps

—

no

—

Prepend each line with its ISO-8601 timestamp.

--since

TEXT

no

—

Only include lines at or after this time (e.g. “2h”, RFC3339).

--until

TEXT

no

—

Only include lines at or before this time (e.g. “30m”, RFC3339).

--query

TEXT

no

—

Only include lines containing every whitespace-separated term.

--severity

[error|warning|info|debug]

no

—

Only include lines at or above this severity.

--json

—

no

—

Output entries as a JSON array.

--interactive, -i

—

no

—

Launch the interactive TUI log viewer.

lightning mmt logs download

Download the complete logs for a multi-machine job.

Usage:

lightning mmt logs download [OPTIONS] [NAME]

Arguments

Name

Type

Required

Description

NAME

text

no

The multi-machine job name. Required.

Options

Option

Type

Required

Default

Description

--teamspace

TEXT

no

—

Teamspace owner/name. Uses the configured default teamspace when omitted.

--timestamps

—

no

—

Prepend each line with its ISO-8601 timestamp.

lightning mmt run

Run async workloads on multiple machines using a docker image.

Usage:

lightning mmt run [OPTIONS]

Options

Option

Type

Required

Default

Description

--name

TEXT

no

—

The name of the job. Needs to be unique within the teamspace.

--num-machines, --num_machines

INTEGER

no

2

The number of Machines to run on.

--machine

[CPU_SMALL|CPU|CPU_X_2|CPU_X_4|CPU_X_8|CPU_X_16|DATA_PREP|DATA_PREP_MAX|DATA_PREP_ULTRA|T4_SMALL|T4|T4_X_2|T4_X_4|T4_X_8|L4|L4_X_2|L4_X_4|L4_X_8|L40S|L40S_X_2|L40S_X_4|L40S_X_8|RTXP_6000|RTXP_6000_X_2|RTXP_6000_X_4|RTXP_6000_X_8|A100|A100_X_2|A100_X_4|A100_X_8|H100|H100_X_2|H100_X_4|H100_X_8|H200|H200_X_2|H200_X_4|H200_X_8|B200|B200_X_8]

no

'CPU'

The machine type to run the job on.

--command

TEXT

no

—

The command to run inside your job. Required if using a studio. Optional if using an image. If not provided for images, will run the container entrypoint and default command.

--studio

TEXT

no

—

The studio env to run the multi-machine job with. Mutually exclusive with image. If both –studio and –image are omitted while running inside a Studio, defaults to that Studio (if its teamspace matches –teamspace).

--image

TEXT

no

—

The docker image to run the multi-machine job with. Mutually exclusive with studio.

--teamspace

TEXT

no

—

The teamspace to use, as a bare name or an ‘owner/teamspace’ slug. Defaults to the current teamspace.

--org

TEXT

no

—

The organization owning the teamspace (if any). Defaults to the current organization.(DEPRECATED: Pass the owner as part of –teamspace instead, e.g. ‘–teamspace owner/teamspace’. This option will be removed in a future release.)

--user

TEXT

no

—

The user owning the teamspace (if any). Defaults to the current user.(DEPRECATED: Pass the owner as part of –teamspace instead, e.g. ‘–teamspace owner/teamspace’. This option will be removed in a future release.)

--cloud

TEXT

no

—

Cloud provider or cloud account to run the job on. Defaults to the studio cloud account or teamspace default.

--env, -e

TEXT

no

['']

Environment variable to set inside the job. Should be of format KEY=VALUE

--interruptible

—

no

—

Whether the job should run on interruptible instances. They are cheaper but can be preempted.

--image-credentials, --image_credentials

TEXT

no

—

The credentials used to pull the image. Required if the image is private. This should be the name of the respective credentials secret created on the Lightning AI platform.

--cloud-account-auth, --cloud_account_auth

—

no

—

Whether to authenticate with the cloud account to pull the image. Required if the registry is part of a cloud provider (e.g. ECR).

--entrypoint

TEXT

no

—

The entrypoint of your docker container. Default runs the provided command in a standard shell. To use the pre-defined entrypoint of the provided image, set this to an empty string. Only applicable when submitting docker jobs.

--path-mapping, --path_mapping

TEXT

no

['']

Maps path inside of containers to paths inside data-connections. Should be of form <CONTAINER_PATH_1>:<CONNECTION_NAME_1>:<PATH_WITHIN_CONNECTION_1> and omitting the path inside the connection defaults to the connections root. Can be specified multiple times for multiple mappings

--path-mappings, --path_mappings

TEXT

no

''

Maps path inside of containers to paths inside data-connections. Should be a comma separated list of form: <MAPPING_1>,<MAPPING_2>,… where each mapping is of the form <CONTAINER_PATH_1>:<CONNECTION_NAME_1>:<PATH_WITHIN_CONNECTION_1> and omitting the path inside the connection defaults to the connections root. Instead of a comma-separated list, consider passing –path-mapping multiple times.

--tags, --tag

TEXT

no

()

Tag to apply to the job. Can be a comma-separated list or passed multiple times. Tags that don’t exist in the teamspace yet are created.

--json

—

no

—

Output the created job as JSON.

--keep-minutes, --keep_minutes

INTEGER RANGE

no

—

Minutes to keep the machines after the job stops. Omit or pass 0 to release the machines immediately.

lightning mmt ssh

SSH into a running multi-machine job.

Usage:

lightning mmt ssh [OPTIONS] NAME

Arguments

Name

Type

Required

Description

NAME

text

yes

No description provided.

Options

Option

Type

Required

Default

Description

--teamspace

TEXT

no

—

the name of the teamspace the multi-machine job lives in. Should be specified as {teamspace_owner}/{teamspace_name} (e.g my-org/my-teamspace). If not specified, uses the configured default teamspace.

--rank

INTEGER

no

0

Machine rank to SSH into. Defaults to 0.

Examples

  lightning mmt ssh my-distributed-job
  lightning mmt ssh my-distributed-job --rank 1

lightning mmt stop

Stop a multi-machine job.

Usage:

lightning mmt stop [OPTIONS] NAME

Arguments

Name

Type

Required

Description

NAME

text

yes

Multi-machine training run name to stop.

Options

Option

Type

Required

Default

Description

--teamspace

TEXT

no

—

the name of the teamspace the multi-machine job lives in. Should be specified as {teamspace_owner}/{teamspace_name} (e.g my-org/my-teamspace). If not specified, uses the configured default teamspace.

--json

—

no

—

Output as JSON.

Was this page helpful?

What went wrong?

↑↓ Navigate↵ Open⌘↵ New tabesc Close