Multi-Machine Training
Multi-node distributed training.
Usage:
lightning mmt [OPTIONS] COMMAND [ARGS]...Commands
Command |
Description |
|---|---|
Delete a multi-machine job. |
|
Inspect a multi-machine job for further details as JSON. |
|
List multi-machine jobs for a given teamspace. |
|
View the logs for a multi-machine job. |
|
Run async workloads on multiple machines using a docker image. |
|
SSH into a running multi-machine job. |
|
Stop a multi-machine job. |
lightning mmt delete
Delete a multi-machine job.
Usage:
lightning mmt delete [OPTIONS] NAMEArguments
Name |
Type |
Required |
Description |
|---|---|---|---|
|
|
yes |
Multi-machine training run name to delete. |
Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
|
Override default teamspace (format: owner/teamspace). |
|
— |
no |
— |
Delete without prompting for confirmation. |
lightning mmt inspect
Inspect a multi-machine job for further details as JSON.
Usage:
lightning mmt inspect [OPTIONS]Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
— |
The multi-machine job name. Required. |
|
|
no |
— |
the name of the teamspace the job lives in. Should be specified as {teamspace_owner}/{teamspace_name} (e.g my-org/my-teamspace). If not specified, uses the configured default teamspace. |
|
— |
no |
— |
Output as JSON (inspect always emits JSON). |
lightning mmt list
List multi-machine jobs for a given teamspace.
Usage:
lightning mmt list [OPTIONS]Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
— |
the teamspace to list multi-machine jobs from. Should be specified as {owner}/{name}. Defaults to the configured teamspace. |
|
— |
no |
— |
if teamspace is not provided, list all multi-machine jobs in all teamspaces. |
|
|
no |
— |
the attribute to sort the multi-machine jobs by. |
|
— |
no |
— |
Output as JSON. |
lightning mmt logs
View the logs for a multi-machine job.
Usage:
lightning mmt logs [OPTIONS] [COMMAND] [ARGS]...Commands
Command |
Description |
|---|---|
Download the complete logs for a multi-machine job. |
Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
— |
Teamspace owner/name. Uses the configured default teamspace when omitted. |
|
— |
no |
— |
Stream new log lines as they are produced. |
|
|
no |
— |
Only show the last N lines. |
|
— |
no |
— |
Prepend each line with its ISO-8601 timestamp. |
|
|
no |
— |
Only include lines at or after this time (e.g. “2h”, RFC3339). |
|
|
no |
— |
Only include lines at or before this time (e.g. “30m”, RFC3339). |
|
|
no |
— |
Only include lines containing every whitespace-separated term. |
|
|
no |
— |
Only include lines at or above this severity. |
|
— |
no |
— |
Output entries as a JSON array. |
|
— |
no |
— |
Launch the interactive TUI log viewer. |
lightning mmt logs download
Download the complete logs for a multi-machine job.
Usage:
lightning mmt logs download [OPTIONS] [NAME]Arguments
Name |
Type |
Required |
Description |
|---|---|---|---|
|
|
no |
The multi-machine job name. Required. |
Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
— |
Teamspace owner/name. Uses the configured default teamspace when omitted. |
|
— |
no |
— |
Prepend each line with its ISO-8601 timestamp. |
lightning mmt run
Run async workloads on multiple machines using a docker image.
Usage:
lightning mmt run [OPTIONS]Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
— |
The name of the job. Needs to be unique within the teamspace. |
|
|
no |
|
The number of Machines to run on. |
|
|
no |
|
The machine type to run the job on. |
|
|
no |
— |
The command to run inside your job. Required if using a studio. Optional if using an image. If not provided for images, will run the container entrypoint and default command. |
|
|
no |
— |
The studio env to run the multi-machine job with. Mutually exclusive with image. If both –studio and –image are omitted while running inside a Studio, defaults to that Studio (if its teamspace matches –teamspace). |
|
|
no |
— |
The docker image to run the multi-machine job with. Mutually exclusive with studio. |
|
|
no |
— |
The teamspace to use, as a bare name or an ‘owner/teamspace’ slug. Defaults to the current teamspace. |
|
|
no |
— |
The organization owning the teamspace (if any). Defaults to the current organization.(DEPRECATED: Pass the owner as part of –teamspace instead, e.g. ‘–teamspace owner/teamspace’. This option will be removed in a future release.) |
|
|
no |
— |
The user owning the teamspace (if any). Defaults to the current user.(DEPRECATED: Pass the owner as part of –teamspace instead, e.g. ‘–teamspace owner/teamspace’. This option will be removed in a future release.) |
|
|
no |
— |
Cloud provider or cloud account to run the job on. Defaults to the studio cloud account or teamspace default. |
|
|
no |
|
Environment variable to set inside the job. Should be of format KEY=VALUE |
|
— |
no |
— |
Whether the job should run on interruptible instances. They are cheaper but can be preempted. |
|
|
no |
— |
The credentials used to pull the image. Required if the image is private. This should be the name of the respective credentials secret created on the Lightning AI platform. |
|
— |
no |
— |
Whether to authenticate with the cloud account to pull the image. Required if the registry is part of a cloud provider (e.g. ECR). |
|
|
no |
— |
The entrypoint of your docker container. Default runs the provided command in a standard shell. To use the pre-defined entrypoint of the provided image, set this to an empty string. Only applicable when submitting docker jobs. |
|
|
no |
|
Maps path inside of containers to paths inside data-connections. Should be of form <CONTAINER_PATH_1>:<CONNECTION_NAME_1>:<PATH_WITHIN_CONNECTION_1> and omitting the path inside the connection defaults to the connections root. Can be specified multiple times for multiple mappings |
|
|
no |
|
Maps path inside of containers to paths inside data-connections. Should be a comma separated list of form: <MAPPING_1>,<MAPPING_2>,… where each mapping is of the form <CONTAINER_PATH_1>:<CONNECTION_NAME_1>:<PATH_WITHIN_CONNECTION_1> and omitting the path inside the connection defaults to the connections root. Instead of a comma-separated list, consider passing –path-mapping multiple times. |
|
|
no |
|
Tag to apply to the job. Can be a comma-separated list or passed multiple times. Tags that don’t exist in the teamspace yet are created. |
|
— |
no |
— |
Output the created job as JSON. |
|
|
no |
— |
Minutes to keep the machines after the job stops. Omit or pass 0 to release the machines immediately. |
lightning mmt ssh
SSH into a running multi-machine job.
Usage:
lightning mmt ssh [OPTIONS] NAMEArguments
Name |
Type |
Required |
Description |
|---|---|---|---|
|
|
yes |
No description provided. |
Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
— |
the name of the teamspace the multi-machine job lives in. Should be specified as {teamspace_owner}/{teamspace_name} (e.g my-org/my-teamspace). If not specified, uses the configured default teamspace. |
|
|
no |
|
Machine rank to SSH into. Defaults to 0. |
Examples
lightning mmt ssh my-distributed-job
lightning mmt ssh my-distributed-job --rank 1lightning mmt stop
Stop a multi-machine job.
Usage:
lightning mmt stop [OPTIONS] NAMEArguments
Name |
Type |
Required |
Description |
|---|---|---|---|
|
|
yes |
Multi-machine training run name to stop. |
Options
Option |
Type |
Required |
Default |
Description |
|---|---|---|---|---|
|
|
no |
— |
the name of the teamspace the multi-machine job lives in. Should be specified as {teamspace_owner}/{teamspace_name} (e.g my-org/my-teamspace). If not specified, uses the configured default teamspace. |
|
— |
no |
— |
Output as JSON. |
Was this page helpful?
Thanks for the feedback.