MMT SDK tutorial

MMT stands for multi-machine training. An MMT run launches the same workload across multiple machines and tracks the group as one managed resource. Use MMT when a training framework or distributed data job needs several workers instead of a single node.

An MMT run has the same runtime choices as a single-machine job:

• A Studio environment, which reuses code and dependencies from a Studio.
• A Docker image, which packages the runtime independently of Studios.

The number of machines must be greater than one.

Prerequisites

Install and authenticate:

$ pip install lightning-sdk -U
$ lightning login

For Studio-backed MMT runs, use an existing Studio in the target teamspace. The SDK example fetches that Studio by name with create_ok=False.

The SDK form lets Python own the launch, wait, and inspection flow:

teamspace = Teamspace(args.teamspace, org=args.org, user=args.user)
studio = Studio(args.studio, teamspace=teamspace, create_ok=False)

mmt = MMT.run(
    name=args.name,
    teamspace=teamspace,
    studio=studio,
    num_machines=args.num_machines,
    machine=Machine.from_str(args.machine),
    env={"RUN_MODE": "distributed"},
    command=args.command,
)

print(f"MMT link: {mmt.link}")
mmt.wait(interval=15, timeout=2 * 60 * 60, stop_on_timeout=True)
print(mmt.json())

if mmt.status == Status.Failed:
    for worker in mmt.machines:
        print(f"Worker {worker.name}: {worker.status}")

For a container runtime, replace studio= with image=:

teamspace = Teamspace(args.teamspace, org=args.org, user=args.user)

mmt = MMT.run(
    name=args.name,
    teamspace=teamspace,
    image=args.image,
    num_machines=args.num_machines,
    machine=Machine.from_str(args.machine),
    command=args.command,
    interruptible=True,
)

mmt.wait(interval=15)
print(f"{mmt.name} used {mmt.num_machines} machines")

Run the companion script directly when you want to execute the SDK example:

$ python python/examples/mmts.py --teamspace teamspace --org owner studio --studio sdk-tutorial-studio
$ python python/examples/mmts.py --teamspace teamspace --org owner image

Operational notes

• MMT.run creates a new multi-machine job; MMT("name", teamspace=...) fetches an existing one.
• mmt.machines returns the per-machine job handles for detailed inspection.
• Each machine writes its own artifacts; there is no combined folder for the run. mmt.download_artifacts gives each machine a folder named after it, and mmt.list_artifacts prefixes each path the same way. The jobs tutorial covers reading artifacts.
• MMT.run rejects num_machines values less than two.
• Studio-backed MMT runs must use a Studio in the same teamspace and cloud account as the run.
• The MMT object exposes high-level status and metadata; inspect the worker jobs when you need machine-level logs or failures.