---
title: Expert
description: >-
  Lightning enables experts focused on researching new ways of optimizing
  distributed training/inference strategies to create new strategies and plug
  them into Lightning.
canonical: 'https://lightning.ai/docs/pytorch/stable/reference/accelerators/gpu_expert'
---
# Expert {#expert}

## GPU training (Expert) {#gpu-training-expert}

**Audience:** Experts creating new scaling techniques such as [FSDP](/docs/pytorch/2.6.6/reference/advanced/model_parallel/fsdp.md) or [DeepSpeed](/docs/pytorch/2.6.6/reference/advanced/model_parallel/deepspeed.md).

> **Warning**
>
> This is an [experimental](/docs/pytorch/2.6.6/reference/versioning.md#experimental-api) feature.

Lightning enables experts focused on researching new ways of optimizing distributed training/inference strategies to create new strategies and plug them into Lightning.

For example, Lightning worked closely with the Microsoft team to develop a [DeepSpeed](/docs/pytorch/2.6.6/reference/advanced/model_parallel/deepspeed.md) integration and with the Facebook (Meta) team to develop a [FSDP](/docs/pytorch/2.6.6/reference/advanced/model_parallel/fsdp.md) integration.

### What is a Strategy? {#what-is-a-strategy}

Strategy controls the model distribution across training, evaluation, and prediction to be used by the [Trainer](/docs/pytorch/2.6.6/core-api/trainer.md). It can be controlled by passing different
strategy with aliases (`"ddp"`, `"ddp_spawn"`, `"deepspeed"` and so on) as well as a custom strategy to the `strategy` parameter for Trainer.

The Strategy in PyTorch Lightning handles the following responsibilities:

* Launch and teardown of training processes (if applicable).
* Setup communication between processes (NCCL, GLOO, MPI, and so on).
* Provide a unified communication interface for reduction, broadcast, and so on.
* Owns the `LightningModule`
* Handles/owns optimizers and schedulers.

Strategy is a composition of one [Accelerator](/docs/pytorch/2.6.6/reference/extensions/accelerator.md), one [Precision Plugin](/docs/pytorch/2.6.6/reference/extensions/plugins.md#precision-plugins), a [CheckpointIO](/docs/pytorch/2.6.6/reference/extensions/plugins.md#checkpointio-plugins)
plugin and other optional plugins such as the [ClusterEnvironment](/docs/pytorch/2.6.6/reference/extensions/plugins.md#cluster-environments).

![Illustration of the Strategy as a composition of the Accelerator and several plugins](https://pl-public-data.s3.amazonaws.com/docs/static/images/strategies/overview.jpeg)

We expose Strategies mainly for expert users that want to extend Lightning for new hardware support or new distributed backends (e.g. a backend not yet supported by [PyTorch](https://pytorch.org/docs/stable/distributed.html#backends) itself).

#### Selecting a Built-in Strategy {#selecting-a-built-in-strategy}

Built-in strategies can be selected in two ways.

1. Pass the shorthand name to the `strategy` Trainer argument
2. Import a Strategy from `lightning.pytorch.strategies`, instantiate it and pass it to the `strategy` Trainer argument

The latter allows you to configure further options on the specific strategy.
Here are some examples:

```python
# Training with the DistributedDataParallel strategy on 4 GPUs
trainer = Trainer(strategy="ddp", accelerator="gpu", devices=4)

# Training with the DistributedDataParallel strategy on 4 GPUs, with options configured
trainer = Trainer(strategy=DDPStrategy(static_graph=True), accelerator="gpu", devices=4)

# Training with the DDP Spawn strategy using auto accelerator selection
trainer = Trainer(strategy="ddp_spawn", accelerator="auto", devices=4)

# Training with the DeepSpeed strategy on available GPUs
trainer = Trainer(strategy="deepspeed", accelerator="gpu", devices="auto")

# Training with the DDP strategy using 3 CPU processes
trainer = Trainer(strategy="ddp", accelerator="cpu", devices=3)

# Training with the DDP Spawn strategy on 8 TPU cores
trainer = Trainer(strategy="ddp_spawn", accelerator="tpu", devices=8)
```

The below table lists all relevant strategies available in Lightning with their corresponding short-hand name:

| Name | Class | Description |
| --- | --- | --- |
| fsdp | `FSDPStrategy` | Strategy for Fully Sharded Data Parallel training. [Learn more.](/docs/pytorch/2.6.6/reference/advanced/model_parallel/fsdp.md) |
| ddp | `DDPStrategy` | Strategy for multi-process single-device training on one or multiple nodes. [Learn more.](/docs/pytorch/2.6.6/reference/accelerators/gpu_intermediate.md#distributed-data-parallel) |
| ddp_spawn | `DDPStrategy` | Same as "ddp" but launches processes using `torch.multiprocessing.spawn` method and joins processes after training finishes. [Learn more.](/docs/pytorch/2.6.6/reference/accelerators/gpu_intermediate.md#distributed-data-parallel-spawn) |
| deepspeed | `DeepSpeedStrategy` | Provides capabilities to run training using the DeepSpeed library, with training optimizations for large billion parameter models. [Learn more.](/docs/pytorch/2.6.6/reference/advanced/model_parallel/deepspeed.md) |
| xla | `XLAStrategy` | Strategy for training on multiple TPU devices using the `torch_xla.distributed.xla_multiprocessing.spawn` method. [Learn more.](/docs/pytorch/2.6.6/reference/accelerators/tpu.md) |
| single_xla | `SingleXLAStrategy` | Strategy for training on a single XLA device, like TPUs. [Learn more.](/docs/pytorch/2.6.6/reference/accelerators/tpu.md) |

#### Third-party Strategies {#third-party-strategies}

There are powerful third-party strategies that integrate well with Lightning but aren't maintained as part of the `lightning` package.
Checkout the gallery over [here](/docs/pytorch/2.6.6/reference/integrations/strategies.md).

#### Create a Custom Strategy {#create-a-custom-strategy}

Every strategy in Lightning is a subclass of one of the main base classes: `Strategy`, `SingleDeviceStrategy` or `ParallelStrategy`.

![Strategy base classes](https://pl-public-data.s3.amazonaws.com/docs/static/images/strategies/hierarchy.jpeg)

As an expert user, you may choose to extend either an existing built-in Strategy or create a completely new one by
subclassing the base classes.

```python
from lightning.pytorch.strategies import DDPStrategy

class CustomDDPStrategy(DDPStrategy):
    def configure_ddp(self):
        self.model = MyCustomDistributedDataParallel(
            self.model,
            device_ids=...,
        )

    def setup(self, trainer):
        # you can access the accelerator and plugins directly
        self.accelerator.setup()
        self.precision_plugin.connect(...)
```

The custom strategy can then be passed into the `Trainer` directly via the `strategy` parameter.

```python
# custom strategy
trainer = Trainer(strategy=CustomDDPStrategy())
```

Since the strategy also hosts the Accelerator and various plugins, you can customize all of them to work together as you like:

```python
# custom strategy, with new accelerator and plugins
accelerator = MyAccelerator()
precision_plugin = MyPrecisionPlugin()
strategy = CustomDDPStrategy(accelerator=accelerator, precision_plugin=precision_plugin)
trainer = Trainer(strategy=strategy)
```

## Strategy Registry {#strategy-registry}

Lightning includes a registry that holds information about Training strategies and allows for the registration of new custom strategies.

The Strategies are assigned strings that identify them, such as "ddp", "deepspeed_stage_2_offload", and so on.
It also returns the optional description and parameters for initialising the Strategy that were defined during registration.

```python
# Training with the DDP Strategy
trainer = Trainer(strategy="ddp", accelerator="gpu", devices=4)

# Training with DeepSpeed ZeRO Stage 3 and CPU Offload
trainer = Trainer(strategy="deepspeed_stage_3_offload", accelerator="gpu", devices=3)

# Training with the TPU Spawn Strategy with `debug` as True
trainer = Trainer(strategy="xla_debug", accelerator="tpu", devices=8)
```

Additionally, you can pass your custom registered training strategies to the `strategy` argument.

```python
from lightning.pytorch.strategies import DDPStrategy, StrategyRegistry, CheckpointIO

class CustomCheckpointIO(CheckpointIO):
    def save_checkpoint(self, checkpoint: Dict[str, Any], path: Union[str, Path]) -> None:
        ...

    def load_checkpoint(self, path: Union[str, Path]) -> Dict[str, Any]:
        ...

custom_checkpoint_io = CustomCheckpointIO()

# Register the DDP Strategy with your custom CheckpointIO plugin
StrategyRegistry.register(
    "ddp_custom_checkpoint_io",
    DDPStrategy,
    description="DDP Strategy with custom checkpoint io plugin",
    checkpoint_io=custom_checkpoint_io,
)

trainer = Trainer(strategy="ddp_custom_checkpoint_io", accelerator="gpu", devices=2)
```

---

**Navigation**

- Parent: [Accelerators](/docs/pytorch/2.6.6/reference/accelerators.md)