---
title: FSDPStrategy
description: Strategy for Fully Sharded Data Parallel provided by torch.distributed.
canonical: >-
  https://lightning.ai/docs/fabric/stable/full-api-reference/api/generated/lightning.fabric.strategies.FSDPStrategy
---
# FSDPStrategy {#fsdpstrategy}

### `class lightning.fabric.strategies.FSDPStrategy(accelerator=None, parallel_devices=None, cluster_environment=None, precision=None, process_group_backend=None, timeout=datetime.timedelta(seconds=1800), cpu_offload=None, mixed_precision=None, auto_wrap_policy=None, activation_checkpointing=None, activation_checkpointing_policy=None, sharding_strategy='FULL_SHARD', state_dict_type='sharded', device_mesh=None, **kwargs)` {#lightning.fabric.strategies.FSDPStrategy}

Bases: [`ParallelStrategy`](/docs/fabric/2.6.6/full-api-reference/api/generated/lightning.fabric.strategies.ParallelStrategy.md#lightning.fabric.strategies.ParallelStrategy "lightning.fabric.strategies.parallel.ParallelStrategy"), `_Sharded`

Strategy for Fully Sharded Data Parallel provided by torch.distributed.

Fully Sharded Training shards the entire model across all available GPUs, allowing you to scale model size, whilst using efficient communication to reduce overhead. In practice, this means we can remain at parity with PyTorch DDP, whilst scaling our model sizes dramatically. The technique is similar to ZeRO-Stage 3.

For more information check out [this blogpost](https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallel-api).

Defaults have been set and options have been exposed, but may require configuration based on your level of memory/speed efficiency. We suggest having a look at [this tutorial](https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html) for more information.

Parameters:

*   **cpu\_offload** (`Union`\[[`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)"), [`CPUOffload`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.CPUOffload "(in PyTorch v2.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – See `cpu_offload` parameter in [`torch.distributed.fsdp.FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)").
    
*   **mixed\_precision** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`MixedPrecision`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.MixedPrecision "(in PyTorch v2.15)")\]) – See `mixed_precision` parameter in [`torch.distributed.fsdp.FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)").
    
*   **auto\_wrap\_policy** (`Union`\[[`set`](https://docs.python.org/3/builtins/stdtypes.html#set "(in Python v3.15)")\[[`type`](https://docs.python.org/3/builtins/functions.html#type "(in Python v3.15)")\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")\]\], [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable "(in Python v3.15)")\[\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)"), [`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)"), [`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")\], [`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")\], `ModuleWrapPolicy`, [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Same as `auto_wrap_policy` parameter in [`torch.distributed.fsdp.FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)"). For convenience, this also accepts a set of the layer classes to wrap.
    
*   **activation\_checkpointing** (`Union`\[[`type`](https://docs.python.org/3/builtins/functions.html#type "(in Python v3.15)")\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")\], [`list`](https://docs.python.org/3/builtins/stdtypes.html#list "(in Python v3.15)")\[[`type`](https://docs.python.org/3/builtins/functions.html#type "(in Python v3.15)")\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")\]\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Deprecated. Use `activation_checkpointing_policy`.
    
*   **activation\_checkpointing\_policy** (`Union`\[[`set`](https://docs.python.org/3/builtins/stdtypes.html#set "(in Python v3.15)")\[[`type`](https://docs.python.org/3/builtins/functions.html#type "(in Python v3.15)")\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")\]\], [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable "(in Python v3.15)")\[\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)"), [`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)"), [`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")\], [`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")\], `ModuleWrapPolicy`, [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Same as `auto_wrap_policy` parameter in [`torch.distributed.fsdp.FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)") but used when selecting the modules for which you want to enable activation checkpointing. Enabling this can free up a significant amount of memory at the cost of speed since activations in these layers need to be recomputed during backpropagation. For convenience, this also accepts a set of the layer classes to wrap.
    
*   **sharding\_strategy** (`Union`\[[`ShardingStrategy`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.ShardingStrategy "(in PyTorch v2.15)"), [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'FULL_SHARD'`, `'SHARD_GRAD_OP'`, `'NO_SHARD'`, `'HYBRID_SHARD'`\]\]) –
    
    Select whether to shard model parameters, gradients, optimizer states, or a combination of them. Available values are:
    
    *   `"FULL_SHARD"`: Shards model parameters, gradients, and optimizer states (default).
        
    *   `"SHARD_GRAD_OP"`: Shards gradients and optimizer states only. Model parameters get replicated.
        
    *   `"NO_SHARD"`: No sharding (identical to regular DDP).
        
    *   `"HYBRID_SHARD"`: Shards model parameters, gradients, and optimizer states within a single machine, but replicates across machines. See also the device\_mesh parameter below.
        
    
    Also accepts a [`torch.distributed.fsdp.ShardingStrategy`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.ShardingStrategy "(in PyTorch v2.15)") enum value.
    
*   **device\_mesh** (`Union`\[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)"), [`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")\], [`DeviceMesh`](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.device_mesh.DeviceMesh "(in PyTorch v2.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – A tuple (replication size, sharding size) that defines over how many devices to shard and replicate the model. The product of the two numbers must equal the world size. Only valid in combination with the HYBRID\_SHARD sharding strategy.
    
*   **state\_dict\_type** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'full'`, `'sharded'`\]) –
    
    The format in which the state of the model and optimizers gets saved into the checkpoint.
    
    *   `"full"`: The full weights and optimizer states get assembled on rank 0 and saved to a single file.
        
    *   `"sharded"`: Each rank saves its shard of weights and optimizer states to a file. The checkpoint is a folder with as many files as the world size.
        
    
*   **\*\*kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – See available parameters in [`torch.distributed.fsdp.FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)").
    

#### `_configure_launcher()` {#lightning.fabric.strategies.FSDPStrategy._configure_launcher}

Attach the launcher based on Strategy.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `all_reduce(tensor, group=None, reduce_op='mean')` {#lightning.fabric.strategies.FSDPStrategy.all_reduce}

Reduces the given tensor (e.g. across GPUs/processes).

Parameters:

*   **tensor** ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")) – the tensor to sync and reduce
    
*   **group** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – the process group to reduce
    
*   **reduce\_op** (`Union`\[[`ReduceOp`](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.ReduceOp "(in PyTorch v2.15)"), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – the reduction operation. Defaults to ‘mean’. Can also be a string ‘sum’ or ReduceOp.
    

Return type:

[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")

#### `barrier(*args, **kwargs)` {#lightning.fabric.strategies.FSDPStrategy.barrier}

Synchronizes all processes which blocks processes until the whole group enters this function.

Parameters:

**name** – an optional name to pass into barrier.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `broadcast(obj, src=0)` {#lightning.fabric.strategies.FSDPStrategy.broadcast}

Broadcasts an object to all processes.

Parameters:

*   **obj** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar "(in Python v3.15)")(`TBroadcast`)) – the object to broadcast
    
*   **src** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – source rank
    

Return type:

[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar "(in Python v3.15)")(`TBroadcast`)

#### `clip_gradients_norm(module, optimizer, max_norm, norm_type=2.0, error_if_nonfinite=True)` {#lightning.fabric.strategies.FSDPStrategy.clip_gradients_norm}

Clip gradients by norm.

Return type:

[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")

#### `load_checkpoint(path, state=None, strict=True, weights_only=None)` {#lightning.fabric.strategies.FSDPStrategy.load_checkpoint}

Load the contents from a checkpoint and restore the state of the given objects.

Return type:

[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]

#### `module_init_context(empty_init=None)` {#lightning.fabric.strategies.FSDPStrategy.module_init_context}

A context manager wrapping the model instantiation.

Here, the strategy can control how the parameters of the model get created (device, dtype) and or apply other patches to the model.

Parameters:

**empty\_init** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")\]) – Whether to initialize the model with empty weights (uninitialized memory). If `None`, the strategy will decide. Some strategies may not support all options.

Return type:

[`AbstractContextManager`](https://docs.python.org/3/library/contextlib.html#contextlib.AbstractContextManager "(in Python v3.15)")

#### `module_sharded_context()` {#lightning.fabric.strategies.FSDPStrategy.module_sharded_context}

A context manager that goes over the instantiation of an [`torch.nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)") and handles sharding of parameters on creation.

By sharding layers directly on instantiation, one can reduce peak memory usage and initialization time.

Return type:

[`AbstractContextManager`](https://docs.python.org/3/library/contextlib.html#contextlib.AbstractContextManager "(in Python v3.15)")

#### `module_to_device(module)` {#lightning.fabric.strategies.FSDPStrategy.module_to_device}

Moves the model to the correct device.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `save_checkpoint(path, state, storage_options=None, filter=None)` {#lightning.fabric.strategies.FSDPStrategy.save_checkpoint}

Save model, optimizer, and other state to a checkpoint on disk.

If the state-dict-type is `'full'`, the checkpoint will be written to a single file containing the weights, optimizer state and other metadata. If the state-dict-type is `'sharded'`, the checkpoint gets saved as a directory containing one file per process, with model- and optimizer shards stored per file. Additionally, it creates a metadata file meta.pt with the rest of the user’s state (only saved from rank 0).

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `setup_environment()` {#lightning.fabric.strategies.FSDPStrategy.setup_environment}

Setup any processes or distributed connections.

This must be called by the framework at the beginning of every process, before any distributed communication takes place.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `setup_module(module)` {#lightning.fabric.strategies.FSDPStrategy.setup_module}

Wraps the model into a [`FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)") module.

Return type:

[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")

#### `setup_module_and_optimizers(module, optimizers, scheduler=None)` {#lightning.fabric.strategies.FSDPStrategy.setup_module_and_optimizers}

Wraps the model into a [`FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)") module and sets use\_orig\_params=True to keep the reference to the original parameters in the optimizer.

Return type:

[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)"), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list "(in Python v3.15)")\[[`Optimizer`](https://docs.pytorch.org/docs/stable/optim.html#torch.optim.Optimizer "(in PyTorch v2.15)")\], [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[`_LRScheduler`\]\]

#### `setup_optimizer(optimizer)` {#lightning.fabric.strategies.FSDPStrategy.setup_optimizer}

Set up an optimizer for a model wrapped with FSDP.

This setup method doesn’t modify the optimizer or wrap the optimizer. The only thing it currently does is verify that the optimizer was created after the model was wrapped with [`setup_module()`](#lightning.fabric.strategies.FSDPStrategy.setup_module "lightning.fabric.strategies.FSDPStrategy.setup_module") with a reference to the flattened parameters.

Return type:

[`Optimizer`](https://docs.pytorch.org/docs/stable/optim.html#torch.optim.Optimizer "(in PyTorch v2.15)")

#### `property distributed_sampler_kwargs: dict[str, Any]` {#lightning.fabric.strategies.FSDPStrategy.distributed_sampler_kwargs}

Arguments for the `DistributedSampler`.

If this method is not defined, or it returns `None`, then the `DistributedSampler` will not be used.

#### `property root_device: device` {#lightning.fabric.strategies.FSDPStrategy.root_device}

Returns the root device.

---

**Navigation**

- Parent: [Generated](/docs/fabric/2.6.6/full-api-reference/api/generated.md)