---
title: thunder.distributed.fsdp
description: Convert model into Fully Sharded Data Parallel.
canonical: >-
  https://lightning.ai/docs/thunder/stable/api-reference/generated/thunder.distributed.fsdp
---
# thunder.distributed.fsdp {#thunder-distributed-fsdp}

### `thunder.distributed.fsdp(model, *, device=None, broadcast_from=None, sharding_strategy=FSDPType.ZERO2, bucketing_strategy=FSDPBucketingStrategy.NONE, move_state_dict_to_cpu=None)` {#thunder.distributed.fsdp}

[\[source\]](https://github.com/Lightning-AI/lightning-thunder/blob/0.2.6/thunder/distributed/__init__.py#L382-L458)

Convert `model` into Fully Sharded Data Parallel.

This splits `model`’s parameters in their first dimension into `world_size` chunks then has rank-`i` host `i`\-th chunks of them. This means the implementation is different from [`torch.distributed.fsdp.FullyShardedDataParallel`](https://docs.pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel "(in PyTorch v2.15)") which creates what’s called `torch.distributed.fsdp._flat_param.FlatParameter` as of [https://github.com/pytorch/pytorch/tree/647f14e7](https://github.com/pytorch/pytorch/tree/647f14e7). PyTorch however seems to be interested in per-parameter sharding as per [https://github.com/pytorch/pytorch/issues/114299](https://github.com/pytorch/pytorch/issues/114299).

To apply bucketing of collective communications, specify either [`LAYER`](/docs/thunder/0.2.6/api-reference/generated/thunder.distributed.FSDPBucketingStrategy.md#thunder.distributed.FSDPBucketingStrategy.LAYER "thunder.distributed.FSDPBucketingStrategy.LAYER") or `BucketingStrategy.BLOCK` as `bucketing_strategy`. The latter uses one collective communication, be it AllGather to unshard parameters or ReduceScatter to shard gradients, for one Transformer block. The former users one per layer such as [`torch.nn.Linear`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear "(in PyTorch v2.15)") and [`torch.nn.LayerNorm`](https://docs.pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html#torch.nn.LayerNorm "(in PyTorch v2.15)").

See /notebooks/dev\_tutorials/fsdp\_tutorial to see how parameters are sharded across devices and how communications calls are inserted.

Parameters:

*   **model** ([`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)") | [`ThunderModule`](/docs/thunder/thunder#thunder.ThunderModule "thunder.core.module.ThunderModule")) – The model to convert.
    
*   **model** ([_torch.nn.modules.module.Module_](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)") _|_ [_thunder.core.module.ThunderModule_](/docs/thunder/thunder#thunder.ThunderModule "thunder.core.module.ThunderModule")) –
    
*   **device** ([_Optional_](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")_\[_[_device_](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.15)")_\]_) –
    
*   **broadcast\_from** ([_Union_](https://docs.python.org/3/library/typing.html#typing.Union "(in Python v3.15)")_\[__None__,_ [_int_](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")_\]_) –
    
*   **sharding\_strategy** ([_FSDPType_](/docs/thunder/0.2.6/api-reference/generated/thunder.distributed.FSDPType.md#thunder.distributed.FSDPType "thunder.distributed.FSDPType")) –
    
*   **bucketing\_strategy** ([_FSDPBucketingStrategy_](/docs/thunder/0.2.6/api-reference/generated/thunder.distributed.FSDPBucketingStrategy.md#thunder.distributed.FSDPBucketingStrategy "thunder.distributed.FSDPBucketingStrategy")) –
    
*   **move\_state\_dict\_to\_cpu** ([_Optional_](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")_\[_[_bool_](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")_\]_) –
    

Keyword Arguments:

*   **device** – The corresponding model shard will be moved to this device. We recommend setting this to `torch.cuda.current_device()`.
    
*   **broadcast\_from** – The rank of the device hosting the parameters to broadcast. If None is passed, broadcasting will be skipped (default). Enabling can be useful for models whose weights have been loaded from a checkpoint in a single rank.
    
*   **sharding\_strategy** –
    
*   **bucketing\_strategy** –
    
*   **move\_state\_dict\_to\_cpu** –
    
    Move all-gather’ed parameters of `original_state_dict()` to CPU
    
    as each all-gather is finished.
    
    Returns:
    
    [`torch.nn.Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")
    

Return type:

[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)") | [`ThunderModule`](/docs/thunder/thunder#thunder.ThunderModule "thunder.core.module.ThunderModule")

---

**Navigation**

- Parent: [Generated](/docs/thunder/0.2.6/api-reference/generated.md)