---
title: DataParallelStrategy
description: >-
  Implements data-parallel training in a single process, i.e., the model gets
  replicated to each device and each

  gets a split of the data.
canonical: >-
  https://lightning.ai/docs/fabric/stable/full-api-reference/api/generated/lightning.fabric.strategies.DataParallelStrategy
---
# DataParallelStrategy {#dataparallelstrategy}

### `class lightning.fabric.strategies.DataParallelStrategy(accelerator=None, parallel_devices=None, checkpoint_io=None, precision=None)` {#lightning.fabric.strategies.DataParallelStrategy}

Bases: [`ParallelStrategy`](/docs/fabric/2.6.6/full-api-reference/api/generated/lightning.fabric.strategies.ParallelStrategy.md#lightning.fabric.strategies.ParallelStrategy "lightning.fabric.strategies.parallel.ParallelStrategy")

Implements data-parallel training in a single process, i.e., the model gets replicated to each device and each gets a split of the data.

#### `all_reduce(collection, group=None, reduce_op='mean')` {#lightning.fabric.strategies.DataParallelStrategy.all_reduce}

Reduces the given tensor (e.g. across GPUs/processes).

Parameters:

*   **tensor** – the tensor to sync and reduce
    
*   **group** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – the process group to reduce
    
*   **reduce\_op** (`Union`\[[`ReduceOp`](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.ReduceOp "(in PyTorch v2.15)"), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – the reduction operation. Defaults to ‘mean’. Can also be a string ‘sum’ or ReduceOp.
    

Return type:

[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar "(in Python v3.15)")(`TReduce`)

#### `barrier(*args, **kwargs)` {#lightning.fabric.strategies.DataParallelStrategy.barrier}

Synchronizes all processes which blocks processes until the whole group enters this function.

Parameters:

**name** – an optional name to pass into barrier.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `batch_to_device(batch, device=None)` {#lightning.fabric.strategies.DataParallelStrategy.batch_to_device}

Moves the batch to the correct device.

The returned batch is of the same type as the input batch, just having all tensors on the correct device.

Parameters:

*   **batch** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – The batch of samples to move to the correct device
    
*   **device** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`device`](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.15)")\]) – The target device
    

Return type:

[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")

#### `broadcast(obj, src=0)` {#lightning.fabric.strategies.DataParallelStrategy.broadcast}

Broadcasts an object to all processes.

Parameters:

*   **obj** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar "(in Python v3.15)")(`TBroadcast`)) – the object to broadcast
    
*   **src** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – source rank
    

Return type:

[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar "(in Python v3.15)")(`TBroadcast`)

#### `get_module_state_dict(module)` {#lightning.fabric.strategies.DataParallelStrategy.get_module_state_dict}

Returns model state.

Return type:

[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), `Union`\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)"), [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\]\]

#### `load_module_state_dict(module, state_dict, strict=True)` {#lightning.fabric.strategies.DataParallelStrategy.load_module_state_dict}

Loads the given state into the model.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `module_to_device(module)` {#lightning.fabric.strategies.DataParallelStrategy.module_to_device}

Moves the model to the correct device.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `reduce_boolean_decision(decision, all=True)` {#lightning.fabric.strategies.DataParallelStrategy.reduce_boolean_decision}

Reduces a boolean decision over distributed processes. By default is analogous to `all` from the standard library, returning `True` only if all input decisions evaluate to `True`. If `all` is set to `False`, it behaves like `any` instead.

Parameters:

*   **decision** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – A single input decision.
    
*   **all** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – Whether to logically emulate `all` or `any`. Defaults to True.
    

Returns:

The reduced boolean decision.

Return type:

[bool](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")

#### `setup_module(module)` {#lightning.fabric.strategies.DataParallelStrategy.setup_module}

Wraps the given model into a [`DataParallel`](https://docs.pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel "(in PyTorch v2.15)") module.

Return type:

[`DataParallel`](https://docs.pytorch.org/docs/stable/generated/torch.nn.DataParallel.html#torch.nn.DataParallel "(in PyTorch v2.15)")

#### `property distributed_sampler_kwargs: None` {#lightning.fabric.strategies.DataParallelStrategy.distributed_sampler_kwargs}

Arguments for the `DistributedSampler`.

If this method is not defined, or it returns `None`, then the `DistributedSampler` will not be used.

#### `property root_device: device` {#lightning.fabric.strategies.DataParallelStrategy.root_device}

Returns the root device.

---

**Navigation**

- Parent: [Generated](/docs/fabric/2.6.6/full-api-reference/api/generated.md)