DataParallelStrategy¶

class pytorch_lightning.strategies.DataParallelStrategy(accelerator=None, parallel_devices=None, checkpoint_io=None, precision_plugin=None)[source]¶

Bases: pytorch_lightning.strategies.parallel.ParallelStrategy

Implements data-parallel training in a single process, i.e., the model gets replicated to each device and each gets a split of the data.

barrier(*args, **kwargs)[source]¶

Synchronizes all processes which blocks processes until the whole group enters this function.

Parameters: name¶ – an optional name to pass into barrier.
Return type: None

batch_to_device(batch, device=None, dataloader_idx=0)[source]¶

Moves the batch to the correct device.

The input and the output is the same type.

Parameters

batch¶ (Any) – The batch of samples to move to the correct device
device¶ (Optional[device]) – The target device
dataloader_idx¶ (int) – The index of the dataloader to which the batch belongs.

Return type

Any

broadcast(obj, src=0)[source]¶

Broadcasts an object to all processes.

Parameters

obj¶ (TypeVar(TBroadcast)) – the object to broadcast
src¶ (int) – source rank

Return type

TypeVar(TBroadcast)

model_to_device()[source]¶

Moves the model to the correct device.

Return type: None

predict_step(*args, **kwargs)[source]¶

The actual predict step.

See predict_step() for more details

Return type: Union[Tensor, Dict[str, Any]]

reduce(collection, group=None, reduce_op='mean')[source]¶

Reduces a collection of tensors from all processes. It can be applied to just a single tensor.

Parameters

collection¶ (TypeVar(TReduce)) – The collection of tensors to sync and reduce.
group¶ (Optional[Any]) – ignored for DP
reduce_op¶ (Union[ReduceOp, str, None]) – ignored for DP

Return type

TypeVar(TReduce)

Returns

Reduced tensor values or the same value if it was not or did not contain a tensor.

reduce_boolean_decision(decision, all=True)[source]¶

Reduces a boolean decision over distributed processes. By default is analagous to all from the standard library, returning True only if all input decisions evaluate to True. If all is set to False, it behaves like any instead.

Parameters

decision¶ (bool) – A single input decision.
all¶ (bool) – Whether to logically emulate all or any. Defaults to True.

Returns

The reduced boolean decision.

Return type

bool

setup(trainer)[source]¶

Setup plugins for the trainer fit and creates optimizers.

Parameters: trainer¶ (Trainer) – the trainer instance
Return type: None

test_step(*args, **kwargs)[source]¶

The actual test step.

See test_step() for more details

Return type: Union[Tensor, Dict[str, Any], None]

training_step(*args, **kwargs)[source]¶

The actual training step.

See training_step() for more details

Return type: Union[Tensor, Dict[str, Any]]

validation_step(*args, **kwargs)[source]¶

The actual validation step.

See validation_step() for more details

Return type: Union[Tensor, Dict[str, Any], None]

property root_device: torch.device¶

Return the root device.

Return type: device