ParallelStrategy¶

class lightning.pytorch.strategies.ParallelStrategy(accelerator=None, parallel_devices=None, cluster_environment=None, checkpoint_io=None, precision_plugin=None)[source]¶

Bases: lightning.pytorch.strategies.strategy.Strategy, abc.ABC

Plugin for training with multiple processes in parallel.

all_gather(tensor, group=None, sync_grads=False)[source]¶

Perform a all_gather on all processes.

Return type: Tensor

block_backward_sync()[source]¶

Blocks ddp sync gradients behaviour on backwards pass.

This is useful for skipping sync when accumulating gradients, reducing communication overhead Returns: context manager with sync behaviour off

Return type: Generator

reduce_boolean_decision(decision, all=True)[source]¶

Reduces a boolean decision over distributed processes. By default is analagous to all from the standard library, returning True only if all input decisions evaluate to True. If all is set to False, it behaves like any instead.

Parameters

decision¶ (bool) – A single input decision.
all¶ (bool) – Whether to logically emulate all or any. Defaults to True.

Returns

The reduced boolean decision.

Return type

bool

teardown()[source]¶

This method is called to teardown the training process.

It is the right place to release memory and free other resources.

Return type: None

property is_global_zero: bool¶

Whether the current process is the rank zero process not only on the local node, but for all nodes.

Return type: bool

abstract property root_device: torch.device¶

Return the root device.

Return type: device