ParallelStrategy¶

class pytorch_lightning.strategies.ParallelStrategy(accelerator=None, parallel_devices=None, cluster_environment=None, checkpoint_io=None, precision_plugin=None)[source]¶

Bases: pytorch_lightning.strategies.strategy.Strategy, abc.ABC

Plugin for training with multiple processes in parallel.

all_gather(tensor, group=None, sync_grads=False)[source]¶

Perform a all_gather on all processes.

Return type:: Tensor

block_backward_sync()[source]¶

Blocks ddp sync gradients behaviour on backwards pass.

This is useful for skipping sync when accumulating gradients, reducing communication overhead Returns: context manager with sync behaviour off

Return type:: Generator

reconciliate_processes(trace)[source]¶

Function to re-conciliate processes on failure.

Return type:: None

reduce_boolean_decision(decision)[source]¶

Reduce a boolean decision across all processes.

Return type:: bool

teardown()[source]¶

This method is called to teardown the training process.

It is the right place to release memory and free other resources.

Return type:: None

property is_global_zero: bool¶: Whether the current process is the rank zero process not only on the local node, but for all nodes.

abstract property root_device: torch.device¶: Return the root device.