Shortcuts

ParallelStrategy

class pytorch_lightning.strategies.ParallelStrategy(accelerator=None, parallel_devices=None, cluster_environment=None, checkpoint_io=None, precision_plugin=None)[source]

Bases: pytorch_lightning.strategies.strategy.Strategy, abc.ABC

Plugin for training with multiple processes in parallel.

all_gather(tensor, group=None, sync_grads=False)[source]

Perform a all_gather on all processes.

Return type:

Tensor

block_backward_sync()[source]

Blocks ddp sync gradients behaviour on backwards pass.

This is useful for skipping sync when accumulating gradients, reducing communication overhead Returns: context manager with sync behaviour off

Return type:

Generator

reconciliate_processes(trace)[source]

Function to re-conciliate processes on failure.

Return type:

None

reduce_boolean_decision(decision)[source]

Reduce a boolean decision across all processes.

Return type:

bool

teardown()[source]

This method is called to teardown the training process.

It is the right place to release memory and free other resources.

Return type:

None

property is_global_zero: bool

Whether the current process is the rank zero process not only on the local node, but for all nodes.

abstract property root_device: torch.device

Return the root device.