---
title: Strategy
description: >-
  Base class for all strategies that change the behaviour of the training,
  validation and test- loop.
canonical: >-
  https://lightning.ai/docs/pytorch/stable/optional-api/api/lightning.pytorch.strategies.Strategy
---
# Strategy {#strategy}

### `class lightning.pytorch.strategies.Strategy(accelerator=None, checkpoint_io=None, precision_plugin=None)` {#lightning.pytorch.strategies.Strategy}

Bases: [`ABC`](https://docs.python.org/3/library/abc.html#abc.ABC "(in Python v3.15)")

Base class for all strategies that change the behaviour of the training, validation and test- loop.

#### `abstract all_gather(tensor, group=None, sync_grads=False)` {#lightning.pytorch.strategies.Strategy.all_gather}

Perform an all\_gather on all processes.

Parameters:

*   **tensor** ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")) – the tensor to all\_gather
    
*   **group** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – the process group to gather results from
    
*   **sync\_grads** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – flag that allows users to synchronize gradients for all\_gather op
    

Return type:

[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")

#### `backward(closure_loss, optimizer, *args, **kwargs)` {#lightning.pytorch.strategies.Strategy.backward}

Forwards backward-calls to the precision plugin.

Parameters:

*   **closure\_loss** ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")) – a tensor holding the loss value to backpropagate
    
*   **optimizer** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Optimizer`](https://docs.pytorch.org/docs/stable/optim.html#torch.optim.Optimizer "(in PyTorch v2.15)")\]) – An optional optimizer that gets passed down to the precision plugin’s backward
    
*   **\*args** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – Positional arguments that get passed down to the precision plugin’s backward, intended as arguments for the actual function that performs the backward, like [`backward()`](https://docs.pytorch.org/docs/stable/generated/torch.Tensor.backward.html#torch.Tensor.backward "(in PyTorch v2.15)").
    
*   **\*\*kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – Keyword arguments for the same purpose as `*args`.
    

Return type:

[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")

#### `abstract barrier(name=None)` {#lightning.pytorch.strategies.Strategy.barrier}

Synchronizes all processes which blocks processes until the whole group enters this function.

Parameters:

**name** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]) – an optional name to pass into barrier.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `batch_to_device(batch, device=None, dataloader_idx=0)` {#lightning.pytorch.strategies.Strategy.batch_to_device}

Moves the batch to the correct device.

The returned batch is of the same type as the input batch, just having all tensors on the correct device.

Parameters:

*   **batch** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – The batch of samples to move to the correct device
    
*   **device** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`device`](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.15)")\]) – The target device
    
*   **dataloader\_idx** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – The index of the dataloader to which the batch belongs.
    

Return type:

[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")

#### `abstract broadcast(obj, src=0)` {#lightning.pytorch.strategies.Strategy.broadcast}

Broadcasts an object to all processes.

Parameters:

*   **obj** ([`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar "(in Python v3.15)")(`TBroadcast`)) – the object to broadcast
    
*   **src** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – source rank
    

Return type:

[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar "(in Python v3.15)")(`TBroadcast`)

#### `connect(model)` {#lightning.pytorch.strategies.Strategy.connect}

Called by the Trainer to connect the strategy with the model.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `lightning_module_state_dict()` {#lightning.pytorch.strategies.Strategy.lightning_module_state_dict}

Returns model state.

Return type:

[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]

#### `model_sharded_context()` {#lightning.pytorch.strategies.Strategy.model_sharded_context}

Provide hook to create modules in a distributed aware context. This is useful for when we’d like to shard the model instantly, which is useful for extremely large models which can save memory and initialization time.

Returns: Model parallel context.

Return type:

[`Generator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Generator "(in Python v3.15)")\[[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]

#### `abstract model_to_device()` {#lightning.pytorch.strategies.Strategy.model_to_device}

Moves the model to the correct device.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_exception(exception)` {#lightning.pytorch.strategies.Strategy.on_exception}

Called when the trainer execution is interrupted by an exception.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_predict_end()` {#lightning.pytorch.strategies.Strategy.on_predict_end}

Called when predict ends.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_predict_start()` {#lightning.pytorch.strategies.Strategy.on_predict_start}

Called when predict begins.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_test_end()` {#lightning.pytorch.strategies.Strategy.on_test_end}

Called when test end.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_test_start()` {#lightning.pytorch.strategies.Strategy.on_test_start}

Called when test begins.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_train_batch_start(batch, batch_idx)` {#lightning.pytorch.strategies.Strategy.on_train_batch_start}

Called in the training loop before anything happens for that batch.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_train_end()` {#lightning.pytorch.strategies.Strategy.on_train_end}

Called when train ends.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_train_start()` {#lightning.pytorch.strategies.Strategy.on_train_start}

Called when train begins.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_validation_end()` {#lightning.pytorch.strategies.Strategy.on_validation_end}

Called when validation ends.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `on_validation_start()` {#lightning.pytorch.strategies.Strategy.on_validation_start}

Called when validation begins.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `optimizer_state(optimizer)` {#lightning.pytorch.strategies.Strategy.optimizer_state}

Returns state of an optimizer.

Allows for syncing/collating optimizer state from processes in custom strategies.

Return type:

[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\]

#### `optimizer_step(optimizer, closure, model=None, **kwargs)` {#lightning.pytorch.strategies.Strategy.optimizer_step}

Performs the actual optimizer step.

Parameters:

*   **optimizer** ([`Optimizer`](https://docs.pytorch.org/docs/stable/optim.html#torch.optim.Optimizer "(in PyTorch v2.15)")) – the optimizer performing the step
    
*   **closure** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable "(in Python v3.15)")\[\[\], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – closure calculating the loss value
    
*   **model** (`Union`\[[`LightningModule`](/docs/pytorch/2.5.5/optional-api/api/lightning.pytorch.core.LightningModule.md#lightning.pytorch.core.LightningModule "lightning.pytorch.core.module.LightningModule"), [`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – reference to the model, optionally defining optimizer step related hooks
    
*   **\*\*kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – Keyword arguments to `optimizer.step`
    

Return type:

[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")

#### `post_backward(closure_loss)` {#lightning.pytorch.strategies.Strategy.post_backward}

Run after precision plugin executes backward.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `post_training_step()` {#lightning.pytorch.strategies.Strategy.post_training_step}

This hook is deprecated.

Override [`training_step()`](#lightning.pytorch.strategies.Strategy.training_step "lightning.pytorch.strategies.Strategy.training_step") instead.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `pre_backward(closure_loss)` {#lightning.pytorch.strategies.Strategy.pre_backward}

Run before precision plugin executes backward.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `predict_step(*args, **kwargs)` {#lightning.pytorch.strategies.Strategy.predict_step}

The actual predict step.

See [`predict_step()`](/docs/pytorch/2.5.5/optional-api/api/lightning.pytorch.core.LightningModule.md#lightning.pytorch.core.LightningModule.predict_step "lightning.pytorch.core.LightningModule.predict_step") for more details

Return type:

[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")

#### `process_dataloader(dataloader)` {#lightning.pytorch.strategies.Strategy.process_dataloader}

Wraps the dataloader if necessary.

Parameters:

**dataloader** ([`object`](https://docs.python.org/3/builtins/functions.html#object "(in Python v3.15)")) – iterable. Ideally of type: [`torch.utils.data.DataLoader`](https://docs.pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader "(in PyTorch v2.15)")

Return type:

[`object`](https://docs.python.org/3/builtins/functions.html#object "(in Python v3.15)")

#### `abstract reduce(tensor, group=None, reduce_op='mean')` {#lightning.pytorch.strategies.Strategy.reduce}

Reduces the given tensor (e.g. across GPUs/processes).

Parameters:

*   **tensor** (`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – the tensor to sync and reduce
    
*   **group** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – the process group to reduce
    
*   **reduce\_op** (`Union`\[[`ReduceOp`](https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.ReduceOp "(in PyTorch v2.15)"), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – the reduction operation. Defaults to ‘mean’. Can also be a string ‘sum’ or ReduceOp.
    

Return type:

`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]

#### `reduce_boolean_decision(decision, all=True)` {#lightning.pytorch.strategies.Strategy.reduce_boolean_decision}

Reduce a boolean decision across all processes.

Return type:

[`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")

#### `remove_checkpoint(filepath)` {#lightning.pytorch.strategies.Strategy.remove_checkpoint}

Remove checkpoint filepath from the filesystem.

Parameters:

**filepath** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path "(in Python v3.15)")\]) – Path to checkpoint

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `save_checkpoint(checkpoint, filepath, storage_options=None)` {#lightning.pytorch.strategies.Strategy.save_checkpoint}

Save model/training states as a checkpoint file through state-dump and file-write.

Parameters:

*   **checkpoint** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – dict containing model and trainer state
    
*   **filepath** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path "(in Python v3.15)")\]) – write-target file’s path
    
*   **storage\_options** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – parameter for how to save to storage, passed to `CheckpointIO` plugin
    

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `setup(trainer)` {#lightning.pytorch.strategies.Strategy.setup}

Sets up the accelerator, plugins and initializes the optimizers (if needed).

Parameters:

**trainer** ([`Trainer`](/docs/pytorch/2.5.5/optional-api/api/lightning.pytorch.trainer.trainer.Trainer.md#lightning.pytorch.trainer.trainer.Trainer "lightning.pytorch.trainer.trainer.Trainer")) – the trainer instance

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `setup_environment()` {#lightning.pytorch.strategies.Strategy.setup_environment}

Setup any processes or distributed connections.

This is called before the LightningModule/DataModule setup hook which allows the user to access the accelerator environment before setup is complete.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `setup_optimizers(trainer)` {#lightning.pytorch.strategies.Strategy.setup_optimizers}

Creates optimizers and schedulers.

Parameters:

**trainer** ([`Trainer`](/docs/pytorch/2.5.5/optional-api/api/lightning.pytorch.trainer.trainer.Trainer.md#lightning.pytorch.trainer.trainer.Trainer "lightning.pytorch.trainer.trainer.Trainer")) – the Trainer, these optimizers should be connected to

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `setup_precision_plugin()` {#lightning.pytorch.strategies.Strategy.setup_precision_plugin}

Attaches the precision plugin to the strategy.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `teardown()` {#lightning.pytorch.strategies.Strategy.teardown}

This method is called to teardown the training process.

It is the right place to release memory and free other resources.

Return type:

[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")

#### `tensor_init_context(empty_init=None)` {#lightning.pytorch.strategies.Strategy.tensor_init_context}

Controls how tensors get created (device, dtype).

Parameters:

**empty\_init** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")\]) – Whether to initialize the model with empty weights (uninitialized memory). If `None`, the strategy will decide. Some strategies may not support all options.

Return type:

[`Generator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Generator "(in Python v3.15)")\[[`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]

#### `test_step(*args, **kwargs)` {#lightning.pytorch.strategies.Strategy.test_step}

The actual test step.

See [`test_step()`](/docs/pytorch/2.5.5/optional-api/api/lightning.pytorch.core.LightningModule.md#lightning.pytorch.core.LightningModule.test_step "lightning.pytorch.core.LightningModule.test_step") for more details

Return type:

`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]

#### `training_step(*args, **kwargs)` {#lightning.pytorch.strategies.Strategy.training_step}

The actual training step.

See [`training_step()`](/docs/pytorch/2.5.5/optional-api/api/lightning.pytorch.core.LightningModule.md#lightning.pytorch.core.LightningModule.training_step "lightning.pytorch.core.LightningModule.training_step") for more details

Return type:

`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]

#### `validation_step(*args, **kwargs)` {#lightning.pytorch.strategies.Strategy.validation_step}

The actual validation step.

See [`validation_step()`](/docs/pytorch/2.5.5/optional-api/api/lightning.pytorch.core.LightningModule.md#lightning.pytorch.core.LightningModule.validation_step "lightning.pytorch.core.LightningModule.validation_step") for more details

Return type:

`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]

#### `property handles_gradient_accumulation: bool` {#lightning.pytorch.strategies.Strategy.handles_gradient_accumulation}

Whether the strategy handles gradient accumulation internally.

#### `abstract property is_global_zero: bool` {#lightning.pytorch.strategies.Strategy.is_global_zero}

Whether the current process is the rank zero process not only on the local node, but for all nodes.

#### `property lightning_module: Optional[LightningModule]` {#lightning.pytorch.strategies.Strategy.lightning_module}

Returns the pure LightningModule without potential wrappers.

#### `property lightning_restore_optimizer: bool` {#lightning.pytorch.strategies.Strategy.lightning_restore_optimizer}

Override to disable Lightning restoring optimizers/schedulers.

This is useful for strategies which manage restoring optimizers/schedulers.

#### `property model: Optional[Module]` {#lightning.pytorch.strategies.Strategy.model}

Returns the potentially wrapped LightningModule.

#### `property restore_checkpoint_after_setup: bool` {#lightning.pytorch.strategies.Strategy.restore_checkpoint_after_setup}

Override to delay restoring from checkpoint till after the setup phase has completed. This is useful when the strategy requires all the setup hooks to run before loading checkpoint.

Returns:

If `True`, restore checkpoint after strategy setup.

#### `abstract property root_device: device` {#lightning.pytorch.strategies.Strategy.root_device}

Returns the root device.

---

**Navigation**

- Parent: [Api](/docs/pytorch/2.5.5/optional-api/api.md)