NativeMixedPrecisionPlugin¶

class pytorch_lightning.plugins.precision.NativeMixedPrecisionPlugin(precision, device, scaler=None)[source]¶

Bases: pytorch_lightning.plugins.precision.mixed.MixedPrecisionPlugin

Plugin for Native Mixed Precision (AMP) training with torch.autocast.

Parameters

precision¶ (Union[str, int]) – Whether to use torch.float16 (16) or torch.bfloat16 ('bf16').
device¶ (str) – The device for torch.autocast.
scaler¶ (Optional[GradScaler]) – An optional torch.cuda.amp.GradScaler to use.

forward_context()[source]¶

Enable autocast context.

Return type: Generator[None, None, None]

on_load_checkpoint(checkpoint)[source]¶

Called by Lightning to restore your model. If you saved something with on_save_checkpoint() this is your chance to restore this.

Parameters: checkpoint¶ (Dict[str, Any]) – Loaded checkpoint
Return type: None

Example:

def on_load_checkpoint(self, checkpoint):
    # 99% of the time you don't need to implement this method
    self.something_cool_i_want_to_save = checkpoint['something_cool_i_want_to_save']

Note

Lightning auto-restores global step, epoch, and train state including amp scaling. There is no need for you to restore anything regarding training.

on_save_checkpoint(checkpoint)[source]¶

Called by Lightning when saving a checkpoint to give you a chance to store anything else you might want to save.

Parameters: checkpoint¶ (Dict[str, Any]) – The full checkpoint dictionary before it gets dumped to a file. Implementations of this hook can insert additional data into this dictionary.
Return type: None

Example:

def on_save_checkpoint(self, checkpoint):
    # 99% of use cases you don't need to implement this method
    checkpoint['something_cool_i_want_to_save'] = my_cool_pickable_object

Note

Lightning saves all aspects of training (epoch, global step, etc…) including amp scaling. There is no need for you to store anything about training.

optimizer_step(model, optimizer, optimizer_idx, closure, **kwargs)[source]¶

Hook to run the optimizer step.

Return type: None

pre_backward(model, closure_loss)[source]¶

Run before precision plugin executes backward.

Parameters

model¶ (LightningModule) – the model to be optimized
closure_loss¶ (Tensor) – the loss value obtained from the closure

Return type

Tensor