Glossary

2D Parallelism

Accelerator

Apple Silicon

Autocast

Barrier

Bfloat16

Broadcast

Callback

Checkpoint

CLI

Cloud

Collective

Compile

CUDA

FabricModule

FSDP

Gather

Gradient Accumulation

GPU

Initialization

Jupyter

Launch

LightningModule

Logger

Mixed Precision

Model Parallelism

MPI

MPS

Multi-GPU

Multi-Node

Notebook

Optimizers

Precision

Quantization

Reduce

SLURM

TensorBoard

Tensor Parallelism

TorchElastic

TorchRun

TPU

Trainer

Weights and Biases

16-bit, 8-bit, 4-bit