How Lightning Helps You Reload Your Data on Every Epoch
Lightning is a lightweight PyTorch wrapper for high-performance AI research that reduces the boilerplate without limiting flexibility. In this series, we are covering all the tricks Lightning offers to supercharge your machine learning training.
By default, Lightning only loads your dataset once so you don’t incur the cost of downloading that data or processing it every single time. However you can change this behavior and force Lightning to reload your dataloader on every single epoch using the reload_dataloaders_every_epoch flag. You would use this if your data is constantly changing or if you sync it in something like a production system.
