I used both a large dataset and a small dataset to train a model (all settings of the model are the same except the batch size). And I found the checkpoint field of using large set is much larger than that one of using small set.
The ‘save_weights_only’ of ‘ModelCheckpoint’ is set to True to lower the size, but the checkpoint size still reaches 8 Gb, causing a long waiting time when saving.
The question comes to (1)what does the checkpoint files contain? (2) Why its size relates to the training dataset? (3) and how to reduce the time for saving the the checkpoint file?