Validation Error: The validation bar turns red and no callbacks are called, no checkpoints are saved

[Thiru]

I currently code in kaggle kernel for a competition where I found this weird behavior.
My validation progress bar turned red and no checkpoints were saved but training proceeded to next epoch while validation was struck.

[goku]

mind share the notebook or reproduce this behavior with Google Colab??

[Thiru]

The link to my notebook

[asvskartheek]

The validation loop is not stuck, this behaviour is just caused by progress_bar_refresh_rate. Your validation loop has 263 steps and the progress bar gets updated only after every progress_bar_refresh_rate (20) steps. So, last it (progress bar) got updated is after 260 steps, because it is divisible by 20. The remaining 3 aren’t updated in that progress bar but the model has completed them.

[Thiru]

thank you for your great explanation. I understood what it is

[goku]

I also tried it, working fine for me. The validation progress bar disappears as expected. Also, I used limit_train_batches=23 just to replicate your case and avoid long runs.