Is there a way to run validation on a different worker or node than the training workers?

implementation help
1

My training is slow because we have to pause all progress to run validation.