How to keep track of training time in DDP setting?

Yes, self.log computes the reduced value, but it will send it to the logger, not the terminal standard output. You could add

self.log(“training_time”, train_time, sync_dist=True, prog_bar=True)

to send it to the progress bar. That might be a good solution for you.

So I don’t know if they are different or not.

To know whether they are different, you would have to call print() and see in the terminal. But they won’t be significantly different, because the training steps in DDP are synchronized.

1 Like