Ignore log in one of the GPUs as it does not have a specific loss

DDP/GPU
1

Hi,
I have some code that runs different evaluation, something like this:

    def on_validation_epoch_end(self):
        for subset in self.validation_step_outputs:
            self.length_and_dist = np.array(self.validation_step_outputs[subset])
            if len(self.length_and_dist):
                w_errr = self.length_and_dist.mean()
                self.log(f"Error/{subset}_L", w_errr, sync_dist=True)
            else:
                print(f'no smaples in subset = {subset} at this GPU')

The problem is that some substes are very small and some GPUs will not have w_errr and I get into a deadlock. How can I overcome this probolem. I can not send a 0 in case the subset is not found at the gpu.

Thanks

2

I think you answered your question yourself: “send a 0 in case the subset is not found”.

 if len(self.length_and_dist):
    w_errr = self.length_and_dist.mean()
else:
    w_err = 0

self.log(f"Error/{subset}_L", w_err, sync_dist=True)
3

Thanks but I think that this will assume that the loss/error in one of the GPUs it 0 which will be not accurate, for example:
3 GPUs, the error in GPUs 1,2 is 0.7 and 0.8 and GPU 3 will not have any of these samples, so if it sends zero the mean of those samples will be not accurate, no ? or maybe I am missing something