Hi there, I’ve been trying to figure out how to implement the following. I have a use case where I would have n different dataloaders (with potentially different lengths) and want to yield batches at corresponding n batch frequencies (one of the n dataloaders has to be sampled from at a frequency of 1, or every batch). For every other dataloader besides the one that is yielded for every batch, these should yielded from with their corresponding frequency (in a sort of infinite loop, so catching StopIteration and reinitializing the iterable). Let’s say that this is the structure:
dataloaders = {"child": DataLoader(child_dataset), "parent": DataLoader(parent_dataset)}
agg_frequencies = {"child": 1, "parent": 5}
What I want is to create maybe some subclass of CombinedLoader that can take in these two variables and produce an Iterable that yields something in this format:
{"child": ...} # batch 0
{"child": ...} # batch 1
{"child": ...} # batch 2
{"child": ...} # batch 3
{"child": ..., "parent": ...} # batch 4
{"child": ...} # batch 5
Any tips on how to do this?