May I ask how do you correctly create a set of latents for each sample in the training dataset? I.e., suppose you would like to have optimizable latent codes for each of the frame. The total number of those latents is then decided by the number of data samples.
I was unable to do it in
LightningModule, as at that point
DataModule is not loaded yet. I have manged to put it at the beginning of
configure_optimizers, however it causes problem when trying to load a trained ckpt for prediction or resuming training, as the loading of weights seem to happen after
DataModule loading, but before
May I ask if there is a way to do it, or it can’t be achieved using lighting CLI?