trying LBFGS with native pytorch code, it works fine and the weights are updating and model is converging. As you mentioned, there might be a bug that needs to be fixed.
—> using LBFGS optimizer in pytorch lightening the model is not converging as compared to native pytoch + LBFGS #4083