How to switch from optimizer during training

we tried a bunch of different learning rate values, however, the model (MNIST, PDE) is not converging, since the weights are not updating correctly. if there was an example using LBFGS optimizer, we could debug our code better.