I am trying to train a binary classification model for x-rays based on a pretrained ResNet50 model (ImageNet). As soon as I adjust the last FC-Layer of the ResNet down to 2 outputs (from originally 1000 for ImageNet1k, the learning rate finder just shows a graph that looks like this:
I have rebuilt the model from scratch and tested the LR-Finder along the way, so I am fairly sure that changing the FC-Layer is what breaks it.
Has anyone ever experienced a problem like this? Thanks!