Difference between BertForSequenceClassification and Bert + nn.Linear

goku · December 17, 2020, 7:57pm

The first thing I’d suggest is to check whether the backbone has the same weights in both cases. Although it seems correct to me, just verify it once though. Second check the dropout rate whether it’s same or not. Also maybe you need to check this too.

github.com

huggingface/transformers/blob/bf713cdec7c27416f514f231ba6728cfc8135120/src/transformers/models/bert/modeling_bert.py#L1336


      
          )
          class BertForSequenceClassification(BertPreTrainedModel):
              def __init__(self, config):
                  super().__init__(config)
                  self.num_labels = config.num_labels
          
          
        self.bert = BertModel(config)
                  self.dropout = nn.Dropout(config.hidden_dropout_prob)
                  self.classifier = nn.Linear(config.hidden_size, config.num_labels)
          
          
        self.init_weights()
          
          
    @add_start_docstrings_to_model_forward(BERT_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
              @add_code_sample_docstrings(
                  tokenizer_class=_TOKENIZER_FOR_DOC,
                  checkpoint="bert-base-uncased",
                  output_type=SequenceClassifierOutput,
                  config_class=_CONFIG_FOR_DOC,
              )
              def forward(
                  self,