Target size that is different to the input size

XYBMA · September 20, 2021, 4:11pm

I am using PyTorch lightning for the first time and I got stuck in a problem that I couldn’t figure out where it comes.
My project is a multi-classification (10 classes) problem with transformers (Bert). I am using the PyTorch lightning library the get my code running on GPU and TPU with minimal change. I am using Google Colab and after installing all the needed libraries, here is the code that I tried with comments :

#I changef all the label to int
possible_labels = df1.classement.unique()

label_dict = {}
for index, possible_label in enumerate(possible_labels):
    label_dict[possible_label] = index

#and split tit o train and val DF
X_train, X_val = train_test_split(df1, test_size=0.05, stratify=df1.label.values)

#Loaded pretrained model
MODEL_NAME = 'bert-base-multilingual-cased' 
tokenizer = BertTokenizer.from_pretrained(MODEL_NAME, do_lower_case=False)

So my data looks like this :

	commentaire	classement
0	Nul à chier	Hate
1	VIDEO DE MERDE KFC EST TRES BIEN ET LE PDG VS …	Hate
2	il faut arreter de faire des videos clash roya…	Hate
3	9 ans ou pas le branleur je me fous en slip ,j…	Hate
4	Nul	Hate

After this, I created a class that inherit from Dataset of Pythorch Lightning to handle that dataset :

class BODDataset(Dataset):

  def __init__(
    self, 
    data: pd.DataFrame, 
    tokenizer: BertTokenizer, 
    max_token_len: int = 512
  ):
    self.tokenizer = tokenizer
    self.data = data
    self.max_token_len = max_token_len
    
  def __len__(self):
    return len(self.data)

  def __getitem__(self, index: int):
    data_row = self.data.iloc[index]

    commentaire = data_row.commentaire
    classes = data_row.label

    encoding = self.tokenizer.encode_plus(
      commentaire,
      add_special_tokens=True,
      max_length=self.max_token_len,
      return_token_type_ids=False,
      padding="max_length",
      truncation=True,
      return_attention_mask=True,
      return_tensors='pt',
    )

    return dict(
      commentaire=commentaire,
      input_ids=encoding["input_ids"].flatten(),
      attention_mask=encoding["attention_mask"].flatten(),
      classes=torch.tensor(classes)
    )

At this pint, I got something like this :

{'attention_mask': tensor([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0]),
 'classes': tensor(5),
 'commentaire': 'Level up GG',
 'input_ids': tensor([  101, 37359, 10741,   144, 11447,   102,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0])}

After this I passed to the ‘LightningDataModule’ :

class CommentDataModule(pl.LightningDataModule):

  def __init__(self, train_df, val_df, tokenizer, batch_size, max_token_len):
    super().__init__()
    self.batch_size = batch_size
    self.train_df = train_df
    self.val_df = val_df
    self.tokenizer = tokenizer
    self.max_token_len = max_token_len

  def setup(self, stage=None):
    self.train_dataset = BODDataset(
      self.train_df,
      self.tokenizer,
      self.max_token_len
    )

    self.test_dataset = BODDataset(
      self.val_df,
      self.tokenizer,
      self.max_token_len
    )

  def train_dataloader(self):
    return DataLoader(
      self.train_dataset,
      batch_size=self.batch_size,
      shuffle=True,
      num_workers=2
    )

  def val_dataloader(self):
    return DataLoader(
      self.test_dataset,
      batch_size=self.batch_size,
      num_workers=2
    )

N_EPOCHS = 10
BATCH_SIZE = 16

data_module = CommentDataModule(
  X_train,
  X_val,
  tokenizer,
  batch_size=BATCH_SIZE,
  max_token_len=MAX_TOKEN_COUNT
)

finally the model :

class BODDetector(pl.LightningModule):

  def __init__(self, n_classes: int, n_training_steps=None, n_warmup_steps=None):
    super().__init__()
    self.bert = BertModel.from_pretrained(MODEL_NAME, return_dict=True)
    self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes)
    self.n_training_steps = n_training_steps
    self.n_warmup_steps = n_warmup_steps
    self.criterion = nn.BCELoss()

  def forward(self, input_ids, attention_mask, labels=None):
    output = self.bert(input_ids, attention_mask=attention_mask)
    output = self.classifier(output.pooler_output)
    output = torch.sigmoid(output)    
    loss = 0
    if labels is not None:
        loss = self.criterion(output, labels)
    return loss, output

  def training_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["classes"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("train_loss", loss, prog_bar=True, logger=True)
    return {"loss": loss, "predictions": outputs, "labels": labels}

  def validation_step(self, batch, batch_idx):
    input_ids = batch["input_ids"]
    attention_mask = batch["attention_mask"]
    labels = batch["classes"]
    loss, outputs = self(input_ids, attention_mask, labels)
    self.log("val_loss", loss, prog_bar=True, logger=True)
    return loss


  def training_epoch_end(self, outputs):
    labels = []
    predictions = []
    for output in outputs:
      for out_labels in output["labels"].detach().cpu():
        labels.append(out_labels)
      for out_predictions in output["predictions"].detach().cpu():
        predictions.append(out_predictions)

    labels = torch.stack(labels).int()
    predictions = torch.stack(predictions)

    for i, name in enumerate(label_dict):
      class_roc_auc = auroc(predictions[:, i], labels[:, i])
      self.logger.experiment.add_scalar(f"{name}_roc_auc/Train", class_roc_auc, self.current_epoch)


  def configure_optimizers(self):

    optimizer = AdamW(self.parameters(), lr=2e-5)

    scheduler = get_linear_schedule_with_warmup(
      optimizer,
      num_warmup_steps=self.n_warmup_steps,
      num_training_steps=self.n_training_steps
    )

    return dict(
      optimizer=optimizer,
      lr_scheduler=dict(
        scheduler=scheduler,
        interval='step'
      )
    )

steps_per_epoch=len(X_train) // BATCH_SIZE
total_training_steps = steps_per_epoch * N_EPOCHS

model = BODDetector( n_classes=len(label_dict), n_training_steps=total_training_steps )

#To save the model
checkpoint_callback = ModelCheckpoint(
  dirpath="checkpoints",
  filename="best-checkpoint",
  save_top_k=1,
  verbose=True,
  monitor="val_loss",
  mode="min"
)

trainer = pl.Trainer(
  gpus=1,
  callbacks=[checkpoint_callback, early_stopping_callback],
  max_epochs=N_EPOCHS,   
  progress_bar_refresh_rate=10
)

Here finally after the run trainer.fit(model, data_module) Igot this error :

Using a target size (torch.Size([16])) that is different to the input size (torch.Size([16, 10])) is deprecated. Please ensure they have the same size.

I know that the 16 is the batch size that I used and 10 is the number of classes but what I couldn’t figure out is how the model wants me to resize the input to [16, 10]. Not to mention that my data has only 2 columns of comments and labels. Can anyone please help me to get through this, please?

yongen9696 · September 23, 2021, 1:53am

XYBMA:

...
class BODDetector(pl.LightningModule):

  def __init__(self, n_classes: int, n_training_steps=None, n_warmup_steps=None):
    super().__init__()
    self.bert = BertModel.from_pretrained(MODEL_NAME, return_dict=True)
    self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes)
...

perhaps it is due to how you have defined your nn.Linear

XYBMA · September 23, 2021, 7:24am

Hello,

Thanks for you reply but, the model I used here is a predefined one (from the transformers of haggingface). I did not define it myself!

WAN · October 13, 2021, 12:24am

Did you solve it? I have similar problem while using bert.
I’m following this toturial https://bugspeed.xyz/multi-label-text-classification-using-bert/
but i got ValueError: Using a target size (torch.Size([1, 4])) that is different to the input size (torch.Size([1, 5])) is deprecated. Please ensure they have the same size.

XYBMA · October 13, 2021, 7:41am

Hello,
Yes, I solve it by changing the model. I am not sure why there is a nn.Linear.
the tuto is the same as I followed I don’t see the purpose of using a linear layer at the end where we can just use BertForSequenceClassification!
if you want to fix this model, you need to check the input/ouput of the bert model and the linear classifier or you can do as I did and change the model. You can use the bert classification model. It is the better solution.
Hope it will help you

WAN · October 13, 2021, 3:30pm

Thanks a lot for your reply. Could you please show me the part of code that you have changed to solve this proplem?
I really appreciate it.

XYBMA · October 13, 2021, 3:56pm

In the __init__() of LightningModule instead of

self.bert = BertModel.from_pretrained(BERT_MODEL_NAME, return_dict=True)
self.classifier = nn.Linear(self.bert.config.hidden_size, n_classes)

you need to put

self.bert = BertForSequenceClassification.from_pretrained(BERT_MODEL_NAME, num_labels=YOUR_NUM_OF_CLASSES)

Of course, don’t forget that this change will affect some other methods in the pl.LightningModule class. so … up to you to change them
good luck!

WAN · October 13, 2021, 4:17pm

Thanks a lot.
I’m not familiar with pytorch. Thanks for your help.
I changed what you have mentioned:

        
self.bert = BertForSequenceClassification.from_pretrained(BERT_MODEL_NAME, num_labels=5,return_dict=True )

and I changed the forward function to be:


    def forward(self, input_ids, attention_mask, labels=None):
        output = self.bert(input_ids, attention_mask=attention_mask)
        #output = self.classifier(output.pooler_output)
        output = torch.sigmoid(output)
        loss = 0
        if labels is not None:
            loss = self.criterion(output, labels)
        return loss, output

then when I fit the model, this error happened:
TypeError: sigmoid(): argument ‘input’ (position 1) must be Tensor, not SequenceClassifierOutput

do you know why?

XYBMA · October 25, 2021, 9:05am

(Sorry for the late reply)

Yes, I know
it is because you are trying to put the output values of the bertClassifier into a function (sigmoid) that needs a tensor to produce while the output of the model just values (loss and logtis). In this case, you don’t need a sigmoid, we can say you just had it from the bertClassifier. you can eliminate output = torch.sigmoid(output).
Hope this helps you

Theory_of_Everything · May 19, 2022, 7:32am

It gives me error message: ‘object has no attribute ‘classifier’’, so what i i have to put as self.classifier? thanks

Theory_of_Everything · May 19, 2022, 8:24am

Sorry, i get message error: ‘SequenceClassifierOutput’ object has no attribute ‘size’

Topic		Replies	Views
How to tune Parameter using ray library with Pytorch lightning LightningModule	0	764	September 27, 2021
Invalid Datatype for loaders - Pytorch Lightning DataModule DataModule	0	443	December 18, 2021
BERT model throws error when used in Pytorch Lightning LightningModule	2	3453	October 17, 2020
Loading PyTorch Lightning Trained checkpoint	0	2038	July 29, 2021
Code structuring for text classification with hf bert-uncase implementation help	2	483	March 23, 2023

Target size that is different to the input size

Related topics