---
title: BERT Score
description: Bert_score Evaluating Text Generation for measuring text similarity.
canonical: 'https://lightning.ai/docs/torchmetrics/stable/text/bert_score'
---
# BERT Score {#bert-score}

## Module Interface {#module-interface}

### `class torchmetrics.text.bert.BERTScore(model_name_or_path=None, num_layers=None, all_layers=False, model=None, user_tokenizer=None, user_forward_fn=None, verbose=False, idf=False, device=None, max_length=512, batch_size=64, num_threads=0, return_hash=False, lang='en', rescale_with_baseline=False, baseline_path=None, baseline_url=None, truncation=False, **kwargs)` {#torchmetrics.text.bert.BERTScore}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/text/bert.py#L59-L366)

[Bert\_score Evaluating Text Generation](https://arxiv.org/abs/1904.09675) for measuring text similarity.

BERT leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity. It has been shown to correlate with human judgment on sentence-level and system-level evaluation. Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks. This implementation follows the original implementation from [BERT\_score](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py).

As input to `forward` and `update` the metric accepts the following input:

*   `preds`: Predicted sentence(s). Can be one of:
    
    *   A single predicted sentence as a string (`str`)
        
    *   A sequence of predicted sentences (`Sequence[str]`)
        
*   `target`: Target/reference sentence(s). Can be one of:
    
    *   A single reference sentence as a string (`str`)
        
    *   A sequence of reference sentences (`Sequence[str]`)
        
    *   A sequence of sequences of reference sentences for multi-reference evaluation (`Sequence[Sequence[str]]`)
        

As output of `forward` and `compute` the metric returns the following output:

*   `score` (`Dict`): A dictionary containing the keys `precision`, `recall` and `f1` with corresponding values
    

Parameters:

*   **preds** (_Union__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_,_ _Sequence__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_\]__\]_) – A single predicted sentence or a sequence of predicted sentences.
    
*   **target** (_Union__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_,_ _Sequence__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_\]__,_ _Sequence__\[__Sequence__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_\]__\]__\]_) – A single target sentence, a sequence of target sentences, or a sequence of sequences of target sentences for multiple references per prediction.
    
*   **model\_type** – A name or a model path used to load `transformers` pretrained model.
    
*   **num\_layers** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")\]) – A layer of representation to use.
    
*   **all\_layers** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether the representation from all model’s layers should be used. If `all_layers=True`, the argument `num_layers` is ignored.
    
*   **model** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")\]) – A user’s own model. Must be of torch.nn.Module instance.
    
*   **user\_tokenizer** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – A user’s own tokenizer used with the own model. This must be an instance with the `__call__` method. This method must take an iterable of sentences (List\[str\]) and must return a python dictionary containing “input\_ids” and “attention\_mask” represented by [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"). It is up to the user’s model of whether “input\_ids” is a [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)") of input ids or embedding vectors. This tokenizer must prepend an equivalent of `[CLS]` token and append an equivalent of `[SEP]` token as `transformers` tokenizer does.
    
*   **user\_forward\_fn** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable "(in Python v3.15)")\[\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)"), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\]\], [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\]\]) – A user’s own forward function used in a combination with `user_model`. This function must take `user_model` and a python dictionary of containing `"input_ids"` and `"attention_mask"` represented by [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)") as an input and return the model’s output represented by the single [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)").
    
*   **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether a progress bar to be displayed during the embeddings’ calculation.
    
*   **idf** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication whether normalization using inverse document frequencies should be used.
    
*   **device** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`device`](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – A device to be used for calculation.
    
*   **max\_length** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed.
    
*   **batch\_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A batch size used for model processing.
    
*   **num\_threads** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A number of threads to use for a dataloader.
    
*   **return\_hash** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether the correspodning `hash_code` should be returned.
    
*   **lang** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")) – A language of input sentences.
    
*   **rescale\_with\_baseline** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether bertscore should be rescaled with a pre-computed baseline. When a pretrained model from `transformers` model is used, the corresponding baseline is downloaded from the original `bert-score` package from [BERT\_score](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py) if available. In other cases, please specify a path to the baseline csv/tsv file, which must follow the formatting of the files from [BERT\_score](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py).
    
*   **baseline\_path** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]) – A path to the user’s own local csv/tsv file with the baseline scale.
    
*   **baseline\_url** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]) – A url path to the user’s own csv/tsv file with the baseline scale.
    
*   **truncation** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether the input sequences should be truncated to the `max_length`.
    
*   **kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – Additional keyword arguments, see [Advanced metric settings](/docs/torchmetrics/1.9.0/user-guide/overview.md#metric-kwargs) for more info.
    

Example

```python
>>> from pprint import pprint
>>> from torchmetrics.text.bert import BERTScore
>>> preds = ["hello there", "general kenobi"]
>>> target = ["hello there", "master kenobi"]
>>> bertscore = BERTScore()
>>> pprint(bertscore(preds, target))
{'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])}
```

Example

```python
>>> from pprint import pprint
>>> from torchmetrics.text.bert import BERTScore
>>> preds = ["hello there", "general kenobi"]
>>> target = [["hello there", "master kenobi"], ["hello there", "master kenobi"]]
>>> bertscore = BERTScore()
>>> pprint(bertscore(preds, target))
{'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])}
```

#### `plot(val=None, ax=None)` {#torchmetrics.text.bert.BERTScore.plot}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/text/bert.py#L318-L366)

Plot a single or multiple values from the metric.

Parameters:

*   **val** (`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.
    
*   **ax** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)")\]) – An matplotlib axis object. If provided will add plot to that axis
    

Return type:

[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`Figure`](https://matplotlib.org/stable/api/_as_gen/matplotlib.figure.Figure.html#matplotlib.figure.Figure "(in Matplotlib v3.11.2)"), `Union`\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)"), [`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray "(in NumPy v2.5)")\]\]

Returns:

Figure and Axes object

Raises:

[**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError "(in Python v3.15)") – If matplotlib is not installed

```python
>>> # Example plotting a single value
>>> from torchmetrics.text.bert import BERTScore
>>> preds = ["hello there", "general kenobi"]
>>> target = ["hello there", "master kenobi"]
>>> metric = BERTScore()
>>> metric.update(preds, target)
>>> fig_, ax_ = metric.plot()
```

[![../../\_images/bert\_score-1.png](/docs/torchmetrics/1.9.0/static/_images/bert_score-1.png)](/docs/torchmetrics/1.9.0/static/_images/bert_score-1.png)

```python
>>> # Example plotting multiple values
>>> from torch import tensor
>>> from torchmetrics.text.bert import BERTScore
>>> preds = ["hello there", "general kenobi"]
>>> target = ["hello there", "master kenobi"]
>>> metric = BERTScore()
>>> values = []
>>> for _ in range(10):
...     val = metric(preds, target)
...     val = {k: tensor(v).mean() for k,v in val.items()}  # convert into single value per key
...     values.append(val)
>>> fig_, ax_ = metric.plot(values)
```

[![../../\_images/bert\_score-2.png](/docs/torchmetrics/1.9.0/static/_images/bert_score-2.png)](/docs/torchmetrics/1.9.0/static/_images/bert_score-2.png)

## Functional Interface {#functional-interface}

### `torchmetrics.functional.text.bert.bert_score(preds, target, model_name_or_path=None, num_layers=None, all_layers=False, model=None, user_tokenizer=None, user_forward_fn=None, verbose=False, idf=False, device=None, max_length=512, batch_size=64, num_threads=0, return_hash=False, lang='en', rescale_with_baseline=False, baseline_path=None, baseline_url=None, truncation=False)` {#torchmetrics.functional.text.bert.bert_score}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/functional/text/bert.py#L349-L590)

[Bert\_score Evaluating Text Generation](https://arxiv.org/abs/1904.09675) for text similirity matching.

This metric leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity. It has been shown to correlate with human judgment on sentence-level and system-level evaluation. Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks.

This implementation follows the original implementation from [BERT\_score](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py).

Parameters:

*   **preds** (_Union__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_,_ _Sequence__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_\]__\]_) – A single predicted sentence or a sequence of predicted sentences.
    
*   **target** (_Union__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_,_ _Sequence__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_\]__,_ _Sequence__\[__Sequence__\[_[_str_](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")_\]__\]__\]_) – A single target sentence, a sequence of target sentences, or a sequence of sequences of target sentences for multiple references per prediction.
    
*   **model\_name\_or\_path** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]) – A name or a model path used to load `transformers` pretrained model.
    
*   **num\_layers** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")\]) – A layer of representation to use.
    
*   **all\_layers** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether the representation from all model’s layers should be used. If `all_layers = True`, the argument `num_layers` is ignored.
    
*   **model** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)")\]) – A user’s own model.
    
*   **user\_tokenizer** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")\]) – A user’s own tokenizer used with the own model. This must be an instance with the `__call__` method. This method must take an iterable of sentences (`List[str]`) and must return a python dictionary containing `"input_ids"` and `"attention_mask"` represented by [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"). It is up to the user’s model of whether `"input_ids"` is a [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)") of input ids or embedding vectors. his tokenizer must prepend an equivalent of `[CLS]` token and append an equivalent of `[SEP]` token as transformers tokenizer does.
    
*   **user\_forward\_fn** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable "(in Python v3.15)")\[\[[`Module`](https://docs.pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module "(in PyTorch v2.15)"), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\]\], [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\]\]) – A user’s own forward function used in a combination with `user_model`. This function must take `user_model` and a python dictionary of containing `"input_ids"` and `"attention_mask"` represented by [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)") as an input and return the model’s output represented by the single [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)").
    
*   **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether a progress bar to be displayed during the embeddings’ calculation.
    
*   **idf** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether normalization using inverse document frequencies should be used.
    
*   **device** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`device`](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – A device to be used for calculation.
    
*   **max\_length** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed.
    
*   **batch\_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A batch size used for model processing.
    
*   **num\_threads** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A number of threads to use for a dataloader.
    
*   **return\_hash** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether the correspodning `hash_code` should be returned.
    
*   **lang** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")) – A language of input sentences. It is used when the scores are rescaled with a baseline.
    
*   **rescale\_with\_baseline** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether bertscore should be rescaled with a pre-computed baseline. When a pretrained model from `transformers` model is used, the corresponding baseline is downloaded from the original `bert-score` package from [BERT\_score](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py) if available. In other cases, please specify a path to the baseline csv/tsv file, which must follow the formatting of the files from [BERT\_score](https://github.com/Tiiiger/bert_score/blob/master/bert_score/utils.py)
    
*   **baseline\_path** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]) – A path to the user’s own local csv/tsv file with the baseline scale.
    
*   **baseline\_url** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]) – A url path to the user’s own csv/tsv file with the baseline scale.
    
*   **truncation** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether the input sequences should be truncated to the maximum length.
    

Return type:

[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), `Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`List`](https://docs.python.org/3/library/typing.html#typing.List "(in Python v3.15)")\[[`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")\], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]\]

Returns:

Python dictionary containing the keys `precision`, `recall` and `f1` with corresponding values.

Raises:

*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If `len(preds) != len(target)`.
    
*   [**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError "(in Python v3.15)") – If tqdm package is required and not installed.
    
*   [**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError "(in Python v3.15)") – If `transformers` package is required and not installed.
    
*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If `num_layer` is larger than the number of the model layers.
    
*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If invalid input is provided.
    

Example

```python
>>> from pprint import pprint
>>> from torchmetrics.functional.text.bert import bert_score
>>> preds = ["hello there", "general kenobi"]
>>> target = ["hello there", "master kenobi"]
>>> pprint(bert_score(preds, target))
{'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])}
```

Example

```python
>>> from pprint import pprint
>>> from torchmetrics.functional.text.bert import bert_score
>>> preds = ["hello there", "general kenobi"]
>>> target = [["hello there", "master kenobi"], ["hello there", "master kenobi"]]
>>> pprint(bert_score(preds, target))
{'f1': tensor([1.0000, 0.9961]), 'precision': tensor([1.0000, 0.9961]), 'recall': tensor([1.0000, 0.9961])}
```

---

**Navigation**

- Parent: [Text](/docs/torchmetrics/1.9.0/text.md)