---
title: Sacre BLEU Score
description: Calculate BLEU score of machine translated text with one or more references.
canonical: 'https://lightning.ai/docs/torchmetrics/stable/text/sacre_bleu_score'
---
# Sacre BLEU Score {#sacre-bleu-score}

## Module Interface {#module-interface}

### `class torchmetrics.text.SacreBLEUScore(n_gram=4, smooth=False, tokenize='13a', lowercase=False, weights=None, **kwargs)` {#torchmetrics.text.SacreBLEUScore}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/text/sacre_bleu.py#L35-L180)

Calculate [BLEU score](https://en.wikipedia.org/wiki/BLEU) of machine translated text with one or more references.

This implementation follows the behaviour of [SacreBLEU](https://github.com/mjpost/sacrebleu). The SacreBLEU implementation differs from the NLTK BLEU implementation in tokenization techniques.

As input to `forward` and `update` the metric accepts the following input:

*   `preds` (`Sequence`): An iterable of machine translated corpus
    
*   `target` (`Sequence`): An iterable of iterables of reference corpus
    

As output of `forward` and `compute` the metric returns the following output:

*   `sacre_bleu` ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")): A tensor with the SacreBLEU Score
    

Note

In the original SacreBLEU, references are passed as a list of reference sets (grouped by reference index). In TorchMetrics, references are passed grouped per prediction (each prediction has its own list of references).

For example:

```python
# Predictions
preds = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

# Original SacreBLEU:
refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'], # First set
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'], # Second set
]

# TorchMetrics SacreBLEU:
target = [
    ['The dog bit the man.', 'The dog had bit the man.'], # References for first prediction
    ['It was not unexpected.', 'No one was surprised.'], # References for second prediction
    ['The man bit him first.', 'The man had bitten the dog.'], # References for third prediction
]
```

Parameters:

*   **n\_gram** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – Gram value ranged from 1 to 4
    
*   **smooth** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – Whether to apply smoothing, see [SacreBLEU](https://github.com/mjpost/sacrebleu)
    
*   **tokenize** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'none'`, `'13a'`, `'zh'`, `'intl'`, `'char'`, `'ja-mecab'`, `'ko-mecab'`, `'flores101'`, `'flores200'`\]) – Tokenization technique to be used. Choose between `'none'`, `'13a'`, `'zh'`, `'intl'`, `'char'`, `'ja-mecab'`, `'ko-mecab'`, `'flores101'` and `'flores200'`.
    
*   **lowercase** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – If `True`, BLEU score over lowercased text is calculated.
    
*   **kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – Additional keyword arguments, see [Advanced metric settings](/docs/torchmetrics/1.9.0/user-guide/overview.md#metric-kwargs) for more info.
    
*   **weights** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")\]\]) – Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used.
    

Raises:

*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If `tokenize` not one of ‘none’, ‘13a’, ‘zh’, ‘intl’ or ‘char’
    
*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If `tokenize` is set to ‘intl’ and regex is not installed
    
*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If a length of a list of weights is not `None` and not equal to `n_gram`.
    

Example

```python
>>> from torchmetrics.text import SacreBLEUScore
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> sacre_bleu = SacreBLEUScore()
>>> sacre_bleu(preds, target)
tensor(0.7598)
```

Additional References:

*   Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och [Machine Translation Evolution](https://aclanthology.org/P04-1077)
    

#### `plot(val=None, ax=None)` {#torchmetrics.text.SacreBLEUScore.plot}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/text/sacre_bleu.py#L138-L180)

Plot a single or multiple values from the metric.

Parameters:

*   **val** (`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.
    
*   **ax** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)")\]) – An matplotlib axis object. If provided will add plot to that axis
    

Return type:

[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`Figure`](https://matplotlib.org/stable/api/_as_gen/matplotlib.figure.Figure.html#matplotlib.figure.Figure "(in Matplotlib v3.11.2)"), `Union`\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)"), [`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray "(in NumPy v2.5)")\]\]

Returns:

Figure and Axes object

Raises:

[**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError "(in Python v3.15)") – If matplotlib is not installed

```python
>>> # Example plotting a single value
>>> from torchmetrics.text import SacreBLEUScore
>>> metric = SacreBLEUScore()
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> metric.update(preds, target)
>>> fig_, ax_ = metric.plot()
```

[![../../\_images/sacre\_bleu\_score-1.png](/docs/torchmetrics/1.9.0/static/_images/sacre_bleu_score-1.png)](/docs/torchmetrics/1.9.0/static/_images/sacre_bleu_score-1.png)

```python
>>> # Example plotting multiple values
>>> from torchmetrics.text import SacreBLEUScore
>>> metric = SacreBLEUScore()
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> values = [ ]
>>> for _ in range(10):
...     values.append(metric(preds, target))
>>> fig_, ax_ = metric.plot(values)
```

[![../../\_images/sacre\_bleu\_score-2.png](/docs/torchmetrics/1.9.0/static/_images/sacre_bleu_score-2.png)](/docs/torchmetrics/1.9.0/static/_images/sacre_bleu_score-2.png)

## Functional Interface {#functional-interface}

### `torchmetrics.functional.text.sacre_bleu_score(preds, target, n_gram=4, smooth=False, tokenize='13a', lowercase=False, weights=None)` {#torchmetrics.functional.text.sacre_bleu_score}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/functional/text/sacre_bleu.py#L459-L555)

Calculate [BLEU score](https://en.wikipedia.org/wiki/BLEU) \[1\] of machine translated text with one or more references.

This implementation follows the behaviour of SacreBLEU \[2\] implementation from [https://github.com/mjpost/sacrebleu](https://github.com/mjpost/sacrebleu).

Note

In the original SacreBLEU, references are passed as a list of reference sets (grouped by reference index). In TorchMetrics, references are passed grouped per prediction (each prediction has its own list of references).

For example:

```python
# Predictions
preds = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']

# Original SacreBLEU:
refs = [
    ['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'], # First set
    ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.'], # Second set
]

# TorchMetrics SacreBLEU:
target = [
    ['The dog bit the man.', 'The dog had bit the man.'], # References for first prediction
    ['It was not unexpected.', 'No one was surprised.'], # References for second prediction
    ['The man bit him first.', 'The man had bitten the dog.'], # References for third prediction
]
```

Parameters:

*   **preds** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]) – An iterable of machine translated corpus
    
*   **target** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]\]) – An iterable of iterables of reference corpus
    
*   **n\_gram** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – Gram value ranged from 1 to 4
    
*   **smooth** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – Whether to apply smoothing - see \[2\]
    
*   **tokenize** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'none'`, `'13a'`, `'zh'`, `'intl'`, `'char'`, `'ja-mecab'`, `'ko-mecab'`, `'flores101'`, `'flores200'`\]) – Tokenization technique to be used. Choose between `'none'`, `'13a'`, `'zh'`, `'intl'`, `'char'`, `'ja-mecab'`, `'ko-mecab'`, `'flores101'` and `'flores200'`.
    
*   **lowercase** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – If `True`, BLEU score over lowercased text is calculated.
    
*   **weights** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")\]\]) – Weights used for unigrams, bigrams, etc. to calculate BLEU score. If not provided, uniform weights are used.
    

Return type:

[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")

Returns:

Tensor with BLEU Score

Raises:

*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If `preds` and `target` corpus have different lengths.
    
*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If a length of a list of weights is not `None` and not equal to `n_gram`.
    

Example

```python
>>> from torchmetrics.functional.text import sacre_bleu_score
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> sacre_bleu_score(preds, target)
tensor(0.7598)
```

References

\[1\] BLEU: a Method for Automatic Evaluation of Machine Translation by Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu [BLEU](https://www.semanticscholar.org/paper/Bleu%3A-a-Method-for-Automatic-Evaluation-of-Machine-Papineni-Roukos/d7da009f457917aa381619facfa5ffae9329a6e9)

\[2\] A Call for Clarity in Reporting BLEU Scores by Matt Post.

\[3\] Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics by Chin-Yew Lin and Franz Josef Och [Machine Translation Evolution](https://aclanthology.org/P04-1077)

---

**Navigation**

- Parent: [Text](/docs/torchmetrics/1.9.0/text.md)