---
title: InfoLM
description: Calculate InfoLM.
canonical: 'https://lightning.ai/docs/torchmetrics/stable/text/infolm'
---
# InfoLM {#infolm}

## Module Interface {#module-interface}

### `class torchmetrics.text.infolm.InfoLM(model_name_or_path='bert-base-uncased', temperature=0.25, information_measure='kl_divergence', idf=True, alpha=None, beta=None, device=None, max_length=None, batch_size=64, num_threads=0, verbose=True, return_sentence_level_score=False, **kwargs)` {#torchmetrics.text.infolm.InfoLM}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/text/infolm.py#L42-L267)

Calculate [InfoLM](https://arxiv.org/abs/2112.01589).

InfoLM measures a distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures:

*   [KL divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence)
    
*   [alpha divergence](https://static.renyi.hu/renyi_cikkek/1961_on_measures_of_entropy_and_information.pdf)
    
*   [beta divergence](https://www.sciencedirect.com/science/article/pii/S0047259X08000456)
    
*   [AB divergence](https://pdfs.semanticscholar.org/744b/1166de34cb099100f151f3b1459f141ae25b.pdf)
    
*   [Rényi divergence](https://static.renyi.hu/renyi_cikkek/1961_on_measures_of_entropy_and_information.pdf)
    
*   L1 distance
    
*   L2 distance
    
*   L-infinity distance
    
*   [Fisher-Rao distance](https://arxiv.org/abs/1711.01530)
    

[InfoLM](https://arxiv.org/abs/2112.01589) is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks.

The implementation of this metric is fully based HuggingFace `transformers`’ package.

As input to `forward` and `update` the metric accepts the following input:

*   `preds` (`Sequence`): An iterable of hypothesis corpus
    
*   `target` (`Sequence`): An iterable of reference corpus
    

As output of `forward` and `compute` the metric returns the following output:

*   `infolm` ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")): If return\_sentence\_level\_score=True return a tuple with a tensor with the corpus-level InfoLM score and a list of sentence-level InfoLM scores, else return a corpus-level InfoLM score
    

Parameters:

*   **model\_name\_or\_path** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`PathLike`](https://docs.python.org/3/library/os.html#os.PathLike "(in Python v3.15)")\]) – A name or a model path used to load `transformers` pretrained model. By default the “bert-base-uncased” model is used.
    
*   **temperature** ([`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")) – A temperature for calibrating language modelling. For more information, please reference [InfoLM](https://arxiv.org/abs/2112.01589) paper.
    
*   **information\_measure** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'kl_divergence'`, `'alpha_divergence'`, `'beta_divergence'`, `'ab_divergence'`, `'renyi_divergence'`, `'l1_distance'`, `'l2_distance'`, `'l_infinity_distance'`, `'fisher_rao_distance'`\]) – A name of information measure to be used. Please use one of: \[‘kl\_divergence’, ‘alpha\_divergence’, ‘beta\_divergence’, ‘ab\_divergence’, ‘renyi\_divergence’, ‘l1\_distance’, ‘l2\_distance’, ‘l\_infinity\_distance’, ‘fisher\_rao\_distance’\]
    
*   **idf** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether normalization using inverse document frequencies should be used.
    
*   **alpha** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")\]) – Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures.
    
*   **beta** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")\]) – Beta parameter of the divergence used for beta and AB divergence measures.
    
*   **device** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`device`](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – A device to be used for calculation.
    
*   **max\_length** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")\]) – A maximum length of input sequences. Sequences longer than `max_length` are to be trimmed.
    
*   **batch\_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A batch size used for model processing.
    
*   **num\_threads** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A number of threads to use for a dataloader.
    
*   **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether a progress bar to be displayed during the embeddings calculation.
    
*   **return\_sentence\_level\_score** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication whether a sentence-level InfoLM score to be returned.
    

Example

```python
>>> from torchmetrics.text.infolm import InfoLM
>>> preds = ['he read the book because he was interested in world history']
>>> target = ['he was interested in world history because he read the book']
>>> infolm = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False)
>>> infolm(preds, target)
tensor(-0.1784)
```

#### `plot(val=None, ax=None)` {#torchmetrics.text.infolm.InfoLM.plot}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/text/infolm.py#L225-L267)

Plot a single or multiple values from the metric.

Parameters:

*   **val** (`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.
    
*   **ax** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)")\]) – An matplotlib axis object. If provided will add plot to that axis
    

Return type:

[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`Figure`](https://matplotlib.org/stable/api/_as_gen/matplotlib.figure.Figure.html#matplotlib.figure.Figure "(in Matplotlib v3.11.2)"), `Union`\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)"), [`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray "(in NumPy v2.5)")\]\]

Returns:

Figure and Axes object

Raises:

[**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError "(in Python v3.15)") – If matplotlib is not installed

```python
>>> # Example plotting a single value
>>> from torchmetrics.text.infolm import InfoLM
>>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False)
>>> preds = ['he read the book because he was interested in world history']
>>> target = ['he was interested in world history because he read the book']
>>> metric.update(preds, target)
>>> fig_, ax_ = metric.plot()
```

```python
>>> # Example plotting multiple values
>>> from torchmetrics.text.infolm import InfoLM
>>> metric = InfoLM('google/bert_uncased_L-2_H-128_A-2', idf=False)
>>> preds = ["this is the prediction", "there is an other sample"]
>>> target = ["this is the reference", "there is another one"]
>>> values = [ ]
>>> for _ in range(10):
...     values.append(metric(preds, target))
>>> fig_, ax_ = metric.plot(values)
```

#### `property higher_is_better: bool` {#torchmetrics.text.infolm.InfoLM.higher_is_better}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/text/infolm.py)

Returns a bool indicating whether a higher value of the information measure is better.

Done this way as depends on if the information measure is positive or negative.

## Functional Interface {#functional-interface}

### `torchmetrics.functional.text.infolm.infolm(preds, target, model_name_or_path='bert-base-uncased', temperature=0.25, information_measure='kl_divergence', idf=True, alpha=None, beta=None, device=None, max_length=None, batch_size=64, num_threads=0, verbose=True, return_sentence_level_score=False)` {#torchmetrics.functional.text.infolm.infolm}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/functional/text/infolm.py#L546-L658)

Calculate [InfoLM](https://arxiv.org/abs/2112.01589) \[1\].

InfoML corresponds to distance/divergence between predicted and reference sentence discrete distribution using one of the following information measures:

*   [KL divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence)
    
*   [alpha divergence](https://static.renyi.hu/renyi_cikkek/1961_on_measures_of_entropy_and_information.pdf)
    
*   [beta divergence](https://www.sciencedirect.com/science/article/pii/S0047259X08000456)
    
*   [AB divergence](https://pdfs.semanticscholar.org/744b/1166de34cb099100f151f3b1459f141ae25b.pdf)
    
*   [Rényi divergence](https://static.renyi.hu/renyi_cikkek/1961_on_measures_of_entropy_and_information.pdf)
    
*   L1 distance
    
*   L2 distance
    
*   L-infinity distance
    
*   [Fisher-Rao distance](https://arxiv.org/abs/1711.01530)
    

[InfoLM](https://arxiv.org/abs/2112.01589) is a family of untrained embedding-based metrics which addresses some famous flaws of standard string-based metrics thanks to the usage of pre-trained masked language models. This family of metrics is mainly designed for summarization and data-to-text tasks.

If you want to use IDF scaling over the whole dataset, please use the class metric.

The implementation of this metric is fully based HuggingFace transformers’ package.

Parameters:

*   **preds** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]\]) – An iterable of hypothesis corpus.
    
*   **target** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)")\]\]) – An iterable of reference corpus.
    
*   **model\_name\_or\_path** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`PathLike`](https://docs.python.org/3/library/os.html#os.PathLike "(in Python v3.15)")\]) – A name or a model path used to load transformers pretrained model.
    
*   **temperature** ([`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")) – A temperature for calibrating language modelling. For more information, please reference [InfoLM](https://arxiv.org/abs/2112.01589) paper.
    
*   **information\_measure** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'kl_divergence'`, `'alpha_divergence'`, `'beta_divergence'`, `'ab_divergence'`, `'renyi_divergence'`, `'l1_distance'`, `'l2_distance'`, `'l_infinity_distance'`, `'fisher_rao_distance'`\]) – A name of information measure to be used. Please use one of: \[‘kl\_divergence’, ‘alpha\_divergence’, ‘beta\_divergence’, ‘ab\_divergence’, ‘renyi\_divergence’, ‘l1\_distance’, ‘l2\_distance’, ‘l\_infinity\_distance’, ‘fisher\_rao\_distance’\]
    
*   **idf** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether normalization using inverse document frequencies should be used.
    
*   **alpha** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")\]) – Alpha parameter of the divergence used for alpha, AB and Rényi divergence measures.
    
*   **beta** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`float`](https://docs.python.org/3/builtins/functions.html#float "(in Python v3.15)")\]) – Beta parameter of the divergence used for beta and AB divergence measures.
    
*   **device** (`Union`\[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str "(in Python v3.15)"), [`device`](https://docs.pytorch.org/docs/stable/tensor_attributes.html#torch.device "(in PyTorch v2.15)"), [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – A device to be used for calculation.
    
*   **max\_length** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")\]) – A maximum length of input sequences. Sequences longer than max\_length are to be trimmed.
    
*   **batch\_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A batch size used for model processing.
    
*   **num\_threads** ([`int`](https://docs.python.org/3/builtins/functions.html#int "(in Python v3.15)")) – A number of threads to use for a dataloader.
    
*   **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication of whether a progress bar to be displayed during the embeddings calculation.
    
*   **return\_sentence\_level\_score** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – An indication whether a sentence-level InfoLM score to be returned.
    

Return type:

`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\]\]

Returns:

A corpus-level InfoLM score. (Optionally) A list of sentence-level InfoLM scores if return\_sentence\_level\_score=True.

Example

```python
>>> from torchmetrics.functional.text.infolm import infolm
>>> preds = ['he read the book because he was interested in world history']
>>> target = ['he was interested in world history because he read the book']
>>> infolm(preds, target, model_name_or_path='google/bert_uncased_L-2_H-128_A-2', idf=False)
tensor(-0.1784)
```

References

\[1\] InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation by Pierre Colombo, Chloé Clavel and Pablo Piantanida [InfoLM](https://arxiv.org/abs/2112.01589)

---

**Navigation**

- Parent: [Text](/docs/torchmetrics/1.9.0/text.md)