---
title: KL Divergence
description: Compute the KL divergence.
canonical: 'https://lightning.ai/docs/torchmetrics/stable/regression/kl_divergence'
---
# KL Divergence {#kl-divergence}

## Module Interface {#module-interface}

### `class torchmetrics.KLDivergence(log_prob=False, reduction='mean', **kwargs)` {#torchmetrics.KLDivergence}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/regression/kl_divergence.py#L31-L172)

Compute the [KL divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence).

```math
D_{KL}(P||Q) = \sum_{x\in\mathcal{X}} P(x) \log\frac{P(x)}{Q{x}}
```

Where $`P`$ and $`Q`$ are probability distributions where $`P`$ usually represents a distribution over data and $`Q`$ is often a prior or approximation of $`P`$. It should be noted that the KL divergence is a non-symmetrical metric i.e. $`D_{KL}(P||Q) \neq D_{KL}(Q||P)`$.

As input to `forward` and `update` the metric accepts the following input:

*   `p` ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")): a data distribution with shape `(N, d)`
    
*   `q` ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")): prior or approximate distribution with shape `(N, d)`
    

As output of `forward` and `compute` the metric returns the following output:

*   `kl_divergence` ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")): A tensor with the KL divergence
    

Parameters:

*   **log\_prob** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – bool indicating if input is log-probabilities or probabilities. If given as probabilities, will normalize to make sure the distributes sum to 1.
    
*   **reduction** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'mean'`, `'sum'`, `'none'`, `None`\]) –
    
    Determines how to reduce over the `N`/batch dimension:
    
    *   `'mean'` \[default\]: Averages score across samples
        
    *   `'sum'`: Sum score across samples
        
    *   `'none'` or `None`: Returns score per sample
        
    
*   **kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – Additional keyword arguments, see [Advanced metric settings](/docs/torchmetrics/1.9.0/user-guide/overview.md#metric-kwargs) for more info.
    

Raises:

*   [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError "(in Python v3.15)") – If `log_prob` is not an `bool`.
    
*   [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError "(in Python v3.15)") – If `reduction` is not one of `'mean'`, `'sum'`, `'none'` or `None`.
    

Attention

Half precision is only support on GPU for this metric.

Example

```python
>>> from torch import tensor
>>> from torchmetrics.regression import KLDivergence
>>> p = tensor([[0.36, 0.48, 0.16]])
>>> q = tensor([[1/3, 1/3, 1/3]])
>>> kl_divergence = KLDivergence()
>>> kl_divergence(p, q)
tensor(0.0853)
```

#### `plot(val=None, ax=None)` {#torchmetrics.KLDivergence.plot}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/regression/kl_divergence.py#L132-L172)

Plot a single or multiple values from the metric.

Parameters:

*   **val** (`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.
    
*   **ax** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)")\]) – An matplotlib axis object. If provided will add plot to that axis
    

Return type:

[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`Figure`](https://matplotlib.org/stable/api/_as_gen/matplotlib.figure.Figure.html#matplotlib.figure.Figure "(in Matplotlib v3.11.2)"), `Union`\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)"), [`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray "(in NumPy v2.5)")\]\]

Returns:

Figure and Axes object

Raises:

[**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError "(in Python v3.15)") – If matplotlib is not installed

```python
>>> from torch import randn
>>> # Example plotting a single value
>>> from torchmetrics.regression import KLDivergence
>>> metric = KLDivergence()
>>> metric.update(randn(10,3).softmax(dim=-1), randn(10,3).softmax(dim=-1))
>>> fig_, ax_ = metric.plot()
```

[![../../\_images/kl\_divergence-1.png](/docs/torchmetrics/1.9.0/static/_images/kl_divergence-1.png)](/docs/torchmetrics/1.9.0/static/_images/kl_divergence-1.png)

```python
>>> from torch import randn
>>> # Example plotting multiple values
>>> from torchmetrics.regression import KLDivergence
>>> metric = KLDivergence()
>>> values = []
>>> for _ in range(10):
...     values.append(metric(randn(10,3).softmax(dim=-1), randn(10,3).softmax(dim=-1)))
>>> fig, ax = metric.plot(values)
```

[![../../\_images/kl\_divergence-2.png](/docs/torchmetrics/1.9.0/static/_images/kl_divergence-2.png)](/docs/torchmetrics/1.9.0/static/_images/kl_divergence-2.png)

## Functional Interface {#functional-interface}

### `torchmetrics.functional.kl_divergence(p, q, log_prob=False, reduction='mean')` {#torchmetrics.functional.kl_divergence}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/functional/regression/kl_divergence.py#L82-L115)

Compute [KL divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence).

```math
D_{KL}(P||Q) = \sum_{x\in\mathcal{X}} P(x) \log\frac{P(x)}{Q{x}}
```

Where $`P`$ and $`Q`$ are probability distributions where $`P`$ usually represents a distribution over data and $`Q`$ is often a prior or approximation of $`P`$. It should be noted that the KL divergence is a non-symmetrical metric i.e. $`D_{KL}(P||Q) \neq D_{KL}(Q||P)`$.

Parameters:

*   **p** ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")) – data distribution with shape `[N, d]`
    
*   **q** ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")) – prior or approximate distribution with shape `[N, d]`
    
*   **log\_prob** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool "(in Python v3.15)")) – bool indicating if input is log-probabilities or probabilities. If given as probabilities, will normalize to make sure the distributes sum to 1
    
*   **reduction** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'mean'`, `'sum'`, `'none'`, `None`\]) –
    
    Determines how to reduce over the `N`/batch dimension:
    
    *   `'mean'` \[default\]: Averages score across samples
        
    *   `'sum'`: Sum score across samples
        
    *   `'none'` or `None`: Returns score per sample
        
    

Return type:

[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")

Example

```python
>>> from torch import tensor
>>> p = tensor([[0.36, 0.48, 0.16]])
>>> q = tensor([[1/3, 1/3, 1/3]])
>>> kl_divergence(p, q)
tensor(0.0853)
```

---

**Navigation**

- Parent: [Regression](/docs/torchmetrics/1.9.0/regression.md)