---
title: Fleiss Kappa
description: >-
  Calculatees Fleiss kappa a statistical measure for inter agreement between
  raters.
canonical: 'https://lightning.ai/docs/torchmetrics/stable/nominal/fleiss_kappa'
---
# Fleiss Kappa {#fleiss-kappa}

## Module Interface {#module-interface}

### `class torchmetrics.nominal.FleissKappa(mode='counts', **kwargs)` {#torchmetrics.nominal.FleissKappa}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/nominal/fleiss_kappa.py#L30-L137)

Calculatees [Fleiss kappa](https://en.wikipedia.org/wiki/Fleiss%27_kappa) a statistical measure for inter agreement between raters.

```math
\kappa = \frac{\bar{p} - \bar{p_e}}{1 - \bar{p_e}}
```

where $`\bar{p}`$ is the mean of the agreement probability over all raters and $`\bar{p_e}`$ is the mean agreement probability over all raters if they were randomly assigned. If the raters are in complete agreement then the score 1 is returned, if there is no agreement among the raters (other than what would be expected by chance) then a score smaller than 0 is returned.

As input to `forward` and `update` the metric accepts the following input:

*   `ratings` ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")): Ratings of shape `[n_samples, n_categories]` or `[n_samples, n_categories, n_raters]` depedenent on `mode`. If `mode` is `counts`, `ratings` must be integer and contain the number of raters that chose each category. If `mode` is `probs`, `ratings` must be floating point and contain the probability/logits that each rater chose each category.
    

As output of `forward` and `compute` the metric returns the following output:

*   `fleiss_k` ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")): A float scalar tensor with the calculated Fleiss’ kappa score.
    

Parameters:

*   **mode** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'counts'`, `'probs'`\]) – Whether ratings will be provided as counts or probabilities.
    
*   **kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any "(in Python v3.15)")) – Additional keyword arguments, see [Advanced metric settings](/docs/torchmetrics/1.9.0/user-guide/overview.md#metric-kwargs) for more info.
    

Example

```python
>>> # Ratings are provided as counts
>>> from torch import randint
>>> from torchmetrics.nominal import FleissKappa
>>> ratings = randint(0, 10, size=(100, 5)).long()  # 100 samples, 5 categories, 10 raters
>>> metric = FleissKappa(mode='counts')
>>> metric(ratings)
tensor(0.0089)
```

Example

```python
>>> # Ratings are provided as probabilities
>>> from torch import randn
>>> from torchmetrics.nominal import FleissKappa
>>> ratings = randn(100, 5, 10).softmax(dim=1)  # 100 samples, 5 categories, 10 raters
>>> metric = FleissKappa(mode='probs')
>>> metric(ratings)
tensor(-0.0075)
```

#### `plot(val=None, ax=None)` {#torchmetrics.nominal.FleissKappa.plot}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/nominal/fleiss_kappa.py#L99-L137)

Plot a single or multiple values from the metric.

Parameters:

*   **val** (`Union`\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)"), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence "(in Python v3.15)")\[[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")\], [`None`](https://docs.python.org/3/builtins/constants.html#None "(in Python v3.15)")\]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.
    
*   **ax** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional "(in Python v3.15)")\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)")\]) – An matplotlib axis object. If provided will add plot to that axis
    

Return type:

[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple "(in Python v3.15)")\[[`Figure`](https://matplotlib.org/stable/api/_as_gen/matplotlib.figure.Figure.html#matplotlib.figure.Figure "(in Matplotlib v3.11.2)"), `Union`\[[`Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html#matplotlib.axes.Axes "(in Matplotlib v3.11.2)"), [`ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html#numpy.ndarray "(in NumPy v2.5)")\]\]

Returns:

Figure and Axes object

Raises:

[**ModuleNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#ModuleNotFoundError "(in Python v3.15)") – If matplotlib is not installed

```python
>>> # Example plotting a single value
>>> import torch
>>> from torchmetrics.nominal import FleissKappa
>>> metric = FleissKappa(mode="probs")
>>> metric.update(torch.randn(100, 5, 10).softmax(dim=1))
>>> fig_, ax_ = metric.plot()
```

[![../../\_images/fleiss\_kappa-1.png](/docs/torchmetrics/1.9.0/static/_images/fleiss_kappa-1.png)](/docs/torchmetrics/1.9.0/static/_images/fleiss_kappa-1.png)

```python
>>> # Example plotting multiple values
>>> import torch
>>> from torchmetrics.nominal import FleissKappa
>>> metric = FleissKappa(mode="probs")
>>> values = [ ]
>>> for _ in range(10):
...     values.append(metric(torch.randn(100, 5, 10).softmax(dim=1)))
>>> fig_, ax_ = metric.plot(values)
```

[![../../\_images/fleiss\_kappa-2.png](/docs/torchmetrics/1.9.0/static/_images/fleiss_kappa-2.png)](/docs/torchmetrics/1.9.0/static/_images/fleiss_kappa-2.png)

## Functional Interface {#functional-interface}

### `torchmetrics.functional.nominal.fleiss_kappa(ratings, mode='counts')` {#torchmetrics.functional.nominal.fleiss_kappa}

[\[source\]](https://github.com/Lightning-AI/torchmetrics/blob/v1.9.0/src/torchmetrics/functional/nominal/fleiss_kappa.py#L61-L99)

Calculatees [Fleiss kappa](https://en.wikipedia.org/wiki/Fleiss%27_kappa) a statistical measure for inter agreement between raters.

```math
\kappa = \frac{\bar{p} - \bar{p_e}}{1 - \bar{p_e}}
```

where $`\bar{p}`$ is the mean of the agreement probability over all raters and $`\bar{p_e}`$ is the mean agreement probability over all raters if they were randomly assigned. If the raters are in complete agreement then the score 1 is returned, if there is no agreement among the raters (other than what would be expected by chance) then a score smaller than 0 is returned.

Parameters:

*   **ratings** ([`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")) – Ratings of shape \[n\_samples, n\_categories\] or \[n\_samples, n\_categories, n\_raters\] depedenent on mode. If mode is counts, ratings must be integer and contain the number of raters that chose each category. If mode is probs, ratings must be floating point and contain the probability/logits that each rater chose each category.
    
*   **mode** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal "(in Python v3.15)")\[`'counts'`, `'probs'`\]) – Whether ratings will be provided as counts or probabilities.
    

Return type:

[`Tensor`](https://docs.pytorch.org/docs/stable/tensors.html#torch.Tensor "(in PyTorch v2.15)")

Example

```python
>>> # Ratings are provided as counts
>>> from torch import randint
>>> from torchmetrics.functional.nominal import fleiss_kappa
>>> ratings = randint(0, 10, size=(100, 5)).long()  # 100 samples, 5 categories, 10 raters
>>> fleiss_kappa(ratings)
tensor(0.0089)
```

Example

```python
>>> # Ratings are provided as probabilities
>>> from torch import randn
>>> from torchmetrics.functional.nominal import fleiss_kappa
>>> ratings = randn(100, 5, 10).softmax(dim=1)  # 100 samples, 5 categories, 10 raters
>>> fleiss_kappa(ratings, mode='probs')
tensor(-0.0075)
```

---

**Navigation**

- Parent: [Nominal](/docs/torchmetrics/1.9.0/nominal.md)