Exact Match

Module Interface

class torchmetrics.ExactMatch(**kwargs)

Compute Exact match (also known as subset accuracy).

Exact Match is a stricter version of accuracy where all labels have to match exactly for the sample to be correctly classified.

This module is a simple wrapper to get the task specific versions of this metric, which is done by setting the task argument to either 'multiclass' or multilabel. See the documentation of MulticlassExactMatch and MultilabelExactMatch for the specific details of each argument influence and examples.

Legacy Example:

>>> from torch import tensor
>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> metric = ExactMatch(task="multiclass", num_classes=3, multidim_average='global')
>>> metric(preds, target)
tensor(0.5000)

>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> metric = ExactMatch(task="multiclass", num_classes=3, multidim_average='samplewise')
>>> metric(preds, target)
tensor([1., 0.])

static __new__(cls, task, threshold=0.5, num_classes=None, num_labels=None, multidim_average='global', ignore_index=None, validate_args=True, **kwargs)

Initialize task metric.

Return type:

Metric

MulticlassExactMatch

class torchmetrics.classification.MulticlassExactMatch(num_classes, multidim_average='global', ignore_index=None, validate_args=True, **kwargs)

Compute Exact match (also known as subset accuracy) for multiclass tasks.

Exact Match is a stricter version of accuracy where all labels have to match exactly for the sample to be correctly classified.

As input to forward and update the metric accepts the following input:

• preds (Tensor): An int tensor of shape (N, ...) or float tensor of shape (N, C, ..). If preds is a floating point we apply torch.argmax along the C dimension to automatically convert probabilities/logits into an int tensor.

• target (Tensor): An int tensor of shape (N, ...).

As output to forward and compute the metric returns the following output:

• mcem (Tensor): A tensor whose returned shape depends on the multidim_average argument:

  • If multidim_average is set to global the output will be a scalar tensor

  • If multidim_average is set to samplewise the output will be a tensor of shape (N,)

If multidim_average is set to samplewise we expect at least one additional dimension ... to be present, which the reduction will then be applied over instead of the sample dimension N.

Parameters:

• num_classes (int) – Integer specifying the number of labels

• multidim_average (Literal['global', 'samplewise']) –

  Defines how additionally dimensions ... should be handled. Should be one of the following:

  • global: Additional dimensions are flatted along the batch dimension

  • samplewise: Statistic will be calculated independently for each sample on the N axis. The statistics in this case are calculated over the additional dimensions.

• ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

• validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Example (multidim tensors):

>>> from torch import tensor
>>> from torchmetrics.classification import MulticlassExactMatch
>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> metric = MulticlassExactMatch(num_classes=3, multidim_average='global')
>>> metric(preds, target)
tensor(0.5000)

Example (multidim tensors):

>>> from torchmetrics.classification import MulticlassExactMatch
>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> metric = MulticlassExactMatch(num_classes=3, multidim_average='samplewise')
>>> metric(preds, target)
tensor([1., 0.])

plot(val=None, ax=None)

Plot a single or multiple values from the metric.

Parameters:

• val (Union[Tensor, Sequence[Tensor], None]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.

• ax (Optional[Axes]) – An matplotlib axis object. If provided will add plot to that axis

Return type:

Tuple[Figure, Union[Axes, ndarray]]

Returns:

Figure object and Axes object

Raises:

ModuleNotFoundError – If matplotlib is not installed

>>> # Example plotting a single value per class
>>> from torch import randint
>>> from torchmetrics.classification import MulticlassExactMatch
>>> metric = MulticlassExactMatch(num_classes=3)
>>> metric.update(randint(3, (20,5)), randint(3, (20,5)))
>>> fig_, ax_ = metric.plot()

>>> from torch import randint
>>> # Example plotting a multiple values per class
>>> from torchmetrics.classification import MulticlassExactMatch
>>> metric = MulticlassExactMatch(num_classes=3)
>>> values = []
>>> for _ in range(20):
...     values.append(metric(randint(3, (20,5)), randint(3, (20,5))))
>>> fig_, ax_ = metric.plot(values)

MultilabelExactMatch

class torchmetrics.classification.MultilabelExactMatch(num_labels, threshold=0.5, multidim_average='global', ignore_index=None, validate_args=True, **kwargs)

Compute Exact match (also known as subset accuracy) for multilabel tasks.

Exact Match is a stricter version of accuracy where all labels have to match exactly for the sample to be correctly classified.

As input to forward and update the metric accepts the following input:

• preds (Tensor): An int tensor or float tensor of shape (N, C, ..). If preds is a floating point tensor with values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. Additionally, we convert to int tensor with thresholding using the value in threshold.

• target (Tensor): An int tensor of shape (N, C, ...).

As output to forward and compute the metric returns the following output:

• mlem (Tensor): A tensor whose returned shape depends on the multidim_average argument:

  • If multidim_average is set to global the output will be a scalar tensor

  • If multidim_average is set to samplewise the output will be a tensor of shape (N,)

If multidim_average is set to samplewise we expect at least one additional dimension ... to be present, which the reduction will then be applied over instead of the sample dimension N.

Parameters:

• num_labels (int) – Integer specifying the number of labels

• threshold (float) – Threshold for transforming probability to binary (0,1) predictions

• multidim_average (Literal['global', 'samplewise']) –

  Defines how additionally dimensions ... should be handled. Should be one of the following:

  • global: Additional dimensions are flatted along the batch dimension

  • samplewise: Statistic will be calculated independently for each sample on the N axis. The statistics in this case are calculated over the additional dimensions.

• ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

• validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Example (preds is int tensor):

>>> from torch import tensor
>>> from torchmetrics.classification import MultilabelExactMatch
>>> target = tensor([[0, 1, 0], [1, 0, 1]])
>>> preds = tensor([[0, 0, 1], [1, 0, 1]])
>>> metric = MultilabelExactMatch(num_labels=3)
>>> metric(preds, target)
tensor(0.5000)

Example (preds is float tensor):

>>> from torchmetrics.classification import MultilabelExactMatch
>>> target = tensor([[0, 1, 0], [1, 0, 1]])
>>> preds = tensor([[0.11, 0.22, 0.84], [0.73, 0.33, 0.92]])
>>> metric = MultilabelExactMatch(num_labels=3)
>>> metric(preds, target)
tensor(0.5000)

Example (multidim tensors):

>>> from torchmetrics.classification import MultilabelExactMatch
>>> target = tensor([[[0, 1], [1, 0], [0, 1]], [[1, 1], [0, 0], [1, 0]]])
>>> preds = tensor([[[0.59, 0.91], [0.91, 0.99], [0.63, 0.04]],
...                 [[0.38, 0.04], [0.86, 0.780], [0.45, 0.37]]])
>>> metric = MultilabelExactMatch(num_labels=3, multidim_average='samplewise')
>>> metric(preds, target)
tensor([0., 0.])

plot(val=None, ax=None)

Plot a single or multiple values from the metric.

Parameters:

• val (Union[Tensor, Sequence[Tensor], None]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.

• ax (Optional[Axes]) – An matplotlib axis object. If provided will add plot to that axis

Return type:

Tuple[Figure, Union[Axes, ndarray]]

Returns:

Figure and Axes object

Raises:

ModuleNotFoundError – If matplotlib is not installed

>>> # Example plotting a single value
>>> from torch import rand, randint
>>> from torchmetrics.classification import MultilabelExactMatch
>>> metric = MultilabelExactMatch(num_labels=3)
>>> metric.update(randint(2, (20, 3, 5)), randint(2, (20, 3, 5)))
>>> fig_, ax_ = metric.plot()

>>> # Example plotting multiple values
>>> from torch import rand, randint
>>> from torchmetrics.classification import MultilabelExactMatch
>>> metric = MultilabelExactMatch(num_labels=3)
>>> values = [ ]
>>> for _ in range(10):
...     values.append(metric(randint(2, (20, 3, 5)), randint(2, (20, 3, 5))))
>>> fig_, ax_ = metric.plot(values)

Functional Interface

exact_match

torchmetrics.functional.classification.exact_match(preds, target, task, num_classes=None, num_labels=None, threshold=0.5, multidim_average='global', ignore_index=None, validate_args=True)

Compute Exact match (also known as subset accuracy).

Exact Match is a stricter version of accuracy where all classes/labels have to match exactly for the sample to be correctly classified.

This function is a simple wrapper to get the task specific versions of this metric, which is done by setting the task argument to either 'multiclass' or 'multilabel'. See the documentation of multiclass_exact_match() and multilabel_exact_match() for the specific details of each argument influence and examples.

Return type:

Tensor

Legacy Example:

>>> from torch import tensor
>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> exact_match(preds, target, task="multiclass", num_classes=3, multidim_average='global')
tensor(0.5000)

>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> exact_match(preds, target, task="multiclass", num_classes=3, multidim_average='samplewise')
tensor([1., 0.])

multiclass_exact_match

torchmetrics.functional.classification.multiclass_exact_match(preds, target, num_classes, multidim_average='global', ignore_index=None, validate_args=True)

Compute Exact match (also known as subset accuracy) for multiclass tasks.

Exact Match is a stricter version of accuracy where all labels have to match exactly for the sample to be correctly classified.

Accepts the following input tensors:

• preds: (N, ...) (int tensor) or (N, C, ..) (float tensor). If preds is a floating point we apply torch.argmax along the C dimension to automatically convert probabilities/logits into an int tensor.

• target (int tensor): (N, ...)

Parameters:

• preds (Tensor) – Tensor with predictions

• target (Tensor) – Tensor with true labels

• num_classes (int) – Integer specifying the number of labels

• multidim_average (Literal['global', 'samplewise']) –

  Defines how additionally dimensions ... should be handled. Should be one of the following:

  • global: Additional dimensions are flatted along the batch dimension

  • samplewise: Statistic will be calculated independently for each sample on the N axis. The statistics in this case are calculated over the additional dimensions.

• ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

• validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Returns:

• If multidim_average is set to global the output will be a scalar tensor

• If multidim_average is set to samplewise the output will be a tensor of shape (N,)

Return type:

The returned shape depends on the multidim_average argument

Example (multidim tensors):

>>> from torch import tensor
>>> from torchmetrics.functional.classification import multiclass_exact_match
>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> multiclass_exact_match(preds, target, num_classes=3, multidim_average='global')
tensor(0.5000)

Example (multidim tensors):

>>> from torchmetrics.functional.classification import multiclass_exact_match
>>> target = tensor([[[0, 1], [2, 1], [0, 2]], [[1, 1], [2, 0], [1, 2]]])
>>> preds = tensor([[[0, 1], [2, 1], [0, 2]], [[2, 2], [2, 1], [1, 0]]])
>>> multiclass_exact_match(preds, target, num_classes=3, multidim_average='samplewise')
tensor([1., 0.])

multilabel_exact_match

torchmetrics.functional.classification.multilabel_exact_match(preds, target, num_labels, threshold=0.5, multidim_average='global', ignore_index=None, validate_args=True)

Compute Exact match (also known as subset accuracy) for multilabel tasks.

Exact Match is a stricter version of accuracy where all labels have to match exactly for the sample to be correctly classified.

Accepts the following input tensors:

• preds (int or float tensor): (N, C, ...). If preds is a floating point tensor with values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. Additionally, we convert to int tensor with thresholding using the value in threshold.

• target (int tensor): (N, C, ...)

Parameters:

• preds (Tensor) – Tensor with predictions

• target (Tensor) – Tensor with true labels

• num_labels (int) – Integer specifying the number of labels

• threshold (float) – Threshold for transforming probability to binary (0,1) predictions

• multidim_average (Literal['global', 'samplewise']) –

  Defines how additionally dimensions ... should be handled. Should be one of the following:

  • global: Additional dimensions are flatted along the batch dimension

  • samplewise: Statistic will be calculated independently for each sample on the N axis. The statistics in this case are calculated over the additional dimensions.

• ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

• validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Returns:

• If multidim_average is set to global the output will be a scalar tensor

• If multidim_average is set to samplewise the output will be a tensor of shape (N,)

Return type:

The returned shape depends on the multidim_average argument

Example (preds is int tensor):

>>> from torch import tensor
>>> from torchmetrics.functional.classification import multilabel_exact_match
>>> target = tensor([[0, 1, 0], [1, 0, 1]])
>>> preds = tensor([[0, 0, 1], [1, 0, 1]])
>>> multilabel_exact_match(preds, target, num_labels=3)
tensor(0.5000)

Example (preds is float tensor):

>>> from torchmetrics.functional.classification import multilabel_exact_match
>>> target = tensor([[0, 1, 0], [1, 0, 1]])
>>> preds = tensor([[0.11, 0.22, 0.84], [0.73, 0.33, 0.92]])
>>> multilabel_exact_match(preds, target, num_labels=3)
tensor(0.5000)

Example (multidim tensors):

>>> from torchmetrics.functional.classification import multilabel_exact_match
>>> target = tensor([[[0, 1], [1, 0], [0, 1]], [[1, 1], [0, 0], [1, 0]]])
>>> preds = tensor([[[0.59, 0.91], [0.91, 0.99], [0.63, 0.04]],
...                 [[0.38, 0.04], [0.86, 0.780], [0.45, 0.37]]])
>>> multilabel_exact_match(preds, target, num_labels=3, multidim_average='samplewise')
tensor([0., 0.])