Continuous Ranked Probability Score (CRPS)

Module Interface

class torchmetrics.regression.crps.ContinuousRankedProbabilityScore(**kwargs)

Computes continuous ranked probability score.

\[CRPS(F, y) = \int_{-\infty}^{\infty} (F(x) - 1_{x \geq y})^2 dx\]

where \(F\) is the predicted cumulative distribution function and \(y\) is the true target. The metric is usually used to evaluate probabilistic regression models, such as forecasting models. A lower CRPS indicates a better forecast, meaning that forecasted probabilities are closer to the true observed values. CRPS can also be seen as a generalization of the brier score for non binary classification problems.

As input to forward and update the metric accepts the following input:

• preds (Tensor): Predicted float tensor with shape (N,d)

• target (Tensor): Ground truth float tensor with shape (N,d)

As output of forward and compute the metric returns the following output:

• cosine_similarity (Tensor): A float tensor with the cosine similarity

Parameters:

• reduction – how to reduce over the batch dimension using ‘sum’, ‘mean’ or ‘none’ (taking the individual scores)

• kwargs (Any) – Additional keyword arguments, see Advanced metric settings for more info.

Example

>>> from torch import randn
>>> from torchmetrics.regression import ContinuousRankedProbabilityScore
>>> preds = randn(10, 5)
>>> target = randn(10)
>>> crps = ContinuousRankedProbabilityScore()
>>> crps(preds, target)
tensor(0.7731)

plot(val=None, ax=None)

Plot a single or multiple values from the metric.

Parameters:

• val (Union[Tensor, Sequence[Tensor], None]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.

• ax (Optional[Axes]) – An matplotlib axis object. If provided will add plot to that axis

Return type:

tuple[Figure, Union[Axes, ndarray]]

Returns:

Figure and Axes object

Raises:

ModuleNotFoundError – If matplotlib is not installed

>>> from torch import randn
>>> # Example plotting a single value
>>> from torchmetrics.regression import ContinuousRankedProbabilityScore
>>> metric = ContinuousRankedProbabilityScore()
>>> metric.update(randn(10,5), randn(10))
>>> fig_, ax_ = metric.plot()

>>> from torch import randn
>>> # Example plotting multiple values
>>> from torchmetrics.regression import ContinuousRankedProbabilityScore
>>> metric = ContinuousRankedProbabilityScore()
>>> values = []
>>> for _ in range(10):
...     values.append(metric(randn(10,5), randn(10)))
>>> fig, ax = metric.plot(values)

Functional Interface

torchmetrics.functional.regression.continuous_ranked_probability_score(preds, target)

Computes continuous ranked probability score.

\[CRPS(F, y) = \int_{-\infty}^{\infty} (F(x) - 1_{x \geq y})^2 dx\]

where \(F\) is the predicted cumulative distribution function and \(y\) is the true target. The metric is usually used to evaluate probabilistic regression models, such as forecasting models. A lower CRPS indicates a better forecast, meaning that forecasted probabilities are closer to the true observed values. CRPS can also be seen as a generalization of the brier score for non binary classification problems.

Parameters:

• preds (Tensor) – a 2d tensor of shape (batch_size, ensemble_members) with predictions. The second dimension represents the ensemble members.

• target (Tensor) – a 1d tensor of shape (batch_size) with the target values.

Return type:

Tensor

Returns:

Tensor with CRPS

Raises:

• ValueError – If the number of ensemble members is less than 2.

• ValueError – If the first dimension of preds and target do not match.

Example::

>>> from torchmetrics.functional.regression import continuous_ranked_probability_score
>>> from torch import randn
>>> preds = randn(10, 5)
>>> target = randn(10)
>>> continuous_ranked_probability_score(preds, target)
tensor(0.7731)