Mean-Average-Precision (mAP)

Module Interface

class torchmetrics.detection.mean_ap.MeanAveragePrecision(box_format='xyxy', iou_type='bbox', iou_thresholds=None, rec_thresholds=None, max_detection_thresholds=None, class_metrics=False, extended_summary=False, average='macro', backend='pycocotools', **kwargs)

Compute the Mean-Average-Precision (mAP) and Mean-Average-Recall (mAR) for object detection predictions.

\[\text{mAP} = \frac{1}{n} \sum_{i=1}^{n} AP_i\]

where \(AP_i\) is the average precision for class \(i\) and \(n\) is the number of classes. The average precision is defined as the area under the precision-recall curve. For object detection the recall and precision are defined based on the intersection of union (IoU) between the predicted bounding boxes and the ground truth bounding boxes e.g. if two boxes have an IoU > t (with t being some threshold) they are considered a match and therefore considered a true positive. The precision is then defined as the number of true positives divided by the number of all detected boxes and the recall is defined as the number of true positives divided by the number of all ground boxes.

As input to forward and update the metric accepts the following input:

• preds (List): A list consisting of dictionaries each containing the key-values (each dictionary corresponds to a single image). Parameters that should be provided per dict

  • boxes (Tensor): float tensor of shape (num_boxes, 4) containing num_boxes detection boxes of the format specified in the constructor. By default, this method expects (xmin, ymin, xmax, ymax) in absolute image coordinates, but can be changed using the box_format parameter. Only required when iou_type=”bbox”.

  • scores (Tensor): float tensor of shape (num_boxes) containing detection scores for the boxes.

  • labels (Tensor): integer tensor of shape (num_boxes) containing 0-indexed detection classes for the boxes.

  • masks (Tensor): boolean tensor of shape (num_boxes, image_height, image_width) containing boolean masks. Only required when iou_type=”segm”.

• target (List): A list consisting of dictionaries each containing the key-values (each dictionary corresponds to a single image). Parameters that should be provided per dict:

  • boxes (Tensor): float tensor of shape (num_boxes, 4) containing num_boxes ground truth boxes of the format specified in the constructor. only required when iou_type=”bbox”. By default, this method expects (xmin, ymin, xmax, ymax) in absolute image coordinates.

  • labels (Tensor): integer tensor of shape (num_boxes) containing 0-indexed ground truth classes for the boxes.

  • masks (Tensor): boolean tensor of shape (num_boxes, image_height, image_width) containing boolean masks. Only required when iou_type=”segm”.

  • iscrowd (Tensor): integer tensor of shape (num_boxes) containing 0/1 values indicating whether the bounding box/masks indicate a crowd of objects. Value is optional, and if not provided it will automatically be set to 0.

  • area (Tensor): float tensor of shape (num_boxes) containing the area of the object. Value is optional, and if not provided will be automatically calculated based on the bounding box/masks provided. Only affects which samples contribute to the map_small, map_medium, map_large values

As output of forward and compute the metric returns the following output:

• map_dict: A dictionary containing the following key-values:

  • map: (Tensor), global mean average precision which by default is defined as mAP50-95 e.g. the mean average precision for IoU thresholds 0.50, 0.55, 0.60, …, 0.95 averaged over all classes and areas. If the IoU thresholds are changed this value will be calculated with the new thresholds.

  • map_small: (Tensor), mean average precision for small objects (area < 32^2 pixels)

  • map_medium:(Tensor), mean average precision for medium objects (32^2 pixels < area < 96^2 pixels)

  • map_large: (Tensor), mean average precision for large objects (area > 96^2 pixels)

  • mar_{mdt[0]}: (Tensor), mean average recall for max_detection_thresholds[0] (default 1) detection per image

  • mar_{mdt[1]}: (Tensor), mean average recall for max_detection_thresholds[1] (default 10) detection per image

  • mar_{mdt[1]}: (Tensor), mean average recall for max_detection_thresholds[2] (default 100) detection per image

  • mar_small: (Tensor), mean average recall for small objects (area < 32^2 pixels)

  • mar_medium: (Tensor), mean average recall for medium objects (32^2 pixels < area < 96^2 pixels)

  • mar_large: (Tensor), mean average recall for large objects (area > 96^2 pixels)

  • map_50: (Tensor) (-1 if 0.5 not in the list of iou thresholds), mean average precision at IoU=0.50

  • map_75: (Tensor) (-1 if 0.75 not in the list of iou thresholds), mean average precision at IoU=0.75

  • map_per_class: (Tensor) (-1 if class metrics are disabled), mean average precision per observed class

  • mar_{mdt[2]}_per_class: (Tensor) (-1 if class metrics are disabled), mean average recall for max_detection_thresholds[2] (default 100) detections per image per observed class

  • classes (Tensor), list of all observed classes

For an example on how to use this metric check the torchmetrics mAP example.

Attention

The map score is calculated with @[ IoU=self.iou_thresholds | area=all | max_dets=max_detection_thresholds ] e.g. the mean average precision for IoU thresholds 0.50, 0.55, 0.60, …, 0.95 averaged over all classes and all areas and all max detections per image. If the IoU thresholds are changed this value will be calculated with the new thresholds. Caution: If the initialization parameters are changed, dictionary keys for mAR can change as well.

Important

This metric supports, at the moment, two different backends for the evaluation. The default backend is "pycocotools", which either require the official pycocotools implementation or this fork of pycocotools to be installed. We recommend using the fork as it is better maintained and easily available to install via pip: pip install pycocotools. It is also this fork that will be installed if you install torchmetrics[detection]. The second backend is the faster-coco-eval implementation, which can be installed with pip install faster-coco-eval. This implementation is a maintained open-source implementation that is faster and corrects certain corner cases that the official implementation has. Our own testing has shown that the results are identical to the official implementation. Regardless of the backend we also require you to have torchvision version 0.8.0 or newer installed. Please install with pip install torchvision>=0.8 or pip install torchmetrics[detection].

Parameters:

• box_format (Literal['xyxy', 'xywh', 'cxcywh']) –

  Input format of given boxes. Supported formats are:

  • ’xyxy’: boxes are represented via corners, x1, y1 being top left and x2, y2 being bottom right.

  • ’xywh’ : boxes are represented via corner, width and height, x1, y2 being top left, w, h being width and height. This is the default format used by pycoco and all input formats will be converted to this.

  • ’cxcywh’: boxes are represented via centre, width and height, cx, cy being center of box, w, h being width and height.

• iou_type (Union[Literal['bbox', 'segm'], tuple[str]]) – Type of input (either masks or bounding-boxes) used for computing IOU. Supported IOU types are "bbox" or "segm" or both as a tuple.

• iou_thresholds (Optional[list[float]]) – IoU thresholds for evaluation. If set to None it corresponds to the stepped range [0.5,...,0.95] with step 0.05. Else provide a list of floats.

• rec_thresholds (Optional[list[float]]) – Recall thresholds for evaluation. If set to None it corresponds to the stepped range [0,...,1] with step 0.01. Else provide a list of floats.

• max_detection_thresholds (Optional[list[int]]) – Thresholds on max detections per image. If set to None will use thresholds [1, 10, 100]. Else, please provide a list of ints of length 3, which is the only supported length by both backends.

• class_metrics (bool) – Option to enable per-class metrics for mAP and mAR_100. Has a performance impact that scales linearly with the number of classes in the dataset.

• extended_summary (bool) –

  Option to enable extended summary with additional metrics including IOU, precision and recall. The output dictionary will contain the following extra key-values:

  • ious: a dictionary containing the IoU values for every image/class combination e.g. ious[(0,0)] would contain the IoU for image 0 and class 0. Each value is a tensor with shape (n,m) where n is the number of detections and m is the number of ground truth boxes for that image/class combination.

  • precision: a tensor of shape (TxRxKxAxM) containing the precision values. Here T is the number of IoU thresholds, R is the number of recall thresholds, K is the number of classes, A is the number of areas and M is the number of max detections per image.

  • recall: a tensor of shape (TxKxAxM) containing the recall values. Here T is the number of IoU thresholds, K is the number of classes, A is the number of areas and M is the number of max detections per image.

  • scores: a tensor of shape (TxRxKxAxM) containing the confidence scores. Here T is the number of IoU thresholds, R is the number of recall thresholds, K is the number of classes, A is the number of areas and M is the number of max detections per image.

• average (Literal['macro', 'micro']) – Method for averaging scores over labels. Choose between “``”macro”`` and "micro".

• backend (Literal['pycocotools', 'faster_coco_eval']) – Backend to use for the evaluation. Choose between "pycocotools" and "faster_coco_eval".

• kwargs (Any) – Additional keyword arguments, see Advanced metric settings for more info.

Raises:

• ModuleNotFoundError – If pycocotools is not installed

• ModuleNotFoundError – If torchvision is not installed or version installed is lower than 0.8.0

• ValueError – If box_format is not one of "xyxy", "xywh" or "cxcywh"

• ValueError – If iou_type is not one of "bbox" or "segm"

• ValueError – If iou_thresholds is not None or a list of floats

• ValueError – If rec_thresholds is not None or a list of floats

• ValueError – If max_detection_thresholds is not None or a list of ints

• ValueError – If class_metrics is not a boolean

Example:

Basic example for when `iou_type="bbox"`. In this case the ``boxes`` key is required in the input dictionaries,
in addition to the ``scores`` and ``labels`` keys.

>>> from torch import tensor
>>> from torchmetrics.detection import MeanAveragePrecision
>>> preds = [
...   dict(
...     boxes=tensor([[258.0, 41.0, 606.0, 285.0]]),
...     scores=tensor([0.536]),
...     labels=tensor([0]),
...   )
... ]
>>> target = [
...   dict(
...     boxes=tensor([[214.0, 41.0, 562.0, 285.0]]),
...     labels=tensor([0]),
...   )
... ]
>>> metric = MeanAveragePrecision(iou_type="bbox")
>>> metric.update(preds, target)
>>> from pprint import pprint
>>> pprint(metric.compute())
{'classes': tensor(0, dtype=torch.int32),
 'map': tensor(0.6000),
 'map_50': tensor(1.),
 'map_75': tensor(1.),
 'map_large': tensor(0.6000),
 'map_medium': tensor(-1.),
 'map_per_class': tensor(-1.),
 'map_small': tensor(-1.),
 'mar_1': tensor(0.6000),
 'mar_10': tensor(0.6000),
 'mar_100': tensor(0.6000),
 'mar_100_per_class': tensor(-1.),
 'mar_large': tensor(0.6000),
 'mar_medium': tensor(-1.),
 'mar_small': tensor(-1.)}

Example:

Basic example for when `iou_type="segm"`. In this case the ``masks`` key is required in the input dictionaries,
in addition to the ``scores`` and ``labels`` keys.

>>> from torch import tensor
>>> from torchmetrics.detection import MeanAveragePrecision
>>> mask_pred = [
...   [0, 0, 0, 0, 0],
...   [0, 0, 1, 1, 0],
...   [0, 0, 1, 1, 0],
...   [0, 0, 0, 0, 0],
...   [0, 0, 0, 0, 0],
... ]
>>> mask_tgt = [
...   [0, 0, 0, 0, 0],
...   [0, 0, 1, 0, 0],
...   [0, 0, 1, 1, 0],
...   [0, 0, 1, 0, 0],
...   [0, 0, 0, 0, 0],
... ]
>>> preds = [
...   dict(
...     masks=tensor([mask_pred], dtype=torch.bool),
...     scores=tensor([0.536]),
...     labels=tensor([0]),
...   )
... ]
>>> target = [
...   dict(
...     masks=tensor([mask_tgt], dtype=torch.bool),
...     labels=tensor([0]),
...   )
... ]
>>> metric = MeanAveragePrecision(iou_type="segm")
>>> metric.update(preds, target)
>>> from pprint import pprint
>>> pprint(metric.compute())
{'classes': tensor(0, dtype=torch.int32),
 'map': tensor(0.2000),
 'map_50': tensor(1.),
 'map_75': tensor(0.),
 'map_large': tensor(-1.),
 'map_medium': tensor(-1.),
 'map_per_class': tensor(-1.),
 'map_small': tensor(0.2000),
 'mar_1': tensor(0.2000),
 'mar_10': tensor(0.2000),
 'mar_100': tensor(0.2000),
 'mar_100_per_class': tensor(-1.),
 'mar_large': tensor(-1.),
 'mar_medium': tensor(-1.),
 'mar_small': tensor(0.2000)}

static coco_to_tm(coco_preds, coco_target, iou_type='bbox', backend='pycocotools')

Utility function for converting .json coco format files to the input format of this metric.

The function accepts a file for the predictions and a file for the target in coco format and converts them to a list of dictionaries containing the boxes, labels and scores in the input format of this metric.

Parameters:

• coco_preds (str) – Path to the json file containing the predictions in coco format

• coco_target (str) – Path to the json file containing the targets in coco format

• iou_type (Union[Literal['bbox', 'segm'], list[str]]) – Type of input, either bbox for bounding boxes or segm for segmentation masks

• backend (Literal['pycocotools', 'faster_coco_eval']) – Backend to use for the conversion. Either pycocotools or faster_coco_eval.

Return type:

tuple[list[dict[str, Tensor]], list[dict[str, Tensor]]]

Returns:

A tuple containing the predictions and targets in the input format of this metric. Each element of the tuple is a list of dictionaries containing the boxes, labels and scores.

Example

>>> # File formats are defined at https://cocodataset.org/#format-data
>>> # Example files can be found at
>>> # https://github.com/cocodataset/cocoapi/tree/master/results
>>> from torchmetrics.detection import MeanAveragePrecision
>>> preds, target = MeanAveragePrecision.coco_to_tm(
...   "instances_val2014_fakebbox100_results.json",
...   "val2014_fake_eval_res.txt.json"
...   iou_type="bbox"
... )  

plot(val=None, ax=None)

Plot a single or multiple values from the metric.

Parameters:

• val (Union[dict[str, Tensor], Sequence[dict[str, Tensor]], None]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.

• ax (Optional[Axes]) – An matplotlib axis object. If provided will add plot to that axis

Return type:

tuple[Figure, Union[Axes, ndarray]]

Returns:

Figure object and Axes object

Raises:

ModuleNotFoundError – If matplotlib is not installed

>>> from torch import tensor
>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision
>>> preds = [dict(
...     boxes=tensor([[258.0, 41.0, 606.0, 285.0]]),
...     scores=tensor([0.536]),
...     labels=tensor([0]),
... )]
>>> target = [dict(
...     boxes=tensor([[214.0, 41.0, 562.0, 285.0]]),
...     labels=tensor([0]),
... )]
>>> metric = MeanAveragePrecision()
>>> metric.update(preds, target)
>>> fig_, ax_ = metric.plot()

>>> # Example plotting multiple values
>>> import torch
>>> from torchmetrics.detection.mean_ap import MeanAveragePrecision
>>> preds = lambda: [dict(
...     boxes=torch.tensor([[258.0, 41.0, 606.0, 285.0]]) + torch.randint(10, (1,4)),
...     scores=torch.tensor([0.536]) + 0.1*torch.rand(1),
...     labels=torch.tensor([0]),
... )]
>>> target = [dict(
...     boxes=torch.tensor([[214.0, 41.0, 562.0, 285.0]]),
...     labels=torch.tensor([0]),
... )]
>>> metric = MeanAveragePrecision()
>>> vals = []
>>> for _ in range(20):
...     vals.append(metric(preds(), target))
>>> fig_, ax_ = metric.plot(vals)

tm_to_coco(name='tm_map_input')

Utility function for converting the input for this metric to coco format and saving it to a json file.

This function should be used after calling .update(…) or .forward(…) on all data that should be written to the file, as the input is then internally cached. The function then converts to information to coco format a writes it to json files.

Parameters:

name (str) – Name of the output file, which will be appended with “_preds.json” and “_target.json”

Return type:

None

Example

>>> from torch import tensor
>>> from torchmetrics.detection import MeanAveragePrecision
>>> preds = [
...   dict(
...     boxes=tensor([[258.0, 41.0, 606.0, 285.0]]),
...     scores=tensor([0.536]),
...     labels=tensor([0]),
...   )
... ]
>>> target = [
...   dict(
...     boxes=tensor([[214.0, 41.0, 562.0, 285.0]]),
...     labels=tensor([0]),
...   )
... ]
>>> metric = MeanAveragePrecision(iou_type="bbox")
>>> metric.update(preds, target)
>>> metric.tm_to_coco("tm_map_input")

property coco: object

Returns the coco module for the given backend, done in this way to make metric picklable.

property cocoeval: object

Returns the coco eval module for the given backend, done in this way to make metric picklable.

property mask_utils: object

Returns the mask utils object for the given backend, done in this way to make metric picklable.