{"id":5648357,"date":"2023-07-04T06:46:00","date_gmt":"2023-07-04T10:46:00","guid":{"rendered":"https:\/\/lightning.ai\/pages\/?p=5648357"},"modified":"2023-07-05T07:12:27","modified_gmt":"2023-07-05T11:12:27","slug":"torchmetrics-v1.0-100-metrics-and-plotting","status":"publish","type":"post","link":"https:\/\/lightning.ai\/pages\/blog\/torchmetrics-v1.0-100-metrics-and-plotting\/","title":{"rendered":"Torchmetrics v1.0: Visualize model performance with 100+ metrics"},"content":{"rendered":"
\n
\n
\n
\n
\n
\n

We are happy to announce that the first major release of TorchMetrics<\/a>, version v1.0, is now publicly available. We have worked hard on a couple of new features for this milestone release, and for v1.0.0, we have also managed to implement over 100<\/strong> metrics in torchmetrics.<\/p>\n

In this blogpost we go over some of the new features and improvements to the library.<\/strong><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

<\/h2>\n

<\/a>Plotting<\/h2>\n

The big new feature of v1.0 is a build-in plotting feature. As the old saying goes:\u00a0“A picture is worth a thousand words”<\/em>. Within machine learning, this is definitely also true for many things. Metrics are one area that in some cases are definitely better showcased in a figure than as list of floats. The only requirement for getting started with the plotting feature is installing\u00a0matplotlib<\/code>. Either install with\u00a0pip install matplotlib<\/code>\u00a0or\u00a0pip install torchmetrics[visual]<\/code>\u00a0(the latter option also installs\u00a0Scienceplots<\/a>\u00a0and uses that as default plotting style).<\/p>\n

The basic interface is the same for\u00a0any<\/strong>\u00a0metric. Just call the new\u00a0.plot<\/code>\u00a0method:<\/p>\n

metric<\/span> =<\/span> AnyMetricYouLike<\/span>()\r\nfor<\/span> _<\/span> in<\/span> range<\/span>(num_updates<\/span>):\r\n    metric<\/span>.update<\/span>(preds<\/span>[i<\/span>], target<\/span>[i<\/span>])\r\nfig<\/span>, ax<\/span> =<\/span> metric<\/span>.plot<\/span>()<\/pre>\n
<\/div>\n
The\u00a0plot<\/code>\u00a0method by default does not require any arguments and will automatically call\u00a0metric.compute<\/code>\u00a0internally on whatever metric states have been accumulated.<\/p>\n
At the most basic level, the plot method can be used to used to plot the value from a single step:<\/p>\n
\n
metric<\/span> =<\/span> torchmetrics<\/span>.Accuracy<\/span>(task<\/span>=<\/span>\"binary\"<\/span>)\r\nfor<\/span> _<\/span> in<\/span> range<\/span>(num_updates<\/span>):\r\n    metric<\/span>.update<\/span>(torch<\/span>.rand<\/span>(10<\/span>,), torch<\/span>.randint<\/span>(2<\/span>, (10<\/span>,)))\r\nfig<\/span>, ax<\/span> =<\/span> metric<\/span>.plot<\/span>()\r\n\r\n<\/pre>\n<\/div>\n
which would generate a plot like<\/pre>\n
\"plot\"<\/a><\/p>\n
A single point plot is not that informative in itself, but if available we will try to include additional information such as the lower and upper bounds the particular metric can take and if the metric should be minimized or maximized to be optimal. For experience users this may not have that big of an value, but for newcomers to machine learning or just somebody wanting to learn about a new metric, this can be beneficial.<\/p>\n
If a metric returns multiple values these can also be plotted, such that each value will be plotted individually and labeled:<\/p>\n
\n
metric<\/span> =<\/span> torchmetrics<\/span>.Accuracy<\/span>(task<\/span>=<\/span>\"multiclass\"<\/span>, num_classes<\/span>=<\/span>3<\/span>, average<\/span>=<\/span>None<\/span>)\r\nfor<\/span> _<\/span> in<\/span> range<\/span>(num_updates<\/span>):\r\n    metric<\/span>.update<\/span>(torch<\/span>.randint<\/span>(3<\/span>, (10<\/span>,)), torch<\/span>.randint<\/span>(3<\/span>, (10<\/span>,)))\r\nfig<\/span>, ax<\/span> =<\/span> metric<\/span>.plot<\/span>()<\/pre>\n
<\/div>\n<\/div>\n
\"plot\"<\/a><\/p>\n
Finally, common practice is to plot a sequence of metric outputs like the progress over a training run. The\u00a0.plot<\/code>\u00a0method support the user providing such a sequence of outputs, which will then be plotted as a line:ne:<\/p>\n
\n
metric<\/span> =<\/span> torchmetrics<\/span>.Accuracy<\/span>(task<\/span>=<\/span>\"binary\"<\/span>)\r\nvalues<\/span> =<\/span> [ ]\r\nfor<\/span> step<\/span> in<\/span> range<\/span>(num_steps<\/span>):\r\n    for<\/span> _<\/span> in<\/span> range<\/span>(num_updates<\/span>):\r\n        metric<\/span>.update<\/span>(preds<\/span>(step<\/span>), target<\/span>(step<\/span>))\r\n    values<\/span>.append<\/span>(metric<\/span>.compute<\/span>())  # save value<\/span>\r\n    metric<\/span>.reset<\/span>()\r\nfig<\/span>, ax<\/span> =<\/span> metric<\/span>.plot<\/span>(values<\/span>)<\/pre>\n
<\/div>\n<\/div>\n
\"plot\"<\/a><\/p>\n
Where the plotting feature really shines are metrics that are not represented by a single or a few numerical values. Two good examples are the classical\u00a0ConfusionMatrix<\/code>\u00a0and\u00a0ROC<\/code>\u00a0metrics. For metrics like these, we have written custom plot routines to generate the appropriate plots, with the interface still only being the\u00a0.plot<\/code>\u00a0method:<\/p>\n
\n
metric<\/span> =<\/span> torchmetrics<\/span>.ConfusionMatrix<\/span>(task<\/span>=<\/span>\"multiclass\"<\/span>, num_classes<\/span>=<\/span>3<\/span>)\r\nfor<\/span> _<\/span> in<\/span> range<\/span>(N<\/span>):\r\n    metric<\/span>.update<\/span>(torch<\/span>.randint<\/span>(3<\/span>, (10<\/span>,)), torch<\/span>.randint<\/span>(3<\/span>, (10<\/span>,)))\r\nmetric<\/span>.plot<\/span>(ax<\/span>=<\/span>ax<\/span>)<\/pre>\n
<\/div>\n<\/div>\n
\"plot\"<\/a><\/p>\n
\n
metric<\/span> =<\/span> torchmetrics<\/span>.ROC<\/span>(task<\/span>=<\/span>\"binary\"<\/span>)\r\nfor<\/span> _<\/span> in<\/span> range<\/span>(N<\/span>):\r\n    metric<\/span>.update<\/span>(torch<\/span>.rand<\/span>(10<\/span>), torch<\/span>.randint<\/span>(2<\/span>, (10<\/span>,)))\r\nmetric<\/span>.plot<\/span>(ax<\/span>=<\/span>ax<\/span>)<\/pre>\n
<\/div>\n<\/div>\n
\"plot\"<\/a><\/p>\n
This feature is not meant (for now) to create publishing ready figures out of the box. Instead it is meant as an quick-and-dirty way to visualize any metric in Torchmetrics. If you have suggestion for how to improve the quality of the plots or make them more informative in some way, feel free to reach out to us.<\/p>\n
For a more explanation on this feature, visit our documentation:\u00a0https:\/\/torchmetrics.readthedocs.io\/en\/latest\/pages\/plotting.html<\/a><\/p>\n
<\/a>New metrics<\/h2>\n
As already stated, for Torchmetrics v1.0 we have also crossed the milestone of having over 100 unique metrics. These metrics are divided between 12 different sub categories of metrics. Since the last release these 15 new metrics have been added to Torchmetrics:<\/p>\n