Lightning AI Studios: Never set up a local environment again →

Log in or create a free Lightning.ai account to track your progress and access additional course materials  

Deep Learning Fundamentals

  • Pages

    • Final certification exam

    Deep Learning Fundamentals > Unit 5 > Unit 5.3
    Course Progress:

    5.3 Computing Metrics Efficiently with TorchMetrics

    Slides

    References

    Code

    What we covered in this video lecture

    In this lecture, we introduced TorchMetrics, which lets us compute the loss incrementally as new data arrives, which is useful for updating the loss batch-by-batch during training and calculating the validation and test set accuracy using PyTorch DataLoaders that fetch the data incrementally one batch at a time.

    Additional resources if you want to learn more

    As of this writing, there are now more than 90 different metrics implemented in TorchMetrics. While we are mostly working with classification accuracy and mean squared error measures in this course, you may have more specialized use cases that require different metrics. If you want to find out what’s currently implemented, check out the official documentation.

    If you are curious about understanding the difference between the .update() and .forward() methods in TorchMetrics, you may also like the hands-on examples in my blog post here.

    Log in or create a free Lightning.ai account to access:

    • Quizzes
    • Completion badges
    • Progress tracking
    • Additional downloadable content
    • Additional AI education resources
    • Notifications when new units are released
    • Free cloud computing credits

    Quiz: 5.3 Computing Metrics Efficiently with TorchMetrics (Part 1)

    Loading the data one batch at a time…

    Incorrect. Since the steps for loading and preprocessing the data are the same, loading the data in batches does not lower the total computational cost. On the contrary, it may even increase the total computation time as we may have to do the operation repeatedly. (E.g., “2 + 4” is cheaper than “2 + 2 + 2”.)

    Correct. In deep learning we often work with large datasets that don’t fit into computer or GPU memory.

    Incorrect. When we are using PyTorch DataLoaders, we still have to store our data somewhere.

    Please answer all questions to proceed.

    Quiz: 5.3 Computing Metrics Efficiently with TorchMetrics (Part 2)

    Suppose you initialized an accuracy instance from TorchMetrics:
    self.val_acc = torchmetrics.Accuracy(task="multiclass", num_classes=10)
    You plan to use it for tracking the validation set accuracy in the validation_step method of a LightningModule. Which of the following are correct ways of computing the validation set accuracy in the validation_step method?

    Correct. This is a correct way of computing the validation set accuracy for a given batch and DataLoader. It’s a shortcut for self.val_acc.forward

    Correct. Like PyTorch modules, this is a more verbose form and equivalent to directly calling self.val_acc(...).

    Incorrect. This is not a valid syntax for computing the validation accuracy via the accuracy instance. TorchMetrics follows the PyTorch syntax using a .forward() method.

    Please answer all questions to proceed.

    Quiz: 5.3 Computing Metrics Efficiently with TorchMetrics (Part 3)

    When we added the test set accuracy calculation in the code, we used a new self.test_acc = torchmetrics.Accuracy() object.

    Could we have reused the validation set accuracy tracker in the test step?

    Correct. Since we are not tracking the validation and test metrics simultaneously, we could have used the same accuracy object here. So, validation and test set trackers don’t necessarily need to be kept separate, although it’s nice for organization purposes and logging later. However, validation and training set trackers need to be kept separate because we use both simultaneously during training. This means that we compute the validation accuracy between epochs while tracking the accuracy across epochs.

    Incorrect. Since we are not tracking the validation and test metrics simultaneously, we could have used the same accuracy object here. So, validation and test set trackers don’t necessarily need to be kept separate, although it’s nice for organization purposes and logging later. However, validation and training set trackers need to be kept separate because we use both simultaneously during training. This means that we compute the validation accuracy between epochs while tracking the accuracy across epochs.

    Please answer all questions to proceed.
    Watch Video 1

    Unit 5.3

    Videos
    Follow along in a Lightning Studio

    DL Fundamentals 5: PyTorch Lightning

    Sebastian
    Launch Studio →
    Questions or Feedback?

    Join the Discussion