Lightning AI Studios: Never set up a local environment again →

Log in or create a free Lightning.ai account to track your progress and access additional course materials  

Deep Learning Fundamentals

  • Pages

    • Final certification exam

    Deep Learning Fundamentals > Unit 1 > Unit 1.7
    Course Progress:

    1.7 Evaluating Machine Learning Models. Parts 1 and 2

    Slides

    References

    What we covered in this video lecture

    In our previous perceptron coding lecture, we only worked with a training set to keep things simple. In this lecture, we learned that creating a test set for model evaluation is a good idea. Moving forward, we will always be using a test set in future units.

    Furthermore, we learned about the importance of data shuffling and saw that it’s a good idea to consider performance baselines before we use a machine learning model. This is because a high classification accuracy does not always imply that the machine learning model makes accurate predictions — in cases of high class imbalance, it is trivial to achieve high classification accuracy by always predicting the most frequent class label.

    Additional resources if you want to learn more

    There are many additional performance metrics that we have not covered in this lecture. Commonly used metrics for binary classification include

    If you are interested in a comprehensive article with more advanced model evaluation techniques, check out the following tutorial by Sebastian Raschka: Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning.

    Log in or create a free Lightning.ai account to access:

    • Quizzes
    • Completion badges
    • Progress tracking
    • Additional downloadable content
    • Additional AI education resources
    • Notifications when new units are released
    • Free cloud computing credits

    Quiz: 1.7 Evaluating Machine Learning Models Part 1

    Typically, we shuffle the dataset before we divide it into training and test set to make sure that the

    Correct. By shuffling we can avoid that some labels are exclusively and don’t show up in the test set if the dataset is sorted.

    Incorrect. Although it might be possible that the dataset was sorted by “difficulty,” this is not the typical problem we want to avoid by shuffling.

    Incorrect. The features in the training and test set should always be the same.

    We typically use the training dataset only once before we tune the model on the validation set.

    Incorrect. We can use the training set as many times as we like.

    Correct. There is no limit in terms of how often we can use the training set

    After we trained the model on the training set, we tune the model on the test set.

    Incorrect. We should not tune the model on the test data. The test set should only be used for the final evaluation. The validation data is for tuning.

    Correct. We should not tune the model on the test data. The test set should only be used for the final evaluation. The validation data is for tuning.

    Please answer all questions to proceed.

    Quiz: 1.7 Evaluating Machine Learning Models Part 2

    Given the true label array [0, 1, 0, 0] and the predicted labels [0, 1, 1, 0], what is the prediction accuracy of the model?

    Correct. There is only one mistake where the model predicted label 1 instead of 0.

    Incorrect. Hint: if you put both arrays on top of each other, you can count the number of times the prediction is correct more easily.

    Incorrect. It seems like you computed the error, not the accuracy

    If we work with a dataset that has more than two unique class labels, we have to use the balanced accuracy instead of the regular accuracy.

    Incorrect. Balanced accuracy addresses the issue of imbalanced class labels (one class has more labels than the other). It does not depend on the number of unique class labels though.

    Correct. Balanced accuracy addresses the issue of imbalanced class labels (one class has more labels than the other). It does not depend on the number of unique class labels though.

    Please answer all questions to proceed.
    Watch Video 1

    Unit 1.7

    Videos
    Follow along in a Lightning Studio

    DL Fundamentals 1: Intro to ML & DL

    Sebastian
    Launch Studio →
    Questions or Feedback?

    Join the Discussion