Deep Learning Fundamentals

Pages

Deep Learning Fundamentals

Deep Learning Fundamentals > Unit 4 > Unit 4.1

Course Progress:

Logistic Regression for Multiple Classes (Part 1-5)

Slides

Part 1: The Softmax Regression Model
Part 2: The Softmax Activation Function
Part 3: From Softmax Scores to Class Labels
Part 4: The Cross Entropy Loss Function for Multiple Classes
Part 5: A Cross Entropy Walkthrough With Code

What we covered in this video lecture

In this video, we extended the binary logistic regression model to a multinomial logistic regression model that works with an arbitrary number of classes. In machine learning and deep learning contexts, this multinomial logistic regression model is commonly called softmax regression.

We saw that only minimal code changes required when we turn a logistic regression model into a softmax regression model. We replaced the logistic sigmoid function with a softmax activation function, and we replaced the binary cross-entropy loss by the categorical cross-entropy loss.

Additional resources if you want to learn more

It can sometimes be tricky to remember the correct inputs for the different cross-entropy loss functions in PyTorch, so I created a small cheat sheet here.

Log in or create a free Lightning.ai account to access:

Quizzes
Completion badges
Progress tracking
Additional downloadable content
Additional AI education resources
Notifications when new units are released
Free cloud computing credits

Quiz: 4.1 Dealing with More than Two Classes: Softmax Regression (PART 1)

Which of the following are changes we make to the Logistic Regression model to convert it into a Softmax Regression Classifier? We…

change the linear layer to a hidden layer.

Incorrect. Softmax Regression does not have hidden layers.

change the sigmoid activation to a softmax function.

Correct. We use the softmax function to compute the class-membership probabilities.

increase the number if input feature as a multiplicative factor of the number of classes.

Incorrect. Softmax regression and logistic regression operate on the same features.

add an additional hidden layer.

Incorrect. Softmax Regression does not have hidden layers.

Please answer all questions to proceed.

Quiz: 4.1 Dealing with More than Two Classes: Softmax Regression (PART 2)

For each training example, the softmax function returns one probability membership score for each class. Which of the following statements about the sum of these scores (for one training example) is correct?

The sum of the scores depends on the training example — it’s differs based on the input.

Incorrect. The sum of these scores is always the same, independent of the training example.

The scores always sum up to 1.

Correct. Due to the normalization term (denominator), the sum is always 1.

The scores always sum up to 1 times the number of classes.

Incorrect. Due to the normalization term, the sum cannot exceed 1.

Please answer all questions to proceed.

Quiz: 4.1 Dealing with More than Two Classes: Softmax Regression (PART 3)

To obtain the class label from the softmax probabilities, we…

we threshold the scores at 0.5.

Incorrect. We would threshold at 0.5 if we use a logistic sigmoid activation function.

apply the argmax function to select the index of the highest score.

Correct. We would threshold at 0.5 if we use a logistic sigmoid activation function.

Please answer all questions to proceed.

Quiz: 4.1 Dealing with More than Two Classes: Softmax Regression (PART 4)

>Suppose we have a dataset with 5 class labels with labels 0 to 4. What is the correct one-hot representation for a training example with label 3?

1, 0, 0, 0, 0

Incorrect. This would correspond to label 0.

0, 1, 0, 0, 0

Incorrect. This would correspond to label 1.

0, 0, 1, 0, 0

Incorrect. Remember that the class labels start at index 0.

0, 0, 0, 1, 0

Correct. Since we are starting at index position 0, the 4th entry corresponds to label 3.

0, 0, 0, 0, 1

Incorrect. This would correspond to label 4.

Please answer all questions to proceed.

Quiz: 4.1 Dealing with More than Two Classes: Softmax Regression (PART 5)

Suppose you have 2 training examples in a 3-class classification setting. What is the cross-entropy loss for a perfectly random prediction?

0.33

Incorrect. A perfectly random prediction is 0.33 for a 3-class case. But the output of the loss is not 0.33.

1.1

Correct. A perfectly random prediction yields a probability score of 1/3 in a 3-class setting. Then we have -log(1/3) = 1.10. Note that this is independent of the number of training examples, since we average: -(log(1/3) + log(1/3)) / 2 = 1.10

0.5

Incorrect. The class-membership probabilities are 0.5 for a perfectly random prediction in the binary case. But we have a 3-class case here, and it’s also not the output of the loss.

0.0

Incorrect. A loss of 0 would correspond to the best possible (and not random) scenario.

infinity

Incorrect. Infinity would indicate a very wrong, not random prediction.

negative infinity

Incorrect. The cross-entropy loss can’t be negative.

Please answer all questions to proceed.