10.1 Trustworthy and Reliable Machine Learning

References

• Trustworthy ML Initiative
• Creating Confidence Intervals for Machine Learning Classifiers
• Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

Code

• Part 2, Implementing Confidence Intervals for PyTorch Models: 10.1-ci.ipynb

What we covered in this video lecture

In this lecture we discuss the fundamental principles for creating trustworthy and reliable machine learning systems. Firstly, there’s the importance of privacy in machine learning applications: consent must be obtained before data collection and sharing models trained on sensitive data should be avoided. Then, there’s the necessity for fairness and ethics in designing machine learning systems, to prevent potential harm or injustices. Also, it’s worth highlighting the significance of transparency and interpretability: details like dataset limitations and known bugs should be documented and shared.

Quantifying uncertainty in machine learning predictions is one of the many aspects behind designing trustworthy systems. There are two main methods to quantify this uncertainty: confidence intervals and conformal predictions. Confidence intervals provide a range of values which contains the true value with a certain level of confidence, and are often used to estimate model accuracy. Conformal predictions, on the other hand, are concerned with the uncertainty of individual predictions themselves, and provide a set of labels that contain the correct label with a given confidence level.

Finally, we look at a code example implementing confidence intervals for a finetuned LLM in PyTorch.

Additional resources if you want to learn more

If you are interested in learning more about trustworthy ML, I highly recommend checking out the curated resources via the Trustworthy ML Initiative. Also, to learn more about machine learning model evaluation in general, I recommend skimming through my Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning article.

To learn more about conformal predictions, I recommend checking out this awesome list of resources here.

Please don’t hesitate to ask questions on the Discussion forum if you have any!

Watch Video 1 Mark complete and go to Unit 10.2 →