Creating Confidence Intervals for Machine Learning Classifiers

Clear and concise write-up, thanks for posting. I especially appreciate the comparative graph at the end.

In practice, I tend to use "method 3" most of the time as I feel it has the best balance of complexity and practicality while still being able to deal with label noise.