Making a neural network say “I Don’t Know”: Bayesian NNs using Pyro and PyTorch | Hacker News Reader

Making a neural network say “I Don’t Know”: Bayesian NNs using Pyro and PyTorch

towardsdatascience.com

126 pointsparaschopra46 comments

Uh, the lead of the article claims it achieves "97% accuracy on MNIST". If you are removing 12% of the MNIST images at your neural network's discretion, you can't claim the 96% accuracy on the remaining set as your overall accuracy! You're essentially self-selecting for test data the classifier is confident in, so of course that accuracy will be high.

If this was allowed then I'd make an ImageNet classifier which just discards all images except for one I'm really good at classifying, then claim I have 100% accuracy on ImageNet.

My main point is not about aggregate accuracy, it's about comparing confidence across classes of datasets.

Edit: the entire point of bayesian approach is that you can make decisions on a loss function where you can make a tradeoff on the (business) cost of making a wrong decision and (business) regret of not making a decision. For real world applications, accuracy is a useless metric. What matters is the net (business) benefit or loss.

Your article bolds the following in the introductory paragraph:

> My final classifier had accuracy of ~97% on MNIST and it refused to classify white noise and the majority of unrelated (non-MNIST) images.

I'm saying that you are making a misleading claim when you say this. It is categorically untrue that you achieve 97% on MNIST with this approach. Accuracy on a standardized test set makes no sense when you select a subset from the test set for maximal accuracy. I'm not saying that your approach doesn't have value or make sense in some business cases, because I think the overall idea is interesting but these statements are not correct.

Correct. That's why people use https://en.wikipedia.org/wiki/F1_score instead of just accuracy.

Yes, this is buried deeply in the article, but the classifier rejected 12% of the handwritten digits - and did not even have good accuracy when forced to guess on those 12%.

I have limited knowledge of machine learning so forgive me if I am wrong. I recall that some classification techniques provide you a confidence score of the classification... so can't you just treat a classification score lower than a certain threshold as "I don't know"?

This question was answered in the comments section of the article.

Basically Softmax will only give you values close to 1 or close to 0, so there's no room for your own interpretation or threshold.

https://medium.com/@paraschopra/softmax-amplifies-small-diff...

Do you need to do softmax? Why not just look at the underlying amplitudes of the output before applying softmax? Or is it that they won't be normalized?

Interesting suggestion, honestly I'm not sure. Although this should be easy to test empirically. I feel if this were to work, it may depend on architecture of the network and what intermediate functions are used (ReLu, sigmoid and tanh). But your suggestion is easy to test empirically and I'll try it.

There was another method proposed recently for approximating uncertainties: using dropout during evaluation (which effectively makes different runs on the same image as samples). But I came across a discussion on the efficacy of the technique and I wasn't sure. https://www.reddit.com/r/MachineLearning/comments/7bm4b2/d_w...

My next project is to implement dropout and see how it compares with the current approach. I'll also try your suggestion.

Yes, they wouldn't be normalized so you can't interpret it as a probability. Softmax is the function that normalizes it. However, even that you can't interpret as a probability of some outcome, because you do not take into account the prior.

For that, you need actual Bayesian techniques.

Can you elaborate on what you mean by not taking into account the prior? What is the output of aoftmax if not a probability?

Well, it's a probability of the output given the fixed, fitted model parameters. This is the likelihood. This is not the probability of getting that output, since we don't know what the parameters should really be. See the inverse probability fallacy [0]. This confusion is literally the main reason behind the replication crisis of scientific research.

In ML, you typically have a model f(w;u), and you are trying to fit the parameters w to some data, given some hyperparameters u that represent some kind of prior information. In frequentist techniques, these could be regularization parameters.

In frequentist techniques, the hyperparameters the fitted parameters are assumed fixed, and not random. In contrast, Bayesian methods assume it is random as well. This extra uncertainty is neglected by the predictions of frequentist techniques.

Nonetheless, the uncertainty is real.

[0] https://en.wikipedia.org/wiki/Confusion_of_the_inverse

Thanks!

I think that post is quite misleading. 90 and 100 are extreme values representing extreme confidence. Your model could just as easily learn [0.09, 0.10, 0.05] on the final layer, and the probabilities would then be [0.337, 0.340, 0.323]. And look, now softmax has the opposite problem! The final activation layers differed by a factor of 2, but softmax output almost identical probabilities!

Except this should not be surprising, because softmax is not scale invariant. Values on the unit scale is what you would normally expect, so if you feed it values 100x larger, it's going to be way more confident. If you feed it values 100x smaller, it's going to be way less confident.

This is interesting. In theory, what you are suggesting can be true but in practice you do think the last layer can output 0.1, 0.09, etc especially when last layer is typically not pushed through sigmoid.

Isn't that mostly true with highly curated datasets with clearly defined categories?

No. Probabilities can be random variables themselves.

Say p is the probability an image is of a cat.

p could be low, e.g. 0.05.

However, perhaps p has the 0.9 credible interval (0.01, 0.1). The width of that interval expresses how confident you are of the point estimate.

The correct translation of "I don't know" is a wide credible interval.

I disagree with this characterization, especially in a Bayesian context (which credible interval comes from).

Parameters into a model can be modeled as random variables. Probabilities themselves cannot be, unless they are themselves parameters (or happen to coincide with a parameter) into another part of a statistical model. A statistical estimation of an event (in the sense of probability theory) must bottom out at a single number.

Let's take a coin toss as an example. It does not make sense to say I think that the coin has a 0.05 chance of flipping heads with a 0.9 credible interval of (0.01, 0.1). It does not make sense even to have a probability distribution on coming up heads (strictly speaking).

Coming up heads is an event and events do not distributions associated with them, but rather single probabilities.

On the other hand, we can model the outcomes of a coin with a Bernoulli distribution with parameter p where the probability of heads is p and that of tails is 1 - p. Notice here that p is a parameter that just happens to coincide with our probability. It then makes sense to talk of p as a random variable and having a distribution, credible interval, etc.

Note, however, that our estimation of the event of the coin coming up heads is still a single number at the end of the day. Our intermediate calculations might just involve integrating over the space of intermediate values of p, which means that p no longer coincides with the probability of heads (indeed it doesn't really make sense because p is no longer a single number).

But how do we express the intuition of "confidence in my probability estimate?" After all it seems like there's a difference between me saying this coin has a 50% chance of heads when I know it's fair and me saying a 50% chance of heads when you have me an unfamiliar coin I've never inspected (even though reasonable statistical models for either case might both say it's 50%).

In Bayesian statistics this is encoded entirely by the function that updates your posterior probability. If you are very "confident" your posterior probability will remain similar to your prior probability no matter what evidence you see (I know for sure this coin is fair so even if it comes up tails 1000 times I still think the next time will be 50% chance of heads). If you are not very confident your posterior probability might change more dramatically (hmm... I saw a string of tails, I'm pretty sure the next coin flip probably won't be heads again and I'll modify my probability accordingly).

Unfortunately you're confused about Bayesian statistics. A classic Bayesian analysis of coin flips would look something like:

x_i ~ Bernoulli(p) p ~ Beta(1,1)

For which the posterior IIRC:

p | x ~ Beta(n_tails+1, n_heads + 1)

Which, if you look up the variance of the Beta distribution, you will see a credible interval p shrink around the truth of p=0.5 if the coin is indeed fair.

This encapsulates the fact that if we have 5 heads and 5 tails we think the coin is fair. But if we have 500 heads and 500 tails we still think the coin is fair, but we're much more confident that it is so.

I think we're talking past each other. I'm saying this:

A probability can only be assigned to an event. A distribution can only be assigned to the collection of all events. A probability cannot be an event except as a parameter into another part of a model.

Hence statements of the form "This is a picture of a cat with probability X with this credible interval" are category errors.

Notice how what you're talking about in this example is the parameter p not the probability that the coin turns up heads. Indeed to find the latter you must explicitly integrate over your p.

We agree on the analysis of confidence (although I'm not sure I'd say "I think the coin is fair with low confidence" after e.g. a single heads and a single tails; I'd probably say "I don't know, but that's the mode of my distribution").

My hand-wavy statement about updates alludes to the general case, where you can't necessarily describe your model easily with parameters, but instead have an arbitrary joint distribution. With parameters to your model you can talk about distributions and summary statistics about your parameters (e.g. p for Bernoulli), which is a special case of a general update function, but even if you don't have parameters for which to talk about, your update function still has all the information about "confidence" you need.

- We flip some coins X_i~Bernoulli(p)

- We can say that p = Probability(X_i = heads)

- We have a posterior distribution for p (e.g. under Beta(1,1) prior).

- We can define a=icdf_p(0.05), b=icdf_p(0.95) under the posterior.

- (a,b) is an event in the sigma algebra (of the posterior probability space) for p whose mass is 0.9.

- (a,b) is also a central credible interval.

I don't think you'll be able to point to one of those statements above and show that it is wrong.

I think the stumbling point here is the notion of random measures. Random measures have sample spaces, sigma algebras of events, and their own distributions. Events have probabilities, as you say. This is the sense in which we can speak of probabilities (i.e. probability distributions) having probabilities (i.e. probability distributions). Because Binomials are one dimensional it's particularly easy to think about intervals.

The second step p = Probability(X_i = heads) is technically wrong (although a common abuse of notation). It's strictly speaking p = Probability(X_i = heads | p).

So while it is meaningful to say:

"p has this probability distribution."

It is not meaningful to say:

"The probability the coin comes up heads has this probability distribution."

This is because p and the probability the coin comes up heads are not the same thing (this is even more apparent if you have a non-Bernoulli scenario) when p is no longer fixed.

It's not really just a case of marginal vs conditional probabilities either. Even in the case of a conditional probability, it's only defined for a fixed condition.

Even in the case of random measures, while you can reasonably talk about the probability of a probability distribution, it still doesn't really make sense to talk about the probability distribution of a probability.

> Even in the case of random measures, while you can reasonably talk about the probability of a probability distribution, it still doesn't really make sense to talk about the probability distribution of a probability.

But that's exactly what it does mean. For the Dirichlet Process, the probability of hitting a set in the partition of a sample space is random. A DP is a couple of generalisations up from the beta distribution on p under discussion [1]. I.e. this specializes to the current situation as:

* The sample space theta is the set {H, T}.

* The only interesting partition to talk about is {A_1, A_2}={{H}, {T}}.

* The Dirichlet distribution on (G(A_1), G(A_2)) is therefore 2 dimensional, which is a Beta.

* G(A_1) is the probability measure of hitting heads.

No specific parametric model involving univariate p needs to be involved.

[1] c.f. eqn 2 of http://www.gatsby.ucl.ac.uk/~ywteh/research/npbayes/dp.pdf

I have to disagree with the top reply. You definitely can interpret low probability as I don't know. Check out this blog post-

http://henripal.github.io/blog/langevin

And these papers-

1. https://arxiv.org/abs/1806.02855 2. http://arxiv.org/abs/1811.11145

In Christopher Bishop’s PRML book, he shows a procedure for getting the full posterior of a neural net using estimates of the Hessian evaluated at the point of inference.

Any idea how sampling compares in performance and computational efficiency?

SVI still uses sampling under the hood. Any probabalistic programming framework (such as pyro) uses random sampling, there's no other way to estimate the curves of a complex set of interacting probability distributions. Obviously there are some optimizations for simple distributions that are known (e.g. we know what the expected values of simple gaussians will be, and the PDF or CDF of a given sample).

For people who are interested, there is an upcoming NIPS paper that might be relevant for this problem:

Paper: http://arxiv.org/abs/1806.01768

Demonstration: https://muratsensoy.github.io/uncertainty.html

I thought you were going adding a class called I don't know.

I tried doing that, but I didn't find any pydontknow module on pip.

Your comment reminded me of this xkcd comic https://xkcd.com/353/

The nice thing about using likelihood as the training criterion, you don't even need an extra category, you can just combine the outputs with basic decision theory, like so:

  thres = wrong_cost / (wrong_cost+right_gain)
  response = (max(out_probs) > thres) ? argmax(out_probs) : "I don't know"

While the bayesian methods can theoretically perform better for out-of-set inputs, it feels like an oversight to not include this simple method as a baseline. My gut feeling is that this would perform equally well, if not better.

This works if you get nicely balanced probabilities out of neural network.

Softmax which is popular way to transform network outputs to probabilities pushes probabilities either to zero or one.

There may be other ways to transform outputs to probabilities where your approach could work but I'm not aware of them.

Also to reply to OP's comments, to have a class like 'I don't know' you need to have a dataset of things that don't look like your training dataset. And you'll always miss examples of things that don't belong to your input class. Bayesian solves it in a nice way by not requiring this infinite not-input training data.

the author of this article would do well to admit that he "doesn't know" bayesian inference very well. unfortunately, the usage of pyro is incorrect here. in particular data subsampling is handled incorrectly (there may be more issues). the weights of the network represent a global random variable. consequently, when doing data-subsampling care must be taken that the ELBO that is constructed is scaled correctly. that is not done here. probabilistic programming languages like pyro are a great tool for making bayesian inference easier and more accessible, but they can be produce arbitrarily bad results if the user uses them incorrectly.

Could you elaborate on how to do this correctly? This is one of the only examples of using pyro to build a Bayesian NN and it would be really valuable have a correct implementation.

How does this compare to Dropout based methods? http://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801aa...

My guess is you use dropout during training.

found this online(might be useful):

Bayesian Approach: Pros and Cons

 	Pros

Can calculate explicit probabilities for hypotheses.

Can be used for statistically-based learning.

Can in some cases outperform other learning methods.

Prior knowledge can be (incrementally) combined with newer knowledge to better approximate perfect knowledge.

Can make probabilistic predictions.

 	Cons

Require initial knowledge of many probabilities.

Can become computationally intractable.

http://www.ru.is/faculty/thorisson/courses/v2008/gervigreind...

> Require initial knowledge of many probabilities.

This is not a really big issue. You can always use uninformative priors, which is simply the Bayesian way of saying "I don't know" what the probability distribution looks like upfront. Then the data can help you infer the right shape of probabilities.

> Can become computationally intractable.

This is a bummer, but modern methods like variational inference and better sampling techniques are making Bayesian analysis on large-scale data and complex models tractable.

I linked a video the article that I highly recommend watching. Variational Bayes and Beyond: Bayesian Inference for Big Data https://www.youtube.com/watch?v=DYRK0-_K2UU

> Require initial knowledge of many probabilities.

Can you make a two-stage design where the initial probabilities are generated by another machine learning model?

sorry for posting twice in this thread but i figure it's a good place to ask: what other methods are there for getting a nn to report "i don't know"

My Master thesis supervisor had a fun trick for this, that can be used not only on on NNs, but also on other classifiers: https://arxiv.org/pdf/1011.3177.pdf

The gist is, you train a single classifier that will work as two classifiers (with certain restrictions): one that says "false/not-false" and another that says "true/not-true".

If the output of the classifier is "not-false" and "not-true", then you consider that as a "I don't know".

Is this what you mean: For example, in MNIST, your model would output 20 probabilities: (1-yes: 0.8, 1-no: 0.2, 2-yes: 0.1, 2-no: 0.9, etc) where each pair sums to 1?

Or train the network with unknown or not-a-number data or pseudo data. https://patents.google.com/patent/US5359699?oq=5%2C359%2C699

how effective would this be in deep nets with small data sets?

edit: does anyone know of any papers that talk about this technique formally?

interesting to see an answer too...usually bays is better with small data sets right?

Bayesian techniques are effective with small data sets if you build your model carefully. For example, if you are able to encode the generating process of your dataset, even a _single_ example could be used for effective training. For example, see this http://web.mit.edu/jgross/Public/lake_etal_cogsci2011.pdf

By providing a good generative model, you are shifting the effort from collecting more data to carefully coding your expert knowledge. In my view, this is an exciting paradigm as it allows combination of human knowledge (as generating process and prjors) that can be overcome (in posterior distributions) if enough data is accumulated against it.

Socrates was on to something.

Reply on news.ycombinator.com