Hacker News Reader

Comment by mikeash | Hacker News Reader

I'm disappointed that you don't score positive infinity for getting an answer correct with 100% certainty. Pass the entire course if you find one question you're really sure of. It would stop the teachers from putting in softball questions.

View on HN

Probability doesn't work like that. Likelihood has an upper bound at 1, and log likelihood has an upper bound at 0. Likelihood just represents the probability you assigned to the actual outcome. You can't have a probability higher than 1.

You can't have a probability lower than zero, but you're still able to obtain negative infinity points. Why is there an upper bound but not a lower bound?

Because it takes a lot of skill to be "always wrong" (you would have to know the right answer in order to answer incorrectly). No skill is represented by getting 50% right/wrong, and indeed it scores you 0 points.

I don't think that answers mikeash's question. He is not talking about always being wrong. He is pointing out that the incentives are not symmetric. If you say that you are 100% certain, and you are wrong, then you fail the class. If the incentives were symmetric, if you said you were 100% certain and you were correct, then you would automatically pass the class.

A potential answer is: this asymmetry is similar to incentives outside the classroom. Claiming 100% certainty and being wrong can be disastrous to your reputation, and likely much more negative than the positive benefits of claiming 100% certainty and being correct.

Getting 50% right/wrong while expressing 100% confidence in your answers scores you negative infinity points. Indeed, getting 99.9% right/wrong while expressing 100% confidence scores you negative infinity points. But getting 100% right with 100% confidence only scores you a finite positive number of points.

That's just the way log probabilities work. Log(0) is negative infinity, but log(1) is just 0.

You can have infinitely high scores if you use log odds though: http://lesswrong.com/lw/mp/0_and_1_are_not_probabilities/

But that can only happen if you get every question 100% right. Whereas getting a single question 100% wrong is easy to do.

Why is log probability the correct way to evaluate this?

It's convenient, because multiplying probabilities together quickly gets very very small. Typically smaller than can be represented in floating point even.

Whereas logarithms make multiplication just addition, and they get small very very slowly.

Personally I prefer geometric mean. Which is equivalent to the average log likelihood. It has an upper bound at 1 and lower bound at 0, and represents the "average" probability you assigned the correct answer.

I think I read a blog post about that somewehre.

mikeash was not advocating for a probability higher than 1. Rather, he was advocating for an infinite reward for indicating a probability of 1. We could reframe his suggestion to be, why are the incentives not symmetric? (I think there are good reasons for this, but it's a valid question.)

I was just saying that's the way the math works out. It's not an arbitrary decision, it just happens that the natural way to score probabilities allows negative infinity from a single question.

You could make an arbitrary scoring metric that does whatever you want. But it wouldn't be principled, or have nice mathematical properties like this.

The math was chosen arbitrarily so every outcome is also arbitrary.

That would make the class too easy to game. Just answer one question you feel most confident about, with 100% certainty, and if you happen to be right, you can blow off the entire rest of the course and get an instant A+.

Reply on news.ycombinator.com