Hacker News Reader

Except this is not what a p-value is supposed to tell you in the first place. It's supposed to tell you how compatible your observations are with the ASSUMED theory. The minute you use it as an interpretation of a posterior probability (let alone a malformed one), you have violated the assumption on which you rely to interpret it.

To use your own example. If you picked a coin from said bag which is mostly full of otherwise unbiased coins, and your hypothesis is "this coin is not biased", a sequence of ten heads would be largely "incompatible" with your hypothesis (as if the hypothesis of unbiasedness was true, you should only expect this result 1/1024 times).

It is up to you to then place this result in the context of other prior assumptions etc, and say "yes it's incompatible but still very likely". The p-value does not preclude you from performing a post-hoc analysis of the base-rate if you so wish. It just so happens that the 'assume a hypothesis' approach is a useful one in the absence of all other information. Which is not so different from a Bayesian saying 'I'll use an uninformative prior and let the data speak for itself'.

But using it as a test of posterior probability, especially when prior information exists, and saying "ha see it sucks" is not a fair criticism for it (whereas criticising its use as a test of posterior probability is).

Yes, that's exactly my point. Saying that it really supposed to be the probability of the data assuming the null hypothesis is true is pretty confusing to most people and often times the gut feeling people keep us that it still somehow tells you how likely it is that the null hypothesis is true. I just like this example because it highlights that there is a distinction between these two probabilities.

In fact, if you look at the document I link to above, you'll see the correct Bayesian calculation where we actually compute p(Biased | 10H). In the first case it's pretty high but in the second case it's very small.

I 100% agree that p-values are hard to get right (not that Bayesian analysis is necessarily more straightforward, mind you). What I disagree with is the assertion that a) the two scenarios listed above are equivalent, and b) that you cannot incorporate a prior analysis for use with p-values (which is best thought simply as a measures of compatibility of the data with respect to a hypothesis). E.g. in the biased vs unbiased example, if your null hypothesis is "The coin is biased", then it is simply the wrong hypothesis for the problem (or, more specifically, the wrong null/alternative pair, since these two only make sense as a pair; a null by itself is not sufficient to define the assumptions for the problem). If your alternative hypothesis is "ok, it's one of the 10^1000-1 unbiased ones", then your null should have necessarily been "it's the biased coin with 1/10^1000 chance of being chosen", which effectively forces you to apply your prior probabilities on the hypotheses themselves. Once you do that, you will easily see that the compatibility of the data with the alternative hypothesis is much higher, exactly as bayesian analysis would predict. In fact, mathematically it does not make sense for a Bayesian result to contradict a frequentist one, when looking at the same outcomes; if this happens one should suspect that the two problems were formulated using different assumptions, and are thus not equivalent, or that one is looking at different outcomes.

I also disagree with the 'mindblowing' example in your blogpost linked above (though I did enjoy reading it, thank you for linking!). Your example only shows the 'superiority' of bayesian reasoning because you (presumably unintentionally) misrepresented the alternative hypothesis by the wrong 'tail' (to keep things simple - one could go further say that there are more effective alternative hypothesis formulations than a 'tailed' one here, since we are not testing simply against the alternative of 'higher bias' but against a known specific bias of 90%). One's choice of alternative hypothesis is part of one's assumptions reflecting prior knowledge. The analysis is misleadingly poor for the frequentist analysis, in that it is equivalent to the bayesian analogue of saying "Assume we have a coin that gives heads 90% of the time and I toss it 10 times; let's represent this with a prior probability Binom(10, 0.1)". What? Of course you can expect such a poor prior to give wrong results with high certainty! It makes no sense to assume that if your null is not true, then you should look for prior knowledge in the lower values. Furthermore, the exact definition of a p value allows great flexibility in the definition of the 'or worse' criterion. You are free to choose a criterion that takes into account prior information (and that forms part of your assumptions). In the 'mindblowing' example, you see that as soon as you choose the ranking measure to be P/(P+Q), and use it to calculate p-values by summing over the probabilities of the 'or worse' cases with regard to that ranking, you get very clear patterns that agree with your bayesian analysis (i.e. that 0 HEADS are more compatible with the unbiased coin hypothesis, than 1 HEADS, and similarly that the reverse applies for the biased hypothesis).

I think this example misleads one's intuition for the following reason: in the proposed scenario, you'd only see the coin come up heads 10 times in a row about 1 in 1024 times you ran the experiment. While your conclusion would likely be incorrect, you almost never run into that scenario.

For example, if you conducted a study every week for 20 years, you'd both be extremely prolific and expect to have drawn about one wrong conclusion.

The example is a case of an absurd premise (i.e., a fair coin comes up heads 10 times in a row) leading to an absurd conclusion (that the coin is biased). Of course, this is exactly the guarantee the hypothesis test provides: under robust assumptions, you'll draw the wrong conclusion only rarely.

A 1 in 1024 chance is not an absurd premise! Something that has a 1 in 1024 chance of ocurring to a person happens to 7 million people. If you get that evidence, you need to be capable of interpreting it correctly.

The OP's example would also worked well enough with even just 5 heads; you'd already pass p < 0.05, despite the actual probability of having picked the biased coin still being minuscule.

You're actually being misled. Formally, you have an observed event, 10H and so you want to know the probability that your coin was fair, conditioned on having seen 10 heads. You have fair coins (F) and biased coins (B)

You have P(H) == 1/1024, P(H|B) == 1/1. And,

Then the probability that you picked a biased coin (P(B))? 1/10^1000. The probability you picked a fair coin (P(F))? (1 - 1/10^1000)

All in all, P(B|H)? Well Bayes rule tells us P(A|B) = P(B|A) * P(A) / P(B). Or in this case, P(B|H) = 1 * 1024 / 10^1000, or approximately 1 / 10 ^ 997.

> For example, if you conducted a study every week for 20 years, you'd both be extremely prolific and expect to have drawn about one wrong conclusion.

This is a prior! In the kind of experimentation I do, I run literally tens of thousands of experiments a day.

I've been playing a board game daily for the last two weeks. As part of this game, a player has to draw one out of six unique cards. In the last five games, I've repeatedly drawn the same card. This is a real example.

1 / (1/6)^5 ≃ 7776

You can't rely on odds after an event has happened to determine probably. I could shuffle a deck containing 1000 unique cards and then look at their order. It's astronomically low that this order occurs but it did happen.

When discussing conditional probability, this is absolutely what you can do.

Let's use your example. Conditional probability is P(A|B), the probability of event A, given that event B was observed. What's the probability that I am a magician, given that I shuffled the deck and when you saw it it was still in new deck order?

Now certainly, there is an astronomically small chance that this was observed due to random chance. And yet if you observed this, I'd expect that you would, with relatively high confidence, believe that I stacked the deck.

I'm not doing anything different than what we are doing when discussing the improbability of a hypothetical 10-heads-in-a-row event.

This scenario is several steps too hypothetical for me.

If you have 10^1000+1 coins, how did you accurately ascertain that exactly 1 coin was biased and the others were not?

And if you have been able to do that, well enough to state it as fact, why do you now need to do this statistical test?

It's just a thought experiment meant to highlight a common fallacy in how people think about p-values.

I find it fallacious to see this as a criticism of p-values.

If I wanted to identify whether a particular coin was the [most] biased out of 10^1000, flipping one coin 10 times isn't a reasonable setup for the experiment.

And, my conclusion to your hypothetical experiment would be: the person who told me that there was a 99.9...% chance this coin was fair was likely incorrect about that.