Comment by btilly | Hacker News Reader

Comment by btilly | Hacker News Reader

Parent Full thread

For giggles and grins, my aunt and uncle are an actual example of one of the classic frequentist vs bayesian examples where frequentist statistics says that something utterly irrelevant should matter.

Scenario 1 (true). Bill and Lorena had 7 children. 6 were boys, 1 was a girl. Are they biased towards having one gender? A 2-sided p-value says that there are 16 possibilities of this strength or more, each of which has probability 1/2^7, for a p-value of 2^4/2^7 = 1/2^3 = 0.125. We therefore fail to reject the null hypothesis at a p-value of 0.05.

Scenario 2 (also true): Bill and Lorena decided to have children until they had a boy and a girl. They had 6 boys then a girl. Are they biased towards having one gender? An event this unlikely could only happen if they had 6 of the same gender in a row, which is 2 possibilities of probability 1/2^6, for a probability of 2/2^5 = 1/2^5 = 0.03125. We therefore reject the null hypothesis at a p-value of 0.05.

Now the Bayesian gotcha. According to Bayes' theorem, the intent of Bill and Lorena can have absolutely NO impact on ANY calculation of posterior probabilities from prior expectations. There is no logical way in which this fact should matter at all. And yet it did!

Is this actually true for the Bayesian model? Throw in a parameter for whether or not Bill and Lorena were trying to have children until they had a boy and a girl. Now your answer will depend quite heavily on your prior!

This is entirely true.

The posterior odds are computed ENTIRELY from the odds of the observed events under the prior beliefs. There is NO WAY in which might-have-beens and didn't-happens enter in. Therefore the posterior probabilities cannot depend on the knowledge of what they would have done if something different had happened.

Of course frequentist statistics are heavily affected by what would have happened if something different had happened.

No, even as a Bayesian I could use either of two models to understand my data:

In world one, I use bayesian update on the model P(# boys | Bias) = Mutltinomial(n = 7, p = Bias)

In world two, I use bayesian update on the model P(# children | Bias) = Geometric(p = Bias)

Might-have-beens and didn't happens do play in, in my choice of model. I should choose the one that I believe, and if I'm not certain, I should use an even more complicated model that incorporates my beliefs about what models might be appropriate.

Not if you follow Bayes' theorem. If you start with a prior distribution of beliefs about the likelihood of various ratios of boy vs girl births, the posterior distribution only depends on the observed outcomes. And the posterior distribution is exactly given by Bayes' theorem.

One possible source of confusion for you is that Bayesian ideas have been a source of inspiration for a lot of ad hoc techniques (eg naive Bayes) which do NOT follow Bayesian rules of inference. The reason is that exact inference in Bayes nets is NP-hard. So you're used to hearing "Bayesian" applied to things that have nothing to do with Bayes' formula.

Our goal is to calculate P(pb > 0.5 | "Six girls and one boy"), where pb is the probability of having a boy. (Ignoring that we have already assumed p is fixed), by applying Bayes' Theorem, we have:

P(pb > 0.5 | "Six girls and one boy") = (P("Six girls and one boy" | pb > 0.5) * P(pb > 0.5)) / P("Six girls and one boy")

Applying Bayes theorem thus requires us to calculate P("Six girls and one boy" | pb > 0.5). How do you suggest that we do this? Why is your answer the unique correct solution?

Well first you have to start with a prior distribution of beliefs, which that is not. And to reduce confusion I'll switch back to the actual genders (6 boys then a girl).

Suppose our prior distribution of beliefs is 0.5 that the probability is exactly 1/2, versus 0.5 that the probability of a boy is some value P which is equally likely to be any value from 0 to 1.

In the first case, the probability of 6 boys and 1 girl is 0.5^7 = 1/2^7. In the second case the probability of 6 boys and 1 girl is P^6(1-P) = P^6 - P^7. The integral from 0 to 1 of P^6 - P^7 is 1/6-1/7 = 1/42. Each case also has a priori odds of 1/2 of holding true.

After observing 6 boys and 1 girl, the first case now has probability (0.5/2^7)/(0.5/2^7 + 0.5/42) = 1/(1 + 64/21) = 21/129 = 0.162790697674419. The second case now has probability 1 - this, which is 0.837209302325581. Furthermore if the second case is true, P is no longer uniformly distributed. In fact its density is now proportional to P^6-P^7.

So the posterior distribution is now going to be:

With probability 21/129, exactly 0.5. And otherwise any value P from 0 to 1 with a probability density of 108/129*(P^6-P^7)/42.

Given this prior and this set of observations, any other answer is wrong. Given a different prior you would get a different posterior, but as long as the prior gives a constant probability of male/female, the only fact that matters is how many boys and girls there are.

The order of births can only start to matter if you start with a prior that gives different probabilities of different genders based on prior events. Even then it is hard to come up with a realistic scenario in which the plans of the parents would make an order of magnitude difference in the posterior distribution.

But if they are having children until they have one of each, the probability of the different observations does depend on the prior events!

The absolute probability of the observation is irrelevant. Only the RELATIVE probabilities of said observation under the different possible theories which are part of the prior.

If the set of prior theories does not include anything that depends on birth order, then birth order and the experimental design are irrelevant to the posterior conclusions.

Right, but this is exactly my point: the correct answer depends on the set of prior theories, which is exactly what the frequentist scenarios you consider are varying.

If you are arguing THAT point, then you shouldn't have disagreed with me anywhere!

We have 3 types of facts.

1. What is the set of prior theories? Probability theory says this should matter. Bayesians are explicit about its involvement. Frequentists ignore it.

2. Observed data. Probability theory says that this should matter. Everyone takes this into account.

3. Experimental design for what would have happened had something different than the observed actually happened. This matters a lot to frequentist approaches and does not matter at all to Bayes' theorem. Bayesian approaches generally do not care about it at all.

The difference between scenario 1 and scenario 2 is a fact of type 3, the conditions under which Bill and Lorena would have stopped having children. Unless you believe that Bill and Lorena's desire for one gender has a material impact on the probability of boys vs girls, this fact is irrelevant to any calculation of posterior probabilities. And is irrelevant in classical Bayesian approaches.

Yet, despite being irrelevant, it matters a lot for frequentist approaches.

1 and 3 are really the same thing. If the complaint is that the frequentist test can't tell you anything if the assumed distribution wasn't the right one (which is what's happening if you would have done something different), consider the bayesian case. There one might argue you at least still have the probability of each hypothesis given the data. But that forgets that it is only the probability of each hypothesis given the data, given that those were the only possible hypotheses. And if you admit the possibility of their being other possible hypotheses, then these probabilities don't really tell you anything either.

Anyway, not sure if we're on the same page, but thanks for the discussion.

We are clearly not on the same page. Because I think that 1 and 3 are rather different things, and you don't.

In particular 1 consists of exact statements about the likelihood of 7 births in a row being mmmmmmf.

By contrast 3 consists of statements about what Bill and Lorena's childbearing plans would have been if something different had happened.

Those are very different types of statement. There is no connection between statements of type 3 and statements of type 1 unless Bill and Lorena's state of mind makes a significant difference to the odds of the next child being a boy.

Okay, my last try.

For Bayes' theorem, we need a theory of how the data is produced given the parameter of interest. Bill and Lorena's plans certainly influence what data I observe: in scenario two, I can never observe the data BBBBGGG, but in scenario one, I can. My point is that your first category is not "exact statements about the likelihood of 7 births in a row being mmmmmmmf", it is "exact statements about the likelihood of observing mmmmmmf", which is, in fact, quite different, if you admit the possibility of Bill and Lorena having particular childbearing plans.

If you include into the priors information about the likelihood of the next child being born, you will indeed get different absolute probabilities. But you will not get different relative probabilities unless your available priors create a correlation between birth order and the likelihood of different genders for the next child if it comes. And therefore the probability of having the next birth cancels out of Bayes' formula and you wind up with the exact same conclusions from the observed data.

You certainly DON'T wind up with anything like the factor of 8 difference that frequentist techniques will see!

Is this oddity just because scenario 1 fails to account for order?

Exactly. There are more ways to get this unusual outcome in scenario 1 than scenario 2.

Reply on news.ycombinator.com