Hacker News Reader

Comment by hackandthink | Hacker News Reader

A famous Bayesian arguing for frequentist statistics?

Gelman tries to steal the concept "Frequentism" from simple minded frequentist statisticians.

His argument seems to be:

Simple minded frequentists statisticians perform a statistical procedure once - they do not think about performing the procedure many times.

They fall into this trap (from Gelman's paper):

"3. Researcher degrees of freedom without fishing: computing a single test based on the data, but in an environment where a different test would have been performed given different data"

View on HN

Gelman is one of the few self-proclaimed Bayesians who doesn't seem to outright hate frequentist approaches. They're complementary approaches. Bayesian methods are great for combining different sources of information. Frequentist methods are great for validating that a method is working well. (For example, Gelman often recommends running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do.)

Frequentism is mostly about how to evaluate a methodology. It's pretty agnostic about what that methodology is. Bayesian methods are about combining different sources of information. In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer.

People say you might as well always use Bayesian methods then. But no matter what, you should always try to validate or poke holes in your model, and Frequentist techniques are great for that. So it's best to be familiar with both!

yes

https://stats.stackexchange.com/questions/115157/what-are-po...

> running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do

Is looking at probability distributions “a pretty frequentist thing to do”? Even when those models and simulations include _prior_ probability distributions? Sure, one can (re)define frequentist to include Bayesian models - as Gelman seems to want to do in that post. I just don’t see how this helps to clarify anything.

>In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer.

Bayesian and frequentist methods always give the same answer because they represent two different ways of translating the same mathematical ideas into English.

What do you call "frequentist methods"?

Imagine the following question: "what's the male/female ratio in gorillas?"

A frequentist method may provide the answer "[1.1 1.3] is a 95% confidence interval" based on taking a sample of zoos and asking them about the sex of the last gorilla born there.

A Bayesian method will provide a different answer - maybe one difficult to reconcile. Because it's not "translating the same mathematical ideas into English". Not only the translation is different - the "mathematical ideas" considered are different as well.

I don't think that's a complete case, you can't just say that the bayesian answer would be difficult to reconcile without offering it.

A Bayesian may put a strong prior on around the 1:1 sex ratio at birth - because in addition to that data regarding a sample of births they incorporate in the calculation knowledge about the plausible ratio coming from previous observations or biological facts about giraffes or related animals - and get a 95% credible interval (which is conceptually completely different from a 95% confidence interval) like [0.99 1.01] or whatever.

You can't just say that Bayesian and frequentist methods _always_ give the same answer without offering even a _single_ example.

What is commonly understood as 'Bayesian methods' will give answers in the form of a probability distribution. What is commonly understood as 'frequentist methods' will never do that. How can they always give the same answer then?

Thanks for adding detail. I didn't offer any examples because I want to use ones that sound representative to you.

The 95% confidence interval is in reference to a probability distribution, I'm not sure what you mean when you say that frequentist answers aren't in terms of those.

As for your bayesian answer, there is a prior that would make their result equal to the frequentist one - and in another example where priors were more obviously crucial (weak evidence) the frequentist would still use Baye's theorem. Their ensemble would be all possible worlds in which they'd ask the question.

> The 95% confidence interval is in reference to a probability distribution, I'm not sure what you mean when you say that frequentist answers aren't in terms of those.

Maybe I should have been more explicit. Let me complete what I wrote above:

"What is commonly understood as 'Bayesian methods' will give answers in the form of a probability distribution FOR THE QUANTITY OF INTEREST. What is commonly understood as 'frequentist methods' will never do that."

In the sex ratio example, the 95% confidence interval [1.1 1.3] DOES NOT mean that the probability that the sex ratio is between 1.1 and 1.3 is 95%. What it means is that when you calculate confidence intervals using this method 95% of them will contain the true value of the parameter. It's not the same thing.

> As for your bayesian answer, there is a prior that would make their result equal to the frequentist one

So it doesn't _always_ give the same answer, does it? (In the best case you get the same answer by twisting what a confidence interval means and only if the "magic" prior is used.)

If you mean that frequentist methods shouldn't be used if they give a different answer I agree!

The problems with frequentist methods have been known for decades: https://www.jstor.org/stable/2281869

They are used because the are easy even though they give the wrong answer (or the answer to the wrong question) - not because they always give the right answer.

Two bayesians can come up with different answers if they use different priors, there are rarely clear-cut rules for choice of prior (if there is a commonly accepted value, how much weight do you put on it?). Two frequentists can come up with different answers if they define their ensembles differently. A frequentist and a bayesian can come up with the same answer if the implicit prior built in to the ensemble matches the explicit prior adopted by the bayesian.

Since bayesians understand that they can't trace back their probability updates all the way back to a single ultimate prior, they do not actually talk about distributions over the quantity of interest either. Ultimately it's the same as what a frequentist would do.

When a bayesian lets a somewhat arbitrarily adopted prior stand in for the "ultimate prior," so that they can interpret their answer as a distribution over the variable of interest, they are not doing anything differently from when a frequentist fudges the exact concepts and treats their answer about the degree to which the model would predict the measured outcomes as if it was the probability of the model being true.

There are no schools of statistics that offer an unqualified "probability that the model is true," and that may be philosophically impossible AFAIK.

We agree that they don't always give the same answer then.

And also that the answers refer to different things. Bayesian methods depend on the prior probablity distribution chosen for the parameter of interest and produce a posterior probability distribution for it [1]. Frequentist methods have no concept of probability distribution regarding the parameter of interest at all.

So long!

[1] "Since bayesians understand that they can't trace back their probability updates all the way back to a single ultimate prior, they do not actually talk about distributions over the quantity of interest either." I'm having issues to understand that though. If priors and posteriors are not "distributions over the quantity of interest" what are they about?

The posterior can be anything depending on the choice of prior. It is not a distribution over the variable of interest any more than a frequentist's ensembles are. Like frequentist answers, the posterior is understood to be something that can stand in for the in fact unobtainable distribution over the variable of interest.

> It is not a distribution over the variable of interest any more than a frequentist's ensembles are.

It can be right or wrong but it is what it is!

The frequentist ensembles in the frequentist inference are absolutely not a distribution over the variable of interest. They are defined for a fixed value of the variable of interest.

The 95% confidence interval means that 95% of the intervals that you generate in the ensemble where the paramater has always whatever its unknown actual value is will include it.

Wouldn't that imply that every Bayesian calculation ever done was wrong? The chances that you'd choose the right ultimate prior function R->R from that set are measure zero.

The calculation Input -> [Model] -> Output is not wrong.

I don't know what are you trying to say but it's no longer related to whether Bayesian methods and frequentist methods give the same answers. They don't even try to represent the same questions.

PS: I didn't have time to reply to your later comment, so here is a final comment reaching for common ground.

> frequentist conceptual machinery

Just to be clear, that's not what I've been talking about. As I'm tried to make clear several times I'm talking about what is commonly understood as frequentist methods. I gave the concrete example of confidence intervals. I started the conversation with a explicit question: What do you call "frequentist methods"?

I agree that if we define "frequentist conceptual machinery" to be "probability" we can do Bayes with it.

Frequentism and bayesianism are more than separate collections of formulas, they're philosophical stances on interpreting theorems about statistics. That's where the separate language comes from. It might not be taught very often (because it's inconvenient and wordy), but it is actually possible to address anything in frequentist language, even cases where you're combining separate sources of differently-weighted evidence. The same goes for bayesian language.

The frequentist test for this attempts to see what would happen with a variety of test designs using likelihood ratio and similar statistical tests. Relaxing it you end up with Generalized Method of Moments family.

A Bayesian would attempt to compute the Bayes factor using approximate Bayesian computation resulting in more or less the same thing. You end up with various information criteria.

Both approaches then converge in using Monte Carlo techniques to evaluate the features of the whole experimental setup using simulated data.

All of the above approaches replace the problem of choosing the test/design based on data by the researcher with one by a data driven algorithm with known properties.

Reply on news.ycombinator.com