Comment by kem | Hacker News Reader

Comment by kem | Hacker News Reader

I'm getting frustrated by the Bayesian train at the moment, as its drawbacks get glossed over. "Oh yeah, there's priors, but they're not important for X, Y and Z reasons."

In large samples, the frequentist and Bayesian methods are the same, so then why does it matter? In small samples, the prior becomes significant, so why use it if it shapes the estimates depending on what you use? You could turn these arguments for Bayesianism on their head.

I'm actually neither Bayesian nor frequentist, or both, depending on how you look at it, but I think Bayesianism is being overhyped. Sure, the prior is part of the model, but if you can estimate something without adding extra baggage, why not?

Imagine doing a meta-analysis, and now having all the extra heterogeneity due to priors. Why add that?

My guess is a lot of the appeal of Bayesianism has to do with the success of the machinery surrounding it, like MCMC, which is sort of automatic and has certain other appeals. As people realize you can do stochastic optimization with raw ML inference, some of the appeal will probably dissipate a bit (although not entirely).

If the inferential question you're interested in is, "given data X, what do I conclude about underlying cause/variable/parameter T?", then you are a Bayesian, like it or not.* Sure you can define a likelihood function p(X|T), but that doesn't give you p(T|X) unless you multiply by a prior p(T).

Now certainly p(T|X) is not always the question, but in the vast majority of cases people do want to use data to draw conclusions, and will misinterpret likelihood-based confidence intervals as posterior credible intervals because the latter are what they intuitively wanted. By doing so they implicitly assume a flat prior regardless of whether that that is reasonable or even mathematically coherent for the problem in question.

The Bayesian argument is not that you have to use an informative prior (though often this can be very helpful!), but that since some sort of prior is mathematically necessary to answer the questions that people intuitively want statistics to answer, we should make that explicit and try to understand how the prior affects our conclusions, not just sweep it under the rug.

* If the question you're interested in is, "if I run some method in many repeated trials, how often will it identify the true parameter?" then you are a frequentist. I think it's much rarer for this to genuinely be someone's intuitive question, but it's certainly valid. And of course it's valid to ask both questions at once, in which case you might end up analyzing the frequentist properties of a Bayes-derived method.

No, if you are drawing conclusions from only the data presented you are not doing Bayesian. Further, there are more than 2 options.

and they are?

you can draw conclusions from only the data presented to you doing bayesian inference.

You are ignoring the prior probability.

Supose you don't know what color a car is, so you look and see it's green and then you look again and still green. Now, what's the Bayesian calculation.

Well how accurate are your eyes, how stable is car color ect.

Have you heard of the universal prior? It allows you to do exactly that with Bayesian statistics.

Although it is so esoteric that one might argue that it is more the field of algorithmic statistics which merely employs Bayesian.

https://en.m.wikipedia.org/wiki/Algorithmic_probability https://en.m.wikipedia.org/wiki/Solomonoff%27s_theory_of_ind...

If you are trying to answer the same question in the same way there isn't much if a difference.

Frequentist and bayesian statistics are different paradigms, not different number crunchers.

---

I talk about this with graphs and such in my blog post (https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of...) but here's the short version:

Common question in our line of work: when should I end my A/B test? End too early and I have wrong conclusions, end too long and it's expensive. Classic exploration vs. exploitation.

Frequentist statistics will tell us to suck it up and just choose a sample size, or write down every time we look at the numbers to adjust for peeking effect, or assume the worst case that we always peek and wait a very long time.

Bayesian statistics tells us not to use a single termination condition, but adjust the samplings according to our continuously updating priors. (This is what Google Analytics does. https://support.google.com/analytics/answer/2844870) In fact, this turns out to be the mathematically proveable optimal solution for maximizing payoff. This latter approach is beyond the reach of frequentist methods because it's just a completely different paradigm.

The question of frequentist and bayesian not "which is more correct", but rather "which paradigm best matches the question I have?" If your testing paradigm is fixed sample sizes, choose frequentist. If it is iterative (like a lot of ML), choose bayesian.

Actually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data[1]. Or, if you want to get fancy you could use a multi-armed bandit approach which is probably optimal in many situations in perhaps a more robust way than many Bayesian methods[2]. Really both can work well. My advice is, use whichever you know well enough to utilize effectively!

[1]: https://en.m.wikipedia.org/wiki/Sequential_analysis

[2]: https://en.m.wikipedia.org/wiki/Multi-armed_bandit

> Actually, in the frequentist paradigm you could choose to run a sequential hypothesis test which will end when you've acquired sufficient data

And, as he said, you have to make adjustments to account for these interim analysis.

Right, as I said, it can be done with frequentist statistics. This is what Optimizely does (http://pages.optimizely.com/rs/optimizely/images/stats_engin...). But (1) it is not simple and (2) it is not optimal.

---

Agreed 100% about multi-armed bandit which is what I was referring to. And the canonical solutions are in fact Bayesian :) See the Google Analytics link or lookup "Thompson sampling"

From your Wikipedia link:

"Probability matching strategies are also known as Thompson sampling or Bayesian Bandits, and surprisingly easy to implement if you can sample from the posterior for the mean value of each alternative."

Oh yeah there are a few Bayesian methods which work great, Bayes-UCB is another. Personally though I think KL-UCB or just plain old UCB would be the ones I'd choose. Like I said earlier, I think these techniques are like programming languages: choose the one you know well enough to get the job done with it.

> so then why does it matter?

I recommend reading this[1] essay which explains exactly the difference between the two methods. (I'm linking to part 3 because it specifically answers your questions, but the entire series is worth reading)

Why does it matter? Frequentist and Bayesian methods offer different interpretations of what your result means.

    ... speaking broadly, frequentists consider model parameters
    to be fixed and data to be random, while Bayesians consider
    model parameters to be random and data to be fixed.

This distinction is important, because...

> the frequentist and Bayesian methods are the same

...this isn't always true. See [1] for the details.

[1] http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...

Frequencist methods do not remove the prior, they merely hide it, make it implicit. Usually the implicit prior is reasonable but being blind of your prior still risks making you trip on subtle trade-offs and biases for more complex problems.

Reply on news.ycombinator.com