Comment by tel | Hacker News Reader

Comment by tel | Hacker News Reader

Parent Full thread

The difficulty I tend to encounter in promoting Bayesian statistics among scientists is the "sudden" appearance of a statistical model. Too many people go one further than believing frequentist statistics are answering "What is the probability that H0 is true" but instead just fully over the analysis and believe that frequentist methods tell you, simply, accurately, objectively, whether an experimental treatment is "significant".

If you try to press on what people believe "significant" to mean it gets ugly fast, but it's generally a good thing and definitely necessary to publish. If you don't get significance it's just because you need a bigger n. If you can think of some factors or covariates then you really need to use ANOVA.

Stating that there is anything more complex to looking at data and deciding what it means is practically unthinkable.

Likelihood ratios are definitely nice, though. I've sort of gotten people to think about it at a high level by talking about "information flows" and log likelihood values.

A separate problem, not dealt with in that particular essay, is the quite hideous degree to which average scientists don't understand the statistics they use.

I would put a good deal of the blame for this squarely on frequentism as well. Bayesianism isn't hard to understand, it's just takes an effort of the teacher to explain well - I've made certain notable efforts in that direction myself. Once you do get it, you get it.

I think, roughly, the blame goes out to Fisher and anyone else who promoted the "Recipe for Understanding the World" style statistics. It's not that people are being blocked by their understanding of complex frequentist methods but instead the idea that they don't need to understand anything more because statistics is just a black box you use for verification.

Insert results, get a green or red "significance" light, move on.

I think you're on to something regarding "significance." Over in my dept. we like to say that significance is a measure of sample size. The question, then, hinges on whether or not something has practical significance. Because we've built the whole research reputation incentive structure on the .05 significance level, studies can be designed to get that.

A great article on the problem with statistical significance (by a Frequentist) is here, called Why Most Published Research Findings Are False: http://www.plosmedicine.org/article/info:doi/10.1371/journal...

The punchline seems to be, well, that there's also a large human element that contributes to the problem. I think it's one thing to rail on the Frequentist way-of-thinking; it's entirely another to state that the institution of scientific research built on it creates unwanted incentives.

If you plan experiments and the like, the definition of a significance level (always) comes with a definition of sample size.

The relationship between significance level and sample size is reliant on a complex set of assumptions to say the least, and, when everything is stripped away, is perhaps best seen as a way of discovering just how difficult it will be to deblur the world. What power prescription we need.

Often (always?) these constraints are all so very much more complex than Gaussian power analysis states. You do it as a way of sketching the depth of a problem I think, not much more.

The linked paper is a pretty clear introduction of the high level problems. I think it's perhaps a little more grim than necessary, but then again that might just be my own bias.

Reply on news.ycombinator.com