Does this make sense? If the p-values are not very good at conveying "confidence in the prediction," is this another argument in favor of a more Bayesian approach to statistics? Any thoughts would be appreciated!
The Bayesian approach would be to say that the thing you generally want to know isn't {the probability of getting this result by chance if the hypothesis is false}, it's the probability the hypothesis is true, given everything you know up to and including the new data.
So they would calculate p(hypothesis is true given the new data) using Bayes theorem -- which requires inputting what they thought p(hypothesis is true) was at the start of the experiment, before the new data came in.
http://www.yudkowsky.net/rational/bayes is a decent explanation.
On the second point, frequentists very commonly split training and testing datasets, and so this practice is pretty much orthogonal to whether you're using Bayesian or frequentist methods.
From the first sentence of the parent comment: "From what I understand, p-values are typically used by frequentist method for verifying that the "prediction" seems correct."
If my reply seemed glib, it was because I thought the question was so far off base, that graffitici almost certainly hadn't read the article. To relate to the topic under discussion, the distribution of possible comments produced by people who have carefully read the article is extremely unlikely to result in a post as extremely misinformed as the one above.
If there was a simple statistical misunderstanding, perhaps that would be worth clearing up. Instead, the parent comment used some statistical words, but was very nearly incoherent. Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.
I disagree. There's an alternative hypothesis that could explain a comment like that - the author harbors some confused views about frequentist and Bayesian approaches to statistics. I'm inclined to believe that this hypothesis is right, because I recognize that comment as something I'd write myself back when I was more confused about this topic. Reading the article is unlikely to affect this particular issue.
> Perhaps I shouldn't have said anything if I wasn't willing to write a protracted essay about the differences between frequentist and Bayesian inference.
I think even one sentence explaining the gist of the author's confusion would be enough. Plenty of essays have been written on the topic, but one needs to be pointed in their general direction in order to benefit.
Bayesian stats aren't related to training and test sets, like in machine learning; scientific experiments rarely partition data like that. A Bayesian analysis answers the question, "Given the evidence I just collected, how should I adjust my estimate of how likely something is?" But Bayesian analyses have issues too: the biggest is not knowing what your initial, "prior" probability should be. (After all, that's usually what you're trying to find out!) With something simple like a coin flip, you have a strong prior assumption of 50% probability of heads.
And neither Bayesian nor frequentist methods address effect size. If you collect enough data, you can detect extremely small (real) effect sizes that pass statistical tests, but are still meaningless outside the context of publishing a paper. Rather than analysis type, we should incorporate more discussion of effect size, especially as the number of data points increase.