Hacker News Reader

Comment by bpodgursky | Hacker News Reader

In the real world, we often have to work on a "preponderance of evidence" standard to actually get things done.

Especially if the second option is cheaper and faster, there's IMO no bayesian prior that the professional ad being better (the null hypothesis) is true.

So... seems like useful data to me.

View on HN

I think there can be prior that a professional looking ad can generate more clicks. Your argument shows a lack of statistical understanding - conditional on this data, the Bayesian approach would be to update the prior (whether A is better, or they’re equally as good) with the data collected. With such a small dataset, you might end up with a belief that there’s a 60% probability that B is better than A, but that’s not significant enough to conclude that B is in fact better than A, as you still have a lot of uncertainty.

With a prior that A is superior, you may still end up believing that A > B after updating, because there’s just so little data.

Yes, but the significance is high here; it's a pretty (un)lucky outcome to get if A and B are equivalent, let alone if A is better than B.

I addressed in my second sentence that I disagree with that prior. I understand the statistics perfectly well.

And my main point is that a 60% probability is in fact actionable in the real world, in a situation where you are forced to take action with incomplete information. Assuming you are running an ad campaign, you have to choose one of the two.

P=.95 still is an arbitrary threshold, even if it's a commonly used one.

Reply on news.ycombinator.com