Hacker News Reader

Comment by jules | Hacker News Reader

This series of articles is just screaming for a Bayesian approach, rather than this ad hoc approach.

That approach was discussed by Anscombe, and I wrote up a summary in the Custora Blog. However just because an approach is frequentist or 'ad-hoc' does not necessarily mean that there is anything wrong with it. The bayesian approach requires making assumptions about the number of visitors to your site after you stop the test, which isn't really any less adhoc than picking an error cut off.

http://blog.custora.com/2012/05/a-bayesian-approach-to-ab-te...

It is no less ad hoc, but it is clearer. Instead of making assumptions about the right p-values, you make assumptions about real world quantities.

And if you have actual data/projections about future visitors, it is less ad hoc.

I like that article, but have one major qualm about it. Everything that you do in a Bayesian model depends on the prior. Yet you often see - as there - someone tell you, "Here is the rule to use" but without telling you the prior.

However the prior actually matters. For instance when you look at what Nate Silver did, most of the mathematical horsepower went to determining a really good prior to use based on historical data. And armed with that he both can and does make inferences. (Which he's willing to publish.)

That said, the Bayesian approach is conceptually so much better that Bayesian with a questionable prior can be better than a frequentist approach.

Finally the fact that a Bayesian approach needs a somewhat arbitrary planning horizon does not particularly bother me. Financial theory tells us that businesses really should apply a discounting factor to future projected income, and when you apply an exponentially decaying discounting factor, the weighted number of future visitors generally comes out to a finite number. And yes, there are a lot of arbitrary factors in how you get to that number. But you can generally do it in a reasonable enough way to be way less sloppy in your A/B test than every other part of the business is. Heck - you can just say that your planning horizon is 1 year, and use the expected number of visitors in that time as a cutoff.

Anyways I'd like to eventually get into this kind of issue with this series. But whether I can, I don't know. It certainly will be hard if I keep on trying to pitch it to the level of mathematical background that I've been aiming for so far.

If you look through the proposed plan, you'll find that I do intend to show Bayesian approaches. But as I get there, I want to explain the trade-offs.

In particular the choice of a Bayesian prior is as arbitrary as anything that I've done so far. And how you do it hides some important implicit assumptions. That said, a Bayesian framework does have substantial conceptual advantages over the frequentist approaches that I've used so far.

At my job we use a Bayesian approach, or at least we've started using it. I'll ask around to see if they'll let me publish it.

The Bayesian approach is indeed quite simple and understandable.

Reply on news.ycombinator.com