Determining A/B test sample size
37signals.com
37signals.com
The issue is the proper tradeoff between exploration and exploitation. What drives the decision is outstanding uncertainty conditional on the data observed (not conditional on the null hypothesis of zero effect and some non-sequential iid sampling process), the discount rate (which is totally absent in this article), and the reward structure (which is not a Type I and Type II error).
The absurdity of the frequentist approach is clear from the admonition not to look at the results of the tests too often.
I'm sure you're aware of this, but I'm just trying to clarify the idea for other readers.
The idea is not well-illustrated in the article. (Although the article does provide some usable guidance until the whole Bayesian framework gets built and populated with correct parameters, like the reward structure.)
So, to be concrete -- Suppose you're flipping coins and you figure (by some procedure) you need 100 flips to reach significance. By the 70th flip, you observe that p(head) ~= 40/70 ~= 57%, so you decide to stop the test because clearly you're not dealing with a 50/50 coin. That's not OK, because you'll always see favorable and unfavorable excursions in a series of coin flips -- if you choose to stop in the middle of such an excursion, you'll bias the result. You've made the stopping time dependent on the observed values.
In some situations you can do this (it's related to http://en.wikipedia.org/wiki/Optional_stopping_theorem), but the way that I described above is not one of them.
There are more general conditions than deciding in advance. The relevant theory in that case is the sequential likelihood ratio test (http://en.wikipedia.org/wiki/Sequential_probability_ratio_te...).
Typically, in an SLRT for a series of independent trials, you compute a running sum of likelihood terms (accumulating as you go) and then at some point you stop if the sum goes above one line or below another line. The separation between the lines tells you how much variability is "allowed" in the partial sums.
What's confusing is thinking about the sampling distribution. But what might have happened in some other world is of no consequence if you condition on the data rather than the parameter.
This is the likelihood principle. http://en.wikipedia.org/wiki/Likelihood_principle. See the example there and how it relates to sequential trials. It's actually rather deep. Other good links are:
http://books.google.com/books?id=_ravDT9e8nMC&lpg=PA17&#...
I'm not even going near the question of whether the Bayesian approach is "better" than the frequentist approach.
I was trying to point out that the frequentist analysis in the OP does make assumptions about the nature of the experiment (that you will run exactly N trials) and that if you break those assumptions by stopping the test for some N' < N because the answers are looking good, then you'd better understand that your earlier analysis did not apply.
And in another reply, I wanted to add that there is a frequentist answer (the Wald test) to the practical question: Can you widen the scope of the analysis so that I can stop early if I'm getting results that point strongly in one direction?
Being sure that your assumed sampling distribution matches the actual experiment is key, even in the Bayesian case.
My graduate statistics class was taught from Berger, your second link, so I'm broadly sympathetic to the "Bayesian choice" -- but more important, I wanted to give some usable insight to someone who just wants to do an A/B test.
All this confusion goes away if you realize you are interested in p(lift | data) rather than p( data | lift=0). The sampling distribution -- the distribution of the statistic under repeated sampling, p(data | lift=0) -- does not play a role in Bayesian statistics. Obviously the "model" (likelihood/prior) does, but this doesn't include the experimental procedure provided that the experiment is only based on observed data.
AB testing, as a decision procedure, is an area where I don't think the standard frequentist - Bayesian debate applies. The Bayesian decision rule is the only profit maximizing solution. That said, I"m sympathetic to being practical. But all the confusion and conflicting advice related to AB testing stems directly from trying to fit it into a frequentist frame.