Advanced A/B testing: Make more profit, learn more about customers

custlabs.com

12 points·efm··2 comments

"data points required for confident results scales linearly with number of distinct states being tested"

My memory of the maths behind this is poor, but I'm not sure this is actually true: I would understand that (all things being equal) the number of required data points grows faster than linearly. The reason is that not only are you spreading your users across more states, but you also need much stronger results to make a conclusion: see the use of ANOVA vs (eg) two sample t tests http://en.wikipedia.org/wiki/Analysis_of_variance.

Of course, if you run a series of base vs variant tests, where the winner stays on, you run into the exact same class of problem too. Meaningful AB testing is tricky.

EDIT: when I say not sure, I mean it. I wish I knew this better (rather than just being aware of the existence of gotchas). If anyone has any great insights on the impact of this stuff, or how to deal with it, I'd love to hear them.

The wording in the original article isn't entirely clear to me. If the test is across different "states" of one factor, then the number of required datapoints should be approximately f*n. So for example, testing four different button placements is like testing four different treatments:

button placement 1 button placement 2 button placement 3 button placement 4

However, if multiple states also coincide with testing multiple factors, such as four different button placements and also three variations of text on the buttons. Then the number of combinations is 12. In this case, if we add another text variation, we would be adding four more treatments.

Reply on news.ycombinator.com