Hacker News Reader

I read this as marketing bluff:

Testing one difference guarantees that one side will beat the other (whether or not there is a true difference).

Multiple testing has a danger of the results appearing inconsistent/random (again, whether or not there is a true difference).

Encouraging people to test one difference benefits MailChimp, for exactly the reason that they say - the results will always appear to be relevant.

View on HN

That’s why you must do at least a back of the envelope test to see if the results are significant. If you send two set of n mails, the expected variations is something like sqrt(n) (I don’t remember now, probably sqrt(n)/4 or sqrt(n)/2, or something like that.) Ask a statistician for the exact number.

One easy experiment is to do the null A/B test. Send two sets of identical mails and analyze the difference of the result of the two sets. Usually you will get more variation that most people naively expect.

Reply on news.ycombinator.com