6 A/B Tests That Did Absolutely Nothing for Us

Nice article as always, thanks Alex!

What I’ve found is that a successful approach to A/B testing is really dependent on the type of company you’re operating and product that you’re offering. Small cosmetic changes to the UI or copy often result in equally small changes to click-thru or conversion rates, and so these A/B tests require relatively greater levels of statistical power in order to achieve significance.

For mega-traffic companies like Google or Amazon, these kinds of tests are worth it because a sub-1% lift still contributes substantially to their bottom line. They also have the traffic numbers to properly power tests of smaller changes in a reasonable amount of time.

But for everyone else, ‘shallow’ A/B tests of a button color or call to action will often yield inconclusive results because they don’t the traffic numbers. For these types of companies, we’ve seen that deeper changes to the product, UI layouts or entire UX workflows are what move the needle. Designing these tests require more thought and development work up-front – but at least you’ll be making substantial improvements in an experimentally rigorous way instead of just spinning your wheels with some one-off design tweaks.

To avoid these kinds of disappointing tests, another thing to consider is setting a minimum detectable effect. The idea here is that validating a small change in improvement requires more statistical power (i.e.: more test subjects) than validating a large change, and at some point in order to justify a continuation of the test you’ll want to achieve some minimum amount of lift. Once you can say with statistical confidence that this desired lift isn’t achievable, you can stop the test earlier and move on to the next.

Most importantly, you should be designing these tests with empathy for your audience. Ask the questions: What changes can I make to my product or website that would motivate my users to take the actions I want to them to take? What are they looking for? What do they care about? And why? More often than not, I think you’ll find that the answer is not ‘a different button color’ :-D

In the end, A/B testing is really a very unsophisticated way of answering the question ‘What works better?’ We’ve done a lot of research into better solutions to this problem, and have found that an automated approach using a learning algorithm almost always leads to faster results and higher average conversion rates. You can read more about that here: http://splitforce.com/resources/auto-optimization/