One of the big problems with AB testing is that the people running tests within a company often don't have a strong understanding of the statistics behind the tests. And unfortunately, they often have little interest in performing statistically valid tests.

They might be from the marketing department, they might be web-designers, they might even be growth hackers (eek), but most likely they are not statisticians.

Their motivations may not align with running tests correctly. E.g. they may want to strengthen the argument for their design over someone else's competing design, or they may be looking for number to serve up to their boss at their monthly review meeting. But performing a statistically-valid test is low on their list of priorities.

I love articles like this that go into the minutiae of how to perform AB tests correctly. But I think they only speak to a small portion of the people who are doing AB tests.