The biggest problem I have with A/B or Split testing is that it makes it easy for inexperienced people to assume there are multiple right answers to every problem and that they all warrant 'discussion' or testing... Most often this is not the case, and people who jump straight to A/B everything are the sort who don't really have/trust any experience and would rather spin wheels and throw spaghetti at the wall.

That said, sometimes there are two (or more) options that warrant testing (say a marketing strategy) and then it works great.

Test strategy, don't test everything, don't test design principles.