A/B testing has two big and obvious problems. One is that A/B testing can only lead to a local maxima in the best case scenario. The other is that A/B testing, being about statistical hypothesis testing, is prone to interpretation problems, which is why the changes you introduce in the variation have to be small, otherwise you don't know what you're measuring.

In other words, yes it can help you optimize the color or size of a Buy Now button. But it can't help you build a product.