Hacker News Reader

I think that's what you need to keep in mind. The "some" that are unhappy are such a small fraction of the Facebook/Twitter userbase that it's a clear net win to make these changes. I can guarantee the changes were A/B tested, the metrics gathered, and a decision made. Of course there will be a slice of the userbase who doesn't like the change, but that's obviously already accounted for in their planning and forecasting.

View on HN

A/B testing has two big and obvious problems. One is that A/B testing can only lead to a local maxima in the best case scenario. The other is that A/B testing, being about statistical hypothesis testing, is prone to interpretation problems, which is why the changes you introduce in the variation have to be small, otherwise you don't know what you're measuring.

In other words, yes it can help you optimize the color or size of a Buy Now button. But it can't help you build a product.

> One is that A/B testing can only lead to a local maxima in the best case scenario.

What you mean is: it can only lead to a local maximum in the worst case scenario. Even blind guesses can occasionally lead to a global maximum, and A/B testing is better than a blind guess.

Again, another insightful and accurate assessment!

Reply on news.ycombinator.com