Hacker News Reader

Comment by philwelch | Hacker News Reader

There are hundreds (thousands?) of commercial content-heavy websites that continually track engagement metrics and A/B test design differences all the time, and virtually every single one of them has chosen to bet their collective livelihoods on richer and more interactive media.

View on HN

A/B testing is a form of markov-chain-monte-carlo experimentation that makes certain assumptions about the landscape it is trying to find the maximum likelihood in. Specifically, it makes the assumption of monotonicity of the likelihood or that the lack of monotonicity is of the same scale as the changes they are likely to make. A/B testing cannot break out of a local maximum of the likelihood function if the assumptions are incorrect.

A complete rewrite of the application from media heavy to plain is far smaller than the typical stepsize used in A/B testing and has some temporal effects on top of it.

Sure, but you’re implying something super weird, namely that there’s a global maximum of engagement at the “text and CSS” extreme end of the spectrum, but as soon as you add any multimedia at all, even a single img tag, there’s a discontinuous drop and then the metrics improve the more and more you add even more multimedia.

I don't think I am saying that. I am saying that the radius of curvature of a peak is potentially greater than the typical step size used in A/B testing. No discontinuity is needed, the landscape could be completely smooth.

I'm not suggesting that the extreme of text + css only is what epitomizes a good minimalist page either. I'm just pointing out that it is absolutely not a given that A/B testing has shown that the global maxima is not near it.

The article isn’t merely arguing that users prefer rich content, other things being equal. It is arguing that users prefer paywalled rich content to free plain content. This is not the kind of thing typically measured by A/B testing.

Reply on news.ycombinator.com