Hacker News Reader

> which of these groups do you think that they use as a control?

When you run an A/B test you randomly divide your users into groups, one (treatment) getting the new behavior and one (control) getting the current production behavior. So your question doesn't make much sense?

View on HN

Depends on the question. If you want to answer a question like “does change X increase engagement?” then a straight A/B test works. But if you want to answer one like “does change X increase engagement while (favoring/not favoring) group 1 over group 2?”, then an A/B test plus measuring groups 1 and 2 will not work without a control group, because without controls you don’t know if any changes to engagement for your measured groups are significant. There is some threshold of change to the engagement for the measured groups which is too small to be significant, and you should ignore results that only measure that noise.

You’re describing a multivariate test. Two groups under two different algorithm variants. Multivariate tests still require control groups.

Reply on news.ycombinator.com