Hacker News Reader

Comment by stdbrouw | Hacker News Reader

Hmmm, again, there are certainly ways that interactions between visitors can cause statistical dependence, but not in the specific case you mention. Let's take an A/B test on a referral funnel. If a user invites all of his friends, and his friends then visit the site, they will be randomized over A and B just like the original user, and so any effect that is not due to changes in the referral experience will simply not matter because it will contribute equally to both groups.

Without better examples it's very hard to judge whether this is a real problem.

View on HN

I understand if you think this is a non-issue, though I don't agree. The speaker I referenced about asked the statisticians at his company about this and they said it was a non-issue because things balanced out. He thought that was an idealization and claimed to have tested it building in some real world data, and reported that interconnected data of this kind drastically affected confidence levels. He didn't get into the details of how he measured interconnectedness, however.

The example you give seems to me to oversimplify the issue of complex interconnections between data points, as if the traffic on a real website came from one set of referrals, while in reality its much more complex, with referrers inducing other referrers and a variety of campaigns, postings, etc. influencing each other, and over time, overlaid in a fairly complex pattern. In other words, a bunch of interrelated data, very little of which is actually independent of other items.

I'm not really asking for an explanation of this in the comment thread here; what I'd like to know is, if there are any studies or other publications that deal with the issue of how to evaluate tests run on interconnected data of this kind.

There are absolutely ways to deal with what you call interconnected data, as I mentioned earlier: paired tests, corrections for autocorrelation, nonparametric and bootstrap methods for non-normal data and so on. But barring any examples of what you mean with interconnectedness in this context, it's hard to recommend any studies or publications because there is no One Method Of Interconnectedness Correction.

Also, statistics deals with many idealizations but the idea that randomization allows you to cleanly measure the effect of an intervention in the face of what would otherwise be confounding is simply not one of them. Sorry to disappoint, but with all you're telling us it simply sounds like the speaker was clueless.

Well, if he was clueless then two very large and successful tech companies had a clueless guy running their AB testing and showing great results in each context.

I'm certainly not looking for "One Method for Interconnectedness Correction" (especially not, as you put it, with each word capitalized). I'm looking for studies or papers that might have addressed anything like the effect of interconnectedness of web data on AB testing. I think you're saying, you don't know of any, and also that you personally don't think it's a real issue.

Reply on news.ycombinator.com