Comment by bildung | Hacker News Reader

Comment by bildung | Hacker News Reader

Parent Full thread

I think we agree. I should have elaborated more.

> Wait, what? Not if you use the right test to evaluate your change.

Sure, but how often to people actually choose the right combination of test, testing tool and estimated effect size for their significance to have actual meaning?

The problem with experiments like your typical website split test is the potentially infinite population. With a big enough N, the probability of reaching significance approaches 1.

> The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical.

Yes, I know. But the magic lies in that "done properly". Have you ever approached a tool in the field that would fail my proposed test? I haven't.

    With a big enough N, the probability of reaching
    significance approaches 1.

Well, not with an A vs A experiment. Then the probability of significance at the p < 0.05 level should stay 5%.

On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care.

    Have you ever approached a tool in the field that
    would fail my proposed test?

Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments.

(If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)

Reply on news.ycombinator.com