How to determine if your A/B test is statistically significant | Hacker News Reader

How to determine if your A/B test is statistically significant

blog.asmartbear.com

87 pointszackattack14 comments

I like this rule of thumb a lot. It's very straight-forward and easy to remember so you can apply it quickly to see significance and then use a more rigorous method if necessary.

There are tons of tools that can test statistical significance for you:

For example, http://www.splittestcalculator.com/ or http://www.usereffect.com/split-test-calculator

But what about the vast majority of people who don't click either ad? That's the "ad impressions" that didn't lead to a click. Shouldn't those count somehow in the statistics?

No, they shouldn't; those are "mistrials."

I don't think those are mistrials. Clicks vs impressions is a separate test and should be treated as such.

The test only counts if an even amount of impressions is given. The viewers aren't given a choice between ad A and B, they have a choice between A (or B) and nothing. I don't think the author explained this well enough. Obviously (to everyone here) the test is inconclusive if one ad is shown much more than the other.

We can do this for you at the click of a button at 360voltage: http://360voltage.com/blog/2009/09/ab-split-testing-out-of-d... (For Free)

</self promotion>

The statistics are a little bit off - the Chi-squared isn't appropriate for the example he gives (sample size too small), although it would be fine in practice. What you really want in that situation is the exact binomial test.

Good point! Although with around N > 25 you're OK. See: http://faculty.chass.ncsu.edu/garson/PA765/binomial.htm

And besides that, she's clearly leading the witness (Hammy). See the outtakes: http://www.youtube.com/watch?v=VmZ7p3ExIc8

So, he's just explained Standard Deviation?

You'd be amazed how many smart people don't have a decent grounding in statistics, so any easy reference to explain this is nice.

Fair point, and yes it's quite informative from that point of view. I didn't like his approach however, he seemed to be beating around the bush a lot with phrases like

  I'm here to rescue you with a statistically sound yet incredibly simple formula

than giving a more honest 'hey, what you need is Standard Deviation, and here's how it works!'

No Torn, you're still not using the correct statistic.

THIS HAS NOTHING TO DO WITH STANDARD DEVIATION.

Also, how can you say that the article isn't "honest," especially given the complete, precise explanation for the stats at the end.

Seems I'm haemorrhaging karma over this. I misjudged the method used (Pearson's Chi Square, as pointed out by hn user pibefision). Fine, that'll teach me to skim read and comment.

My point above was that the author could have come out and labelled it as such in the article.

No, he explained Pearson's Chi Square.

Reply on news.ycombinator.com