Hacker News Reader

From 3 years ago but it certainly still holds true. As mentioned in the comments, many PPC'ers use http://www.splittester.com to help determine if there's enough data to determine a winner.

SEOBook recently opened another site focused primarily on PPC at PPCBlog.com and this was a recent discussion in the forums.

View on HN

It's worth noting that splittester.com calculates 95% confidence that those two (identical) ads will have different long term response rates.

Which should be a pretty big wake up call to anyone not doing the math. 1 in 20 events happen all the time, yet naively reading the results makes it look like one is almost certainly better than the other.

Agreed, this is such a common problem. Another really common manifestation is when you A/B test while looking at multiple variables. If you observe the results of 20 variables, you should expect one of them to have 95% confidence!

Sure because 95% leaves 5% of room to cherry pick aberrations.

Reply on news.ycombinator.com