Hacker News Reader

btbilly: I read your article but before I digg into the math and try to understand it - can you please take a look at the tool I use now http://mystatscalc.com and see 1) how it might work 2) do you think it's giving correct results?

Waiting for rest from the series!

Grzegorz

View on HN

That tool is giving you a standard frequentist p-value. Which means that if you insist on 95% confidence, and take multiple looks, then you'll have more than a 5% chance of eventually getting to confidence randomly.

If you stop your tests on the p-value cutoffs that I gave in my third graph, you'll get very strong a priori guarantees that the decisions that you make will be right. The downside is that you won't have any guarantee of making decisions in reasonable time. But that is the subject of the next article.

Reply on news.ycombinator.com