Statistics: How Many Would You Check?

blog.jpalardy.com

7 points·cawel··5 comments

I've always used a Clopper-Pearson interval for binomial confidence intervals, but never questioned why (it's the default in ROOT). I found http://en.wikipedia.org/wiki/Binomial_proportion_confidence_... quite useful. Sounds like using Clopper-Pearson is safer than using a Wilson interval .

It's a trade off. Clopper-Pearson can be overly conservative in many instances. I tend to use Jeffreys interval personally, which is a Bayesian method.

Brown et al. give a formal treatment of the subject with simulations showing actual coverage probabilities for the various methods. It is worth a read if you want to really dive into the subject.

ref: Brown, Lawrence D.; Cai, T. Tony; DasGupta, Anirban (2001). "Interval Estimation for a Binomial Proportion". Statistical Science 16 (2): 101–133

Thanks. To save others the trouble, a (apparently accessible to all?) link to that article is at http://projecteuclid.org/download/pdf_1/euclid.ss/1009213286

It looks like Clopper-Pearson is the only method guaranteed to not underestimate the size of the confidence interval (which makes sense, given its derivation), but it almost always overestimate its (unless your confidence interval exactly matches the discrete p-values allowed by the binomial distribution).

To be a pedant and assuming your confidence interval is at the 95% confidence level. Taking 73 checks will produce the 95% lower bound 95% of the time if you take any random 73 samples.

Right, in this specific case it's awkward to mention both 95% (confidence) and 95% (lower bound) in the same sentence.

It could have been clearer if I used different values.

Reply on news.ycombinator.com