Confidence Games

It may be an "easy interpretation," but it doesn't seem like a valid one: "The 95% probability relates to the reliability of the estimation procedure, not to a specific calculated interval." - https://en.wikipedia.org/wiki/Confidence_interval#Misunderst...

Similarly, the given example does not seem to represent how statistical confidence intervals work. As just mentioned, the 95% refers to the reliability, not to the probability that a specific interval contains the true parameter value! (In fact, the example is missing which parameter is being estimated.)

---

Let's see if we can rework the example to show that the 95% CI is actually well-defined and not arbitrary. To be specific about the problem:

- First, the population is given to be the uniform distribution between 0 and 1.

- Second, we must choose a parameter to estimate. For simplicity, let's suppose we're estimating the population mean. (Note the true population mean is 0.5.)

- Third, each sample is an independently, identically distributed single number from the population (we technically also compute its mean, which is the number itself).

---

Here is the challenge: Given an arbitrary sample, can we provide a confidence interval (CI) such that 95% of all CIs computed in this way contain the population mean 0.5?

Here is a Python program showing that given a sample mean x, the 95% CI is [x - 0.475, x + 0.475]:

  from random import random
  
  NUM_SAMPLES = 1000000
  ERROR = 0.475
  POP_MEAN = (1 - 0) / 2
  
  means = [random() for _ in range(NUM_SAMPLES)]
  num_correct = sum(x - ERROR <= POP_MEAN <= x + ERROR for x in means)
  
  print(f'% CIs containing the pop mean: {num_correct / NUM_SAMPLES}')

Experimentally, you can see that as NUM_SAMPLES gets larger, the percentage of intervals that contain 0.5 approaches 95%.

I agree that often in practice a confidence interval will end up being similar to a Bayesian credible interval. However, having a flat prior is not enough to guarantee this: the post gives an example with a uniform prior and a valid 70% confidence set that only captures 37.5% of posterior mass, and even an example with a uniform prior and a valid 70% confidence interval that captures 0% of posterior mass (because, similarly to your example, the set is empty).