Coverage Probability vs Confidence Level

Fix a method, a sample size, and a true parameter value, and the coverage probability becomes a computable number: the probability that the interval the method is about to produce contains that value. It is the rate the method achieves. The confidence level is the rate it advertises. When the two agree the method is exact, and when the first falls below the second the method is undercovering.

For the one-proportion interval $\hat{p} \pm z^*\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$ you can compute it exactly, because a sample of $n = 50$ has only 51 possible outcomes. Build the 95% interval from each count $x$ and ask whether it contains the true $p$. With $p = 0.60$ the counts 24 through 36 give intervals that capture it and the other 38 miss. Adding the binomial probabilities of those thirteen counts gives 0.9406, so the coverage probability is 94.1%, not 95%.

The claim to be careful with is that a 95% method covers 95% of the time. The 95% is an input you selected. The coverage probability is an output you have to compute, and for this interval the two rarely match. The gap is not always small: at $n = 20$ with $p = 0.90$ the same 95% formula captures $p$ only 87.6% of the time.

More data does not repair it smoothly. Because the count $x$ is discrete, coverage oscillates instead of climbing. At $p = 0.50$ the 95% interval covers 95.7% of the time when $n = 30$ and 91.9% when $n = 40$: ten more observations made it worse. The long trend is toward the nominal level, but no single increase in $n$ is guaranteed to improve on the one before it.

This is what the large counts condition is for. Requiring at least 10 successes and 10 failures rules out the collapse cases, where $\hat{p}$ lands so near 0 or 1 that the standard error vanishes. It does not make coverage exact and it does not promise a small shortfall: $n = 40$ with $p = 0.50$ clears the condition twice over and still covers 91.9%.

A confidence level is chosen before the data are collected and it does two jobs at once. It fixes the critical value the interval formula uses, and it names the rate at which that formula succeeds across repeated random samples of the same size. Write it $C$. The parameter never moves. The interval does, because a fresh sample produces fresh endpoints.

Suppose 240 of 400 randomly sampled adults say yes, so $\hat{p} = 0.60$ (p-hat) and the standard error is $\sqrt{\frac{0.60(0.40)}{400}} = 0.0245$. At 90% the critical value is $z^* = 1.645$ and the interval runs 0.560 to 0.640. At 95% it is $z^* = 1.960$ and the interval runs 0.552 to 0.648. At 99% it is $z^* = 2.576$ and the interval runs 0.537 to 0.663. One data set, three intervals. Nothing about the sample changed, only the level.

The sentence to stop writing is "there is a 95% probability that the true proportion is between 0.552 and 0.648." Once those two numbers exist they are fixed, the proportion was always fixed, and so that probability is 0 or 1 with no way to tell which. The randomness was spent when the sample was drawn. The 95% belongs to the procedure: it is a statement about the samples you did not take. The coverage simulator makes that visible, with the true $p$ drawn as a line and each new sample laying down one more interval to count.

A confidence level is what the method promises, not always what it delivers. For the proportion interval above the two come apart: at $n = 50$ with a true $p$ of 0.60, its exact long-run capture rate is 94.1% against a nominal 95%, because the standard error is built from $\hat{p}$ rather than from the unknown $p$. The rate a method actually achieves is its coverage probability.

Raising the level buys reliability with width and nothing else. At a fixed $n$ the 99% interval above is 0.126 wide against 0.096 at 95%, and a 100% interval would be every value from 0 to 1: certain to capture $p$ and worth nothing.