Margin of Error vs Confidence Level

The margin of error is a product of two pieces. The critical value comes from the confidence level; the standard error comes from the data and the sample size. Multiply them and you have the reach on each side of the estimate, so the interval is $\text{estimate} \pm \text{margin of error}$ and the margin is exactly half the interval's width. For a proportion it is $z^*\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$.

A poll of 1000 randomly selected adults finds 47% support. The standard error is $\sqrt{\frac{0.47(0.53)}{1000}} = 0.01578$ and the 95% critical value is $z^* = 1.960$, so the margin of error is $1.960 \times 0.01578 = 0.0309$, about 3.1 percentage points, and the reported interval runs from 43.9% to 50.1%. That is where a poll's plus or minus 3 points comes from.

"The margin of error tells you how far off the poll could be" is the reading to kill. It measures one source of error only: the variability from surveying a random sample rather than everybody. Nonresponse, a sampling frame that misses part of the population, leading question wording, and respondents who change their minds all sit outside it, and any of them can move a result by more than 3 points. A survey of self-selected volunteers has a perfectly computable margin of error and no useful accuracy, which is why undercoverage and nonresponse bias are named separately.

Two smaller slips are common. A margin of 3.1 percentage points is not 3.1 percent of 47%, which would be 1.5 points. And the margin is not a wall: a 95% method is built to miss about 1 time in 20, so a true value outside 43.9% to 50.1% is not evidence the poll was run badly.

Shrinking it is expensive. The margin falls with the square root of the sample size, so quadrupling a poll from 1000 respondents to 4000 takes 3.1 percentage points down to 1.5, not to 0.8. Raising the confidence level pushes it back up. To hit a target margin you solve the formula for $n$, which is a sample size calculation.

The most-watched monthly figure that carries one of these is the US unemployment rate, which is estimated from a household survey rather than counted from everyone: the unemployment rate.

A confidence level is chosen before the data are collected and it does two jobs at once. It fixes the critical value the interval formula uses, and it names the rate at which that formula succeeds across repeated random samples of the same size. Write it $C$. The parameter never moves. The interval does, because a fresh sample produces fresh endpoints.

Suppose 240 of 400 randomly sampled adults say yes, so $\hat{p} = 0.60$ (p-hat) and the standard error is $\sqrt{\frac{0.60(0.40)}{400}} = 0.0245$. At 90% the critical value is $z^* = 1.645$ and the interval runs 0.560 to 0.640. At 95% it is $z^* = 1.960$ and the interval runs 0.552 to 0.648. At 99% it is $z^* = 2.576$ and the interval runs 0.537 to 0.663. One data set, three intervals. Nothing about the sample changed, only the level.

The sentence to stop writing is "there is a 95% probability that the true proportion is between 0.552 and 0.648." Once those two numbers exist they are fixed, the proportion was always fixed, and so that probability is 0 or 1 with no way to tell which. The randomness was spent when the sample was drawn. The 95% belongs to the procedure: it is a statement about the samples you did not take. The coverage simulator makes that visible, with the true $p$ drawn as a line and each new sample laying down one more interval to count.

A confidence level is what the method promises, not always what it delivers. For the proportion interval above the two come apart: at $n = 50$ with a true $p$ of 0.60, its exact long-run capture rate is 94.1% against a nominal 95%, because the standard error is built from $\hat{p}$ rather than from the unknown $p$. The rate a method actually achieves is its coverage probability.

Raising the level buys reliability with width and nothing else. At a fixed $n$ the 99% interval above is 0.126 wide against 0.096 at 95%, and a 100% interval would be every value from 0 to 1: certain to capture $p$ and worth nothing.