Sampling variability

Draw a random sample, compute a statistic, put it back and draw again: the number changes. That change is sampling variability. It is a property of the procedure rather than a fault in any sample, it is what makes a statistic a random variable, and its size is measured by the standard deviation of that statistic. For a sample proportion from a random sample, that standard deviation is $\sqrt{\frac{p(1-p)}{n}}$.

Put numbers on it rather than words. With a population proportion of $p = 0.40$ and samples of $n = 50$, the standard deviation is $\sqrt{(0.40)(0.60)/50} = 0.069$. Five simulated random samples returned 18, 22, 17, 25 and 19 successes, so $\hat{p}$ (p-hat) came out 0.36, 0.44, 0.34, 0.50 and 0.38. Nothing is wrong with any of them. The binomial puts 94 percent of samples of 50 within two standard deviations of 0.40, a window running from 0.26 to 0.54, so at $n = 50$ an honest poll can land anywhere across a 28 point span.

The sentence to stop writing is "the two samples disagree, so one of them was done wrong," along with its cousin, "we got 0.34 instead of 0.40, so the method is biased." Neither follows. Bias is where the sampling distribution is centered, judged across all possible samples; sampling variability is how wide that distribution is. A perfectly unbiased method still hands you 0.34 on a Tuesday, and one sample can never diagnose bias, because a center is not visible from a single point.

It shrinks with $n$, slowly, since $n$ sits under a square root, and the variability of an estimator works through what that costs. It does not shrink because the population is large, or because you were careful, and only a census removes it. It also means nothing for a nonrandom sample: with no random mechanism there is no sampling distribution, and the formula above describes nothing.

Topic 2.12 turns the phenomenon into an object, the sampling distribution.