Sampling Variability vs Sampling Error

Draw a random sample, compute a statistic, put it back and draw again: the number changes. That change is sampling variability. It is a property of the procedure rather than a fault in any sample, it is what makes a statistic a random variable, and its size is measured by the standard deviation of that statistic. For a sample proportion from a random sample, that standard deviation is $\sqrt{\frac{p(1-p)}{n}}$.

Put numbers on it rather than words. With a population proportion of $p = 0.40$ and samples of $n = 50$, the standard deviation is $\sqrt{(0.40)(0.60)/50} = 0.069$. Five simulated random samples returned 18, 22, 17, 25 and 19 successes, so $\hat{p}$ (p-hat) came out 0.36, 0.44, 0.34, 0.50 and 0.38. Nothing is wrong with any of them. The binomial puts 94 percent of samples of 50 within two standard deviations of 0.40, a window running from 0.26 to 0.54, so at $n = 50$ an honest poll can land anywhere across a 28 point span.

The sentence to stop writing is "the two samples disagree, so one of them was done wrong," along with its cousin, "we got 0.34 instead of 0.40, so the method is biased." Neither follows. Bias is where the sampling distribution is centered, judged across all possible samples; sampling variability is how wide that distribution is. A perfectly unbiased method still hands you 0.34 on a Tuesday, and one sample can never diagnose bias, because a center is not visible from a single point.

It shrinks with $n$, slowly, since $n$ sits under a square root, and the variability of an estimator works through what that costs. It does not shrink because the population is large, or because you were careful, and only a census removes it. It also means nothing for a nonrandom sample: with no random mechanism there is no sampling distribution, and the formula above describes nothing.

Topic 2.12 turns the phenomenon into an object, the sampling distribution.

Sampling error is the gap between a statistic and the parameter it estimates, for one particular sample: $\hat{p} - p$ (p-hat minus p) for a proportion, or $\bar{x} - \mu$ (x-bar minus mu) for a mean. The word error is used in its statistical sense of distance from a target, not in the everyday sense of a blunder. It exists because a sample is only part of a population, so a flawless simple random sample still lands somewhere other than the truth, and it can come out positive or negative.

Suppose the true population proportion is $p = 0.50$ and one random sample of 100 gives $\hat{p} = 0.46$. The sampling error is $0.46 - 0.50 = -0.04$. Judge that against the spread it came from: the standard deviation of $\hat{p}$ here is $\sqrt{0.5(0.5)/100} = 0.05$, so the sample missed by 0.8 standard deviations. Counting exactly, about 48 percent of all samples of 100 miss by at least 0.04. Nothing went wrong in this sample; roughly half of them do this.

The misreading is right there in the name: "the poll had a sampling error, so the pollsters made a mistake." No. The companion version is "we eliminated sampling error by being careful." Care does not touch it. Only a larger $n$ shrinks it, and only a census removes it, which is why every honest poll publishes a number quantifying how big its sampling error is likely to be.

In practice you can never compute it, because computing $\hat{p} - p$ would require knowing $p$, and if you knew $p$ you would not be sampling. What gets reported instead is a margin of error, which bounds the likely size of the sampling error at a stated confidence level.

One boundary. Sampling error averages out to zero across repeated samples when the estimator is unbiased, so it is variation rather than a lean. Bias does not average out, and neither do non-sampling errors such as undercoverage, nonresponse, or a leading question. A bigger sample shrinks the first and leaves the others exactly where they were.