Sampling Error vs Bias of an Estimator

Sampling error is the gap between a statistic and the parameter it estimates, for one particular sample: $\hat{p} - p$ (p-hat minus p) for a proportion, or $\bar{x} - \mu$ (x-bar minus mu) for a mean. The word error is used in its statistical sense of distance from a target, not in the everyday sense of a blunder. It exists because a sample is only part of a population, so a flawless simple random sample still lands somewhere other than the truth, and it can come out positive or negative.

Suppose the true population proportion is $p = 0.50$ and one random sample of 100 gives $\hat{p} = 0.46$. The sampling error is $0.46 - 0.50 = -0.04$. Judge that against the spread it came from: the standard deviation of $\hat{p}$ here is $\sqrt{0.5(0.5)/100} = 0.05$, so the sample missed by 0.8 standard deviations. Counting exactly, about 48 percent of all samples of 100 miss by at least 0.04. Nothing went wrong in this sample; roughly half of them do this.

The misreading is right there in the name: "the poll had a sampling error, so the pollsters made a mistake." No. The companion version is "we eliminated sampling error by being careful." Care does not touch it. Only a larger $n$ shrinks it, and only a census removes it, which is why every honest poll publishes a number quantifying how big its sampling error is likely to be.

In practice you can never compute it, because computing $\hat{p} - p$ would require knowing $p$, and if you knew $p$ you would not be sampling. What gets reported instead is a margin of error, which bounds the likely size of the sampling error at a stated confidence level.

One boundary. Sampling error averages out to zero across repeated samples when the estimator is unbiased, so it is variation rather than a lean. Bias does not average out, and neither do non-sampling errors such as undercoverage, nonresponse, or a leading question. A bigger sample shrinks the first and leaves the others exactly where they were.

Bias is a number, not a verdict: subtract the parameter from the mean of the estimator's sampling distribution, $E(\hat{\theta}) - \theta$. Zero is what unbiased means. A negative value says the estimator lands below the parameter on average, a positive one says above. Because it is an average over all possible samples, bias is a property of the estimator and the sampling scheme, never of the single estimate in front of you.

Some estimators are biased for a reason you can see without arithmetic. A sample can never hold a value larger than the largest value in the population, so the sample maximum is either exactly right or too small and never too big; averaged over samples it has to fall short. Draw 10 values with replacement from the integers 1 through 100 and the sample maximum averages 91.40, a bias of $-8.60$ against a true maximum of 100.

The sentence to stop writing is this one: "my sample mean came out 3.2 when the true mean is 3.0, so my estimate is biased." A single estimate cannot be biased. Missing by 0.2 is sampling error, and unbiased estimators do it constantly, which is the whole point of having a sampling distribution. Bias only shows up across all possible samples, so you argue for it from the method rather than read it off your data.

Two different things get called bias in this course and they behave differently. The kind above is a mathematical property of a formula, and here it does shrink with $n$: at $n = 100$ the sample maximum averages 99.43. Bias built into how people were selected, such as undercoverage or nonresponse, does not shrink at all, and a larger sample only buys a more precise answer to the wrong question.

Topic 3.1, Estimators, pairs bias with variability. They are the two axes of the target diagram, and an estimator can be good on one and bad on the other.