Bias vs variability: center vs spread

Bias vs variability: the short answer

Both words describe one object, the sampling distribution of a statistic: the collection of values that statistic would take across every sample the method could produce. They pick out different features of it.

• Bias is the gap between the center of that distribution and the parameter you are estimating. Zero bias means the method lands on the right number on average.
• Variability is the spread of that same distribution, measured by its standard deviation. Low variability means two different samples would hand you nearly the same estimate.

The consequence students miss is where the sample size $n$ shows up. For a sample mean or a sample proportion it sits in the spread and appears nowhere in the center. Quadruple the data and the spread halves, while the center does not move at all. So a slanted method run on 40,000 people is no closer to the truth than the same method on 400; it is the same wrong answer reported with a tighter interval around it, which is harder to argue with and no more correct.

Everything below puts numbers on that, and the separate question of what actually removes bias is answered in does a bigger sample fix bias.

Two questions about the same distribution

Say you want the population mean $\mu$ ("mu") and you estimate it with the sample mean $\bar{x}$ ("x-bar"). Imagine running your whole data-collection plan again and again and writing down every $\bar{x}$ it produces. Those values pile up into a distribution, and two numbers summarize it.

Where is it centered? That is bias:

A positive bias means the method runs high every time you use it, a negative one means it runs low, and zero means it is unbiased.

How wide is it? That is variability, the standard deviation of those same $\bar{x}$ values. For a random sample from a population with standard deviation $\sigma$ ("sigma") it is $\frac{\sigma}{\sqrt{n}}$, provided the sample is under 10% of the population.

The two are separate properties, not two ends of one scale, so a method can be good at either one and bad at the other. Bias comes from who ends up in the data and how they are measured. Variability comes from chance and shrinks when you buy more data. A common shorthand calls the first accuracy and the second precision, and a target with four shot groups is the usual picture, but a picture will not tell you which sampling plan to run. Four plans will, so the next section uses four.

One footnote, so the vocabulary does not trip you later. Statisticians also apply the word bias to a property of an estimator's formula, and a few of those do fade as $n$ grows: the sample maximum, for instance, cannot exceed the population maximum and so runs low by less and less as the sample gets bigger. See bias of an estimator for that sense. Bias built into how individuals are selected or measured is the kind on this page, and it does not fade at all.

The four quadrants, as four real surveys

A district has 20,000 students and you want their mean nightly sleep. Across the district, sleep is roughly normal with standard deviation 1.2 hours. Of those students, 5,000 play a school sport and average 8.0 hours, with the same standard deviation of 1.2 hours inside that group, and the other 15,000 average 6.8 hours with a slightly tighter spread, which is what keeps the district standard deviation at 1.2. The parameter is the district mean, $\mu = 0.25(8.0) + 0.75(6.8) = 7.1$ hours, and nobody running the survey knows it.

Two choices decide which quadrant your survey lands in: the list you draw names from, and how many names you draw.

Read the table by column, not by row. The roster sets the center: draw at random from the whole district and $\bar{x}$ centers on 7.1, draw at random from the athletics roster and it centers on 8.0 no matter how many you draw, a bias of $+0.9$ hours, or 54 minutes. The sample size sets the width: both lists have a standard deviation of 1.2 hours, so 400 gives $\frac{1.2}{\sqrt{400}} = 0.06$ hours, about 3.6 minutes, and 9 gives $\frac{1.2}{3} = 0.40$ hours, or 24 minutes. Each sample is well under 10% of the list it came from, since 400 is 2% of the district and 8% of the 5,000 athletes, so those formulas apply.

Plan D is the one plan with nothing going for it, drawn from the wrong list and imprecise on top of that, so it needs no argument. Plan C is the dangerous one. It is large, it is drawn at random, its estimates repeat to within a few minutes of each other, and it is 54 minutes off. Every quality signal a reader can see is green.

The differences side by side

The last two rows are the ones worth memorizing, and the section after next is about them.

What a bigger sample actually changes

Take the biased mechanism behind plans C and D, drawing at random from a group whose mean is 8.0 hours with a standard deviation of 1.2, and simulate 200,000 samples at each of the two sizes. Across the 200,000 sample means at $n = 9$, the average was 7.9995 and the standard deviation 0.4008. At $n = 400$, the average was 8.0001 and the standard deviation 0.0598. About 44 times the data cut the spread by a factor of 6.7, which is $\sqrt{44.4}$, and shifted the center by 0.0006 hours, simulation noise rather than progress.

Now score those same samples on the only thing that matters, landing within 15 minutes of the true 7.1 hours. At $n = 9$, 10,042 of the 200,000 samples managed it, about 5%, because the distribution is wide enough that a lucky low sample can reach the truth. At $n = 400$, none of the 200,000 did, and the closest any of them came was 0.624 hours, or 37 minutes. More data made this survey strictly less likely to produce a usable answer while making its output look far more authoritative.

That is the whole asymmetry. Variability is a resource problem, and money fixes it. Bias is a design problem, and no amount of data touches it. Watching the spread close in while the center stays put is easier to believe once you have moved the slider yourself in the sampling distribution and CLT interactive.

The classic mix-up and how to avoid it

Two errors, running in opposite directions.

Calling one bad estimate biased. A sample mean of 6.7 when the truth is 7.1 is not evidence of bias. Plan B is unbiased and misses by that much routinely, since 0.4 hours is exactly one standard deviation for it, and an unbiased method scatters around the right answer rather than hitting it. Bias is a statement about the center of all possible samples, so one sample cannot show it: a center is not visible from a single point. Sampling variability alone accounts for what you saw, so one miss is no evidence either way.

Treating a narrow interval as a correct one. A margin of error is computed from the standard error, which measures variability alone, so it carries no allowance for a slanted method. A plan C sample landing exactly on its center, $\bar{x} = 8.0$, gives a 95% interval of $8.0 \pm 1.96(0.06)$, which is 7.88 to 8.12 hours. Correct arithmetic, wrong conclusion, and 7.1 is not in it. Widening does not rescue you either: a plan D sample landing on the same center gives $8.0 \pm 1.96(0.40)$, or 7.22 to 8.78, which still misses.

The habit that keeps them straight is to ask the two questions in the right order and about the right thing. Variability you can estimate from your data with the standard error. Bias you cannot, at any sample size, so you audit the procedure instead: who could not be selected, who did not answer, and what the question or the instrument might have pushed people to say. How to identify the type of bias walks through naming it, and the topic list names both halves of this page, with Unit 3 topic 3.1 titled Estimators and Unit 1 topic 1.12 titled Potential Problems with Sampling.