What is selection bias? Direction, not just the label

Q: Does a larger sample reduce selection bias?

No. Increasing n shrinks random sampling error but leaves the systematic tilt exactly where it was. In the landline example, the bias stays at 7.71 percentage points while the 95% margin of error falls from 9.74 points at n = 100 to 0.97 points at n = 10,000, producing a precise wrong answer.

By Jude Wallis · Published July 31, 2026

Selection bias is bias built into the sampling method: the way the sample was chosen gives some individuals a better chance of being in it, so the sample is systematically unlike the population. Name the direction the estimate is pushed, not just the label, and a bigger sample cannot fix it.

AP Statistics: Unit 1 (topics 1.11 Random Sampling, 1.12 Potential Problems with Sampling). Sampling problems are Unit 1 topic 1.12 of the Fall 2026 AP Statistics course, sitting right after topic 1.11 on random sampling methods. Unit 1 is 20% to 30% of the multiple-choice section, and questions here ask you to identify the source of a bias and describe how it affects the estimate.

Selection bias in one sentence

Selection bias happens when the method used to choose the sample makes some kinds of individuals more likely to be included than others, in a way that is related to what you are measuring.

That last clause is the one people skip. A sampling method that overrepresents left-handed people is harmless if you are estimating average income, because handedness and income are unrelated. The same method is ruinous if you are estimating what fraction of people are left-handed. Bias is a relationship between the selection mechanism and the variable, not a property of the mechanism alone.

The damage is systematic, which is what separates bias from ordinary sampling variability. Sampling variability is random: draw another sample and the estimate bounces the other way, and averaging many samples lands on the truth. Bias leans the same direction every time you repeat the method, so the estimate is centered on the wrong value and the average of a thousand samples still misses.

That is a claim about the long run, not about any one sample. Individual estimates still scatter around that wrong center, so a single sample can land on the correct side of the truth by luck. In the landline poll worked below, the method is aimed at 55.71% against a true 48%, and yet at $n = 100$ the standard deviation of the estimate is about 5 percentage points, so roughly 6% of samples still come out below 48%. One sample landing near the truth is not evidence that the method is sound, and one sample landing far from it is not proof of bias. What never averages away is the lean.

That is why the honest description of selection bias always includes a direction: too high or too low, and for what reason.

Selection bias vs nonresponse bias vs response bias

Three kinds of bias get confused because they all end with a wrong number. They differ in where the lean enters, and the fastest way to keep them straight is to follow one person through the process.

Type	Where it enters	The person in question	Example
Selection bias	Choosing who to ask	Never had a fair chance of being chosen	Only shoppers in one store are surveyed
Nonresponse bias	Getting an answer	Was chosen, but never answered	The mailed survey is thrown away
Response bias	The answer itself	Answered, but not truthfully or not accurately	Understating alcohol use to an interviewer

Selection bias is decided before anyone is contacted. If your sampling frame is a list of landline numbers, everyone without a landline is already out, whatever they would have said. Undercoverage, convenience sampling, and voluntary response are all ways selection bias gets in.

Nonresponse happens after selection. The person was picked, and the method was sound, but no answer came back. It hurts for the same reason: the people who do not answer usually differ from those who do.

Response bias happens after that. The person answered, and the answer leans away from the truth because of question wording, an interviewer effect, or a socially awkward topic.

How to tell which type of bias a survey has turns this into a four-question decision flow, which is the version to use under exam pressure.

Five methods, and the direction each one tilts

The label alone earns little. Name who is overrepresented, then say which way that pushes the number.

Voluntary response. A radio station asks listeners to call in about a proposed tax. People with strong feelings, usually angry ones, are far likelier to call, so the estimate of opposition comes out too high.
Convenience sample. A researcher surveys shoppers at a mall on a Tuesday morning to estimate average household income. That overrepresents retirees and people not working weekday hours, so the estimated income comes out too low.
Undercoverage. A poll dials landline numbers only. Younger adults are far less likely to have a landline, so if support for a measure is lower among young adults, the estimate comes out too high.
Frame mismatch. A study of "all adults in the county" samples from a registered-voter list. Adults who are not registered are excluded entirely, and they differ from registered voters in age, mobility, and civic engagement, so any estimate correlated with those traits leans toward the registered group.
Frequency-weighted intercepts. A gym surveys members as they walk in the door to estimate average visits per week. Frequent visitors have far more chances to be intercepted, so the estimate of average visits comes out much too high. The first worked example below puts a number on it: 3.29 visits against a true 2.1.

A useful sentence pattern: "This method overrepresents (group), who tend to (differ in this way), so the estimate will be (too high or too low)". Both halves are needed. "The sample is biased" is not an answer.

Why a bigger sample does not help

Increasing $n$ shrinks the random part of the error and leaves the systematic part untouched. Take the landline poll from the second worked example, where the true support is 48% and the biased method is aimed at 55.71%. Watch what more data buys:

Sample size	Margin of error at 95%	Bias
100	9.74 points	7.71 points
400	4.87 points	7.71 points
1600	2.43 points	7.71 points
10000	0.97 points	7.71 points

At $n = 100$ the interval is wide enough to cover the truth by accident, which is the sample-to-sample scatter described in the first section rather than any virtue of the method. At $n = 10000$ it is a tight interval centered on the wrong number, confidently excluding the truth. More data has made a wrong answer more precise, which is worse than an honestly vague one.

This is the single most useful thing to know about bias, and does a bigger sample fix bias is the guide that drills it. The only fixes are changes to the method: a proper sampling frame, random selection from it, and effort spent on follow-up.

Naming the direction, in context

Free-response questions about bias almost always want three things, and dropping any one of them costs credit.

The mechanism. Who was more or less likely to be selected, and why.
The relevant difference. How that group differs on the variable being measured.
The direction. Whether the estimate will be too high or too low, stated about the actual parameter in context.

A weak answer: "This is a biased sample." A strong one: "Surveying members as they arrive overrepresents people who come often, since a member who visits six times a week has six chances to be stopped and a member who visits once has one. Those frequent visitors have higher visit counts by definition, so the sample mean will overestimate the mean weekly visits of all members."

The direction is also what tells you whether the bias matters. A tilt that pushes an estimate away from a decision threshold is a nuisance. A tilt that pushes it across the threshold changes the conclusion.

One further consequence worth carrying into later units: everything you fit to a biased sample inherits the tilt. A least-squares line fitted to volunteers describes volunteers, and both its slope and its y-intercept belong to that group rather than to the population you meant to study. Can you generalize results works through when the leap to the population is allowed.

How to reduce it, and mistakes to avoid

Selection bias is prevented at the design stage, not repaired at the analysis stage.

Build a sampling frame that actually covers the population, then select randomly from it. A simple random sample, in which every possible group of $n$ individuals is equally likely, is what removes the mechanism. Equal chances for individuals is not enough on its own, since a systematic sample gives everyone the same chance while allowing only a handful of the possible groups.
Use stratified sampling when you know the population has groups that differ, so each group is represented in the right proportion by design.
Chase nonresponse with follow-up contacts rather than replacing nonrespondents with whoever is available, which converts a nonresponse problem into a convenience-sample problem.

The mistakes that cost the most:

Naming the type without the direction. "Undercoverage" alone is half an answer.
Calling any unrepresentative sample selection bias when the individuals were chosen properly and simply did not answer. That is nonresponse.
Treating a large sample as evidence of quality. Size and bias are unrelated.
Assuming random assignment fixes it. Random assignment inside an experiment supports causal conclusions about the subjects you have; it does nothing about how those subjects were recruited.
Saying the sample "is not random" and stopping there. The grader wants the mechanism and the direction.

The gym survey that overstates the mean by 56.5%

A gym has 1000 members: 600 visit once a week, 300 visit three times a week, and 100 visit six times a week. Management wants the mean number of visits per week. A staff member surveys people as they walk in the door during one week and averages their answers. What does the survey estimate, and how does it compare to the truth?

Compute the true parameter. Total weekly visits $= 600(1) + 300(3) + 100(6) = 600 + 900 + 600 = 2100$ visits across 1000 members, so the mean is $\frac{2100}{1000} = 2.1$ visits per week.
Identify the selection mechanism. The survey samples visits, not members. A member who comes six times has six chances to be stopped in that week; a member who comes once has one chance.
Turn that into selection probabilities. Of the 2100 visits in the week, $\frac{600}{2100} \approx 0.2857$ belong to once-a-week members, $\frac{900}{2100} \approx 0.4286$ to three-times members, and $\frac{600}{2100} \approx 0.2857$ to six-times members.
Notice the distortion. Once-a-week members are 60% of the membership but only about 28.6% of the interviews you collect. Six-times members are 10% of the membership but also about 28.6% of the interviews you collect.
Compute the expected sample mean: $0.2857(1) + 0.4286(3) + 0.2857(6)$ , which is $\frac{600(1) + 900(3) + 600(6)}{2100} = \frac{600 + 2700 + 3600}{2100} = \frac{6900}{2100} \approx 3.2857$ .
Compare: 3.29 visits estimated against 2.1 visits true, an overstatement of about 1.19 visits, or 56.5% too high.
State the direction with its reason: the door survey overrepresents frequent visitors, who by definition have high visit counts, so it overestimates the mean weekly visits of all members.
Name the fix: draw a simple random sample from the membership list and contact those members, so how often a member visits no longer affects the chance of being selected.

The true mean is 2.1 visits per week; the door survey is aimed at about 3.29 visits, roughly 56.5% too high. This is selection bias, because the method samples visits rather than members and frequent visitors get more chances to be selected. Sampling randomly from the membership list removes it.

Undercoverage with a known direction

A county has 10,000 adults: 4,000 under 40, of whom 30% support a bond measure, and 6,000 aged 40 or over, of whom 60% support it. A polling firm dials landline numbers. It can reach 20% of the under-40 adults and 80% of the 40-and-over adults. Assuming everyone reached answers honestly, what will the poll estimate, and in which direction is it wrong?

Compute the true parameter. Supporters $= 4000(0.30) + 6000(0.60) = 1200 + 3600 = 4800$ out of 10,000, so true support is $\frac{4800}{10000} = 0.48$ , or 48%.
Find who is actually reachable. Under 40: $4000(0.20) = 800$ adults. Aged 40 and over: $6000(0.80) = 4800$ adults. The reachable frame holds $800 + 4800 = 5600$ adults.
Compare the frame to the population. Under-40 adults are 40% of the county but only $\frac{800}{5600} \approx 14.3\%$ of the frame. The frame is far older than the population.
Compute what a perfect random sample of that frame estimates: supporters in the frame $= 800(0.30) + 4800(0.60) = 240 + 2880 = 3120$ .
So the expected estimate is $\frac{3120}{5600} \approx 0.5571$ , or 55.71%.
Compare with the truth: $55.71\% - 48\% = 7.71$ percentage points too high.
State the direction with its reason: the landline frame undercovers adults under 40, who support the measure at a much lower rate (30% against 60%), so the poll overestimates support.
Confirm that sample size will not help. At $n = 1600$ from this frame the 95% margin of error is about 2.43 percentage points, far smaller than the 7.71-point bias, so the interval would sit entirely above the true 48%.

The poll is aimed at about 55.71% support against a true 48%, overstating by 7.71 percentage points. The cause is undercoverage, a form of selection bias: the landline frame reaches only 20% of under-40 adults, who support the measure at half the rate of older adults. A larger sample narrows the interval around the wrong value.

Frequently asked questions

What is the difference between selection bias and nonresponse bias?

Selection bias enters when the sample is chosen: some individuals never had a fair chance of being included, as when a landline-only frame leaves out people without landlines. Nonresponse bias enters afterward: the person was properly selected but never answered. Both distort the result, and only the second one can be attacked with follow-up contacts.

How do you describe selection bias for full credit?

Give three things: the mechanism (who was more likely to be selected and why), the relevant difference (how that group differs on the variable), and the direction (whether the estimate is too high or too low, in context). "The sample is biased" earns nothing without the direction.

Does a larger sample reduce selection bias?

No. Increasing $n$ shrinks random sampling error but leaves the systematic tilt exactly where it was. In the landline example, the bias stays at 7.71 percentage points while the 95% margin of error falls from 9.74 points at $n = 100$ to 0.97 points at $n = 10{,}000$ , producing a precise wrong answer.

Is a voluntary response sample an example of selection bias?

Yes. The individuals select themselves, so people with strong opinions are overrepresented and the method never gave everyone else a fair chance. Convenience samples and undercoverage are the other common ways selection bias gets in, and AP questions usually want the specific name rather than the umbrella term.

Can selection bias be fixed after the data are collected?

Not reliably. Weighting can partly correct for groups you know are underrepresented, but it can do nothing about people the frame never reached and it depends on knowing the population composition. The dependable fixes are design choices: a frame that covers the population, random selection from it, and follow-up on nonrespondents.