How to tell which type of bias a survey has

Ask where the lean entered

Every survey moves through four stages, and each named bias enters at exactly one of them. The population is who you want to describe. The list is who your method could actually reach. The sample is who got selected. The data is what those people said.

• Undercoverage bias enters at the list: part of the population was left off it, or made less likely to be picked.
• Voluntary response bias enters at selection: the people in the sample chose themselves by answering an open invitation.
• Nonresponse bias enters after selection: the right people were chosen, and some of them never answered.
• Response bias enters at the data: the right people answered, but their answers lean away from the truth.

In all four cases bias means the same thing. It is a systematic error built into the method that pushes a statistic consistently above or consistently below the parameter it estimates, so the sampling distribution of a sample proportion $\hat{p}$ is centered away from the population proportion $p$. Single samples still bounce around by chance, and some of them land on the far side of $p$, but the center of that bouncing sits on the wrong side and repeating the method never moves it back. Bias belongs to the method, not to one unlucky sample. That is why a bigger sample never fixes it: you get a more precise version of the same wrong answer.

The decision flow: four questions in order

Read the scenario once, then walk these four questions in order. The first yes names the bias you lead with, but keep going after it, because a scenario can trip more than one and the exam wants each one named.

1. Did the people in the sample put themselves there? An invitation to call in, click a link, mail back a form, or post a review means the sample is made of volunteers. That is voluntary response bias.
2. Was some part of the population unable to be selected? If the frame the researcher drew from left people out, or gave them a smaller chance, that is undercoverage.
3. Were people selected properly and then never heard from? Refusals, no one home, blank questions. That is nonresponse.
4. Did the chosen people answer, but with answers you would not trust? Leading wording, an interviewer standing there, an embarrassing question. That is response bias.

Work the questions in that order because self-selection is the loudest signal in the wording and settles a scenario fastest, while response bias is the one you can only see once you read the question the subjects were asked. The four stages above tell you where each bias enters; this list tells you which to check first. When two of them apply, name both and say where each one entered. Here is what each one looks like in the wording of a problem.

Undercoverage vs nonresponse: the pair that costs points

This is the confusion that shows up most, and one question settles it: could that person have been selected at all?

If the answer is no, it is undercoverage. Those people were never in the running. A phone poll drawn from landline numbers cannot reach cell-only households. An online panel cannot reach people with no internet. A roster built on Monday misses students who enrolled on Tuesday.

If the answer is yes, and the person was selected but never gave you data, it is nonresponse. They refused, they were not home, they ignored the email, they skipped that one item on the form.

Undercoverage happens before anyone is contacted. Nonresponse happens after. That is the whole distinction.

The trap is that both leave you with missing people, so "only 34 of the 100 selected students replied" can read like a coverage problem. It is not: all 100 were selected, so the 66 silent students are nonresponse. And notice the second half of the definition. A low response rate by itself is not automatically bias. It becomes nonresponse bias when the people who did not reply differ from the people who did in a way that matters for the question being asked.

Voluntary response vs convenience: who did the choosing

Both are nonrandom ways to get a sample, and the difference is who made the choice.

In a voluntary response sample the researcher opens a door and waits. Radio call-in polls, online opinion polls, comment cards, product reviews. People select themselves, and people with strong feelings select themselves far more often than people who are indifferent.

In a convenience sample the researcher picks whoever is easiest to reach. The first 30 students in the cafeteria, shoppers at one mall entrance, the teacher's own class. Nobody volunteered; the researcher chose them for convenience.

On the exam, convenience sampling is usually named as the method, and the bias you then describe works like undercoverage: every student who was not in that cafeteria had no chance of being selected. So name the method, then say who was shut out and how those people differ. Both methods fail for the same underlying reason, which is that chance did not decide who got in. That is exactly what random selection is for, and the four random methods are compared in simple random vs stratified sampling.

Name the direction, in context, or you lose the point

Naming the type is half the answer. The course asks you to describe how the bias affects the result, and graders want a direction attached to the actual variable in the study. Use this shape:

Because [the affected group] tends to [differ in this specific way], this method will [overstate or understate] [the parameter, in context].

How to find the direction for each type:

• Voluntary response. People who answer an open call feel strongly, so the estimate overstates how common that strong view is. A restaurant's online review score is usually pulled by diners who loved it or hated it, not by the quiet middle.
• Undercoverage. The direction comes from how the missing group differs. If the excluded people are less likely to have the trait, the estimate comes out too high; if they are more likely, too low.
• Nonresponse. Same logic, one stage later. Ask how non-responders differ from responders, then push the estimate toward the responders.
• Response bias. Embarrassing behavior gets under-reported and admirable behavior gets over-reported, so self-reported cheating runs low and self-reported exercise runs high. A leading question pushes answers toward whatever it hints at.

Two sentences that earn nothing: "the results will be biased" and "the sample was too small." The first has no direction and no context; the second is not bias at all. If you truly cannot pin the direction, still say who is missing and how the estimate would move if that group differs, which is the reasoning graders are looking for.

What is not bias

Three things get called bias in student answers and are not.

• Sampling variability. A perfectly good random method gives a different statistic every time you run it. That spread is chance, not a lean. Bias is the method missing the parameter in the same direction over and over.
• A small sample. Small samples are more variable, not more biased. Raising $n$ narrows the spread of the sampling distribution and does nothing at all to a systematic lean, which is worked through in does a bigger sample fix bias.
• One odd-looking sample. You cannot diagnose bias from a single result. You diagnose it from the procedure that produced it, which is why exam questions hand you the procedure.

One more boundary. This topic is about surveys and sampling. Confounding, control groups, and placebo effects belong to experimental design, so if the scenario assigns treatments rather than asking questions, you are in a different topic. See experiments vs observational studies for that line, and parameter vs statistic for the vocabulary that makes "the statistic misses the parameter" precise.