Identifying bias worksheet with answer key

8 problems on identifying bias, ready to print for class or homework. The answer key prints on its own page, so hand out everything before it.

Problem 1

For each study, name the type of bias and say whether the estimate is pushed too high or too low.

(a) A bakery wants the proportion of all its customers who would buy a new sourdough loaf. It leaves a stack of comment cards by the register inviting customers to write in if they would buy the new loaf, and counts the cards dropped in the box.

(b) A library wants the mean number of books its 8,000 cardholders read last year. It draws a random sample of 300 names from the list of cardholders who have visited in the past 30 days.

(c) A university mails a survey to a simple random sample of 800 alumni asking whether they gave money to any charity last year. 210 surveys come back.

(d) A school nurse asks each of 60 randomly selected students, face to face in her office, how many energy drinks they had last week.

(e) A town puts this item on a mailed ballot: "Should the town waste 4 million dollars on a new roundabout?"

Problem 2

A gym chain wants the mean number of days per week its 3,000 members visit. A staffer stands at the door one Tuesday evening and surveys every 5th person who walks in, until 60 members have answered.

Suppose the truth is that 900 of the 3,000 members come 4 days a week and the other 2,100 come 1 day a week, and that each member's visits are spread evenly across the week.

(a) Name the bias in this method and explain the mechanism. (b) Find the true mean number of visits per week. (c) Of the visits made in a week, what fraction are made by the frequent members? Use that to find the mean weekly visits of the member the staffer actually intercepts. (d) State the size and direction of the bias.

Problem 3

An online retailer emails a one-question satisfaction survey to all 40,000 of its customers. 5,000 answer, and 4,300 of them say they are satisfied. Customers who were disappointed had mostly stopped opening the retailer's email months earlier.

(a) Find p^\hat{p} and the response rate. (b) Name the bias, the mechanism, and the direction. (c) A later audit that tracks down a random subsample of the non-answerers puts true satisfaction across all 40,000 customers at 0.71. How large is the bias? (d) A manager says the fix is to email 120,000 customers next quarter. If the response rate holds at the same level, find the standard deviation of p^\hat{p} before and after, and compare each one to the bias. (e) Answer the manager.

Problem 4

A teacher knows that exactly 0.24 of her school's 5,000 students bike to school. Her class simulates two survey methods, running each one 200 times and recording the 200 sample proportions.

Method A: a simple random sample of 50 students from the full enrollment roster. The 200 results have mean 0.241 and standard deviation 0.060.

Method B: a random sample of 50 students drawn from the roster of the 2,400 students who live within one mile of the school. The 200 results have mean 0.385 and standard deviation 0.069.

(a) Which method is biased, and which two numbers show it? (b) Estimate the size of the bias and explain the mechanism and direction. (c) Method A missed 0.24 by 0.001. Is that evidence of bias? Support the answer with a calculation. (d) Both classes now repeat the simulation with samples of 200 instead of 50. Predict all four summary numbers. (e) State the difference between bias and sampling variability in one sentence.

Problem 5

A school board wants the proportion of parents who support moving the high school start time from 7:20 a.m. to 8:15 a.m. A member drafts this question: "Given that teenagers who sleep longer perform better in school, do you support moving the high school start time to 8:15 a.m.?"

(a) Name the bias this wording creates and predict the direction. (b) Rewrite the question neutrally. (c) The board tests both wordings. It takes a simple random sample of 400 parents, splits them at random into two groups of 200, and reaches every parent by phone, with the caller introducing herself as calling on behalf of the school board and asking one group the draft question and the other group the rewrite. The draft gets 148 yes and the rewrite gets 106 yes. Find both proportions and the gap. (d) Why does splitting the sample at random matter for what part (c) proves? (e) The rewrite's result still is not the parameter. Name one bias that survives the fix and give its direction.

Problem 6

A university health center mails a survey to a simple random sample of 500 students asking whether they got a flu shot this fall. 180 students answer and 126 of them say yes.

(a) Find the response rate and the proportion who said yes. (b) Name the bias, the mechanism, and the direction. (c) Without any assumption about the non-answerers, find the smallest and largest values the proportion vaccinated among the 500 selected students could take. (d) The health center reports the estimate with the usual 95% margin of error computed from its 180 answers, using the critical value 1.96. Find it and compare it to the range in part (c). (e) What should the health center do?

Problem 7

A city council wants the proportion of residents who would use a proposed weekend shuttle to the beach. Staff set up a table at the downtown farmers market on three Saturday mornings and ask passersby, "Would you use a free weekend shuttle to the beach instead of fighting for parking?" 600 people stop and answer, and 462 say yes.

(a) This study has two distinct problems. Name both types and give the mechanism for each. (b) Give the direction each problem pushes the estimate. (c) The council later runs a proper survey: a simple random sample of 500 residents contacted by phone and mail with follow-ups, reaching 430 of them, of whom 129 say yes to a neutral version of the question. Find both proportions and the gap. (d) A staffer says the market result would have been trustworthy with 6,000 responses instead of 600. Compute what that change would actually do, and answer him.

Problem 8

A fitness app publishes the claim "72% of Americans exercise at least three times a week." The evidence is an in-app pop-up asking users to report their weekly workouts. The app has 320,000 users, and 38,400 of them answered.

(a) Name the population the claim is about and the population this method can describe. (b) Identify three distinct sources of bias, with a mechanism and a direction for each. (c) Find the response rate. (d) The app's data scientist says the margin of error is under half a percentage point, so the estimate is reliable. Check the arithmetic with the critical value 1.96, then say precisely what that number does and does not cover. (e) Write one sentence the app could defensibly publish instead.

Identifying bias: answer key

  1. (a) Voluntary response, too high: only customers who care about the bakery's bread fill out a card. (b) Undercoverage of inactive cardholders, too high: the missing people are the light readers. (c) Nonresponse, too high: donors are likelier to return a survey about giving. (d) Response bias from face-to-face self-report, too low: students underreport a disapproved behavior. (e) Response bias from question wording, too low: "waste" pushes voters away from the roundabout.

  2. (a) Undercoverage, in a convenience sample: a member who visits 4 days a week is 4 times as likely to be at the door as a member who visits once, so selection probability rises with the variable being measured. (b) 57003000=1.9\frac{5700}{3000} = 1.9 days. (c) 36005700≈0.6316\frac{3600}{5700} \approx 0.6316 of visits are made by frequent members, giving an intercepted mean of 165005700≈2.89\frac{16500}{5700} \approx 2.89 days. (d) Bias ≈2.89−1.9=0.99\approx 2.89 - 1.9 = 0.99 days too high, because heavy users are overrepresented at the door.

  3. (a) p^=43005000=0.86\hat{p} = \frac{4300}{5000} = 0.86, response rate 500040000=0.125\frac{5000}{40000} = 0.125. (b) Nonresponse bias: unhappy customers had already stopped opening the retailer's email, so the answerers lean satisfied and 0.86 is too high. (c) Bias =0.86−0.71=0.15= 0.86 - 0.71 = 0.15, 15 percentage points too high. (d) The standard deviation falls from about 0.0049 to about 0.0028, roughly 0.2 percentage points, while the bias stays at 15 points, moving from about 31 to about 53 standard deviations. (e) Emailing more people cannot move the center; chase the non-answerers with a random follow-up subsample instead.

  4. (a) Method B, because its 200 results center at 0.385 while p=0.24p = 0.24; Method A centers at 0.241. (b) Bias ≈0.385−0.240=0.145\approx 0.385 - 0.240 = 0.145 too high, from undercoverage: students living more than a mile away were left off the list and they almost never bike. (c) No. The mean of 200 results has standard deviation 0.060200≈0.0042\frac{0.060}{\sqrt{200}} \approx 0.0042, so a 0.001 miss is about a quarter of one standard deviation, while Method B misses by about 30. (d) A: mean about 0.240, standard deviation about 0.030. B: mean about 0.385, standard deviation about 0.0345. (e) Bias is how far the method's center sits from the parameter; sampling variability is the spread around that center, and only the spread responds to nn.

  5. (a) Response bias from question wording, overstating support, because the sleep claim supplies a one-sided reason to say yes. (b) "Do you support, oppose, or have no opinion about moving the high school start time from 7:20 a.m. to 8:15 a.m.?" (c) 148200=0.74\frac{148}{200} = 0.74 and 106200=0.53\frac{106}{200} = 0.53, a gap of 21 percentage points. (d) The two groups were formed by chance from one random sample, so wording is the only systematic difference and the gap can be attributed to it. (e) Response bias in a second form: deference to a caller who says she represents the board, which pushes the reported support up.

  6. (a) Response rate 180500=0.36\frac{180}{500} = 0.36; p^=126180=0.70\hat{p} = \frac{126}{180} = 0.70. (b) Nonresponse bias: vaccinated students have the easy, approved answer to send back to a health center, so 0.70 is too high. (c) From 126500=0.252\frac{126}{500} = 0.252 to 446500=0.892\frac{446}{500} = 0.892, a band 0.640 wide. (d) 1.960.21180≈0.06691.96\sqrt{\frac{0.21}{180}} \approx 0.0669, about 6.7 percentage points, so a 13.4 point interval inside a 64 point band; the margin of error covers sampling variability only. (e) Chase a random subsample of the 320 non-answerers to near-complete response, rather than mailing more surveys.

  7. (a) Voluntary response bias, since passersby chose themselves at a market that only some residents visit, and response bias from question wording, since "free" and "instead of fighting for parking" sell the shuttle inside the question. (b) Both overstate support, and they compound. (c) 462600=0.77\frac{462}{600} = 0.77 against 129430=0.30\frac{129}{430} = 0.30, a gap of 47 percentage points. (d) The standard deviation of p^\hat{p} would fall from about 0.0172 to about 0.0054, roughly 1.2 percentage points, while the 47 point lean is untouched; a bigger market sample measures the wrong quantity more precisely.

  8. (a) The claim is about all Americans; the method describes only active users of this one app. (b) Undercoverage (non-users cannot be selected, and app users exercise more), voluntary response (only the 12% who chose to tap through answered, and the active users are the proud ones), and response bias (self-reported exercise gets rounded up). All three push 72% above the truth. (c) 38400320000=0.12\frac{38400}{320000} = 0.12. (d) 1.960.201638400≈0.00451.96\sqrt{\frac{0.2016}{38400}} \approx 0.0045, about 0.45 percentage points, and it is correct; it measures run-to-run variability only and contains no term for a broken frame. (e) "Among the 38,400 users who chose to answer an in-app survey, 72% reported exercising at least three times a week."

The key gives final answers only. The full worked solution for each problem is on the matching practice set, and every other sheet is on the worksheets page.

More worksheets from this unit