Identifying bias practice problems (8 solved)

(a) Who put these people in the sample? The customers did, by choosing to fill out a card. That is voluntary response bias. Customers with a strong feeling about the bakery's bread are the ones who bother, and interest in a new loaf is exactly what is being measured, so the estimate of the proportion who would buy overstates the truth.

(b) Could every cardholder have been selected? No. The list is recent visitors only, so any cardholder who has not been in for a month had no chance of selection at all. That is undercoverage. The people left off the list are the light users, so the mean number of books read comes out too high.

(c) The 800 were selected properly by a random mechanism, and 590 of them never answered, so this is nonresponse bias, not undercoverage. Two things push the same way here: an alum who gives to charity has a creditable answer to write down, and an alum who gives to this university in particular is likelier to open its mail in the first place. Both raise the return rate among givers, so the estimated proportion who gave is too high.

(d) These 60 were randomly selected, and they all answered, so the selection is sound and the lean is in the answers themselves. That is response bias, from a self-reported behavior that adults disapprove of, asked face to face by a school employee. Students shade their counts downward, so the mean number of energy drinks is too low.

(e) Everyone on the mailing list could be selected and the answers come from the ballot itself, so the problem is again response bias, in its question wording form. The word "waste" supplies a verdict before the question is asked, and it sits on one side only. Support for the roundabout is understated.

Sort the five. Selection created the problem in (a) and (b), the gap between selection and data created it in (c), and the data itself carries it in (d) and (e). Three estimates are pushed up and two are pushed down.

(a) Name it. Taking every 5th arrival is a systematic selection, and that part is fine. The damage is in who can be at the door at all. A member who never comes on a Tuesday evening cannot be selected, and a member who comes 4 days a week is 4 times as likely to be there as a member who comes once. That is undercoverage: the method makes part of the population less likely to be selected, and it does so based on the very variable being measured. It is also a convenience sample, since the staffer surveys whoever happens to walk past.

(b) True mean. Add up all the weekly visits and divide by the number of members: $\bar{x} = \frac{900(4) + 2100(1)}{3000} = \frac{3600 + 2100}{3000} = \frac{5700}{3000} = 1.9$ days per week, where $\bar{x}$ (x-bar) is the mean visits per member.

(c) Who walks through the door. In one week the frequent members make $900 \times 4 = 3600$ visits and the occasional members make $2100 \times 1 = 2100$ visits, so $3600 + 2100 = 5700$ visits happen in total. Because visits are spread evenly across the week, treat the person the staffer intercepts as one randomly chosen visit. Then $\frac{3600}{5700} \approx 0.6316$ of the people at the door are frequent members, even though frequent members are only $\frac{900}{3000} = 0.30$ of the membership.

Average over the visits, not over the members. The mean weekly visits of an intercepted member is $\frac{3600(4) + 2100(1)}{5700} = \frac{14400 + 2100}{5700} = \frac{16500}{5700} \approx 2.8947$, or about 2.89 days per week.

(d) Size and direction. The method is centered on 2.89 while the parameter is 1.9, so the bias is $2.8947 - 1.9 \approx 0.99$, about one full day per week too high. The direction follows directly from the mechanism: the people who use the gym most are the people most likely to be standing in the doorway, so the sample overrepresents them and the estimate overstates how often the typical member comes.

(a) Two rates, two meanings. The sample proportion is $\hat{p} = \frac{4300}{5000} = 0.86$, and the response rate is $\frac{5000}{40000} = 0.125$, or 12.5%. The second number is the warning sign: 35,000 selected customers are missing from the result.

(b) Name it. The right people were contacted and most never replied, so this is nonresponse bias. The mechanism is stated in the setup: staying subscribed to the retailer's email is itself a mild sign of satisfaction, so the 5,000 who opened and answered lean happier than the 35,000 who did not. The estimate overstates satisfaction.

(c) Size of the bias. The method is centered on 0.86 and the parameter is 0.71, so the bias is $0.86 - 0.71 = 0.15$, or 15 percentage points too high.

(d) What the extra email buys. Holding the response rate at 12.5% gives $120000 \times 0.125 = 15000$ answers. Treating the answerers as a random draw from the answering group, whose satisfaction rate is about 0.86, the standard deviation of $\hat{p}$ is $\sqrt{\frac{0.86(0.14)}{5000}} = \sqrt{\frac{0.1204}{5000}} = \sqrt{0.00002408} \approx 0.0049$ before, and $\sqrt{\frac{0.1204}{15000}} = \sqrt{0.0000080267} \approx 0.0028$ after. Tripling $n$ divides the spread by $\sqrt{3} \approx 1.732$, so it falls by about 0.2 percentage points.

Compare the two quantities. Before, the bias is $\frac{0.15}{0.0049} \approx 31$ standard deviations. After, it is $\frac{0.15}{0.0028} \approx 53$ standard deviations. The extra email makes the estimate more precisely wrong: the center never moved, and the noise that used to partly disguise the error got smaller.

(e) The answer to give. Sample size is the dial for sampling variability only, and this estimate's problem is a 15 point lean that lives in the method. Raise the response rate instead: follow up by phone or text with a random subsample of the customers who did not answer, and report the estimate from that subsample, which represents the missing 87.5% rather than ignoring them.

(a) Read the centers, not the spreads. Bias is a question about where a method is centered, so compare each mean to the parameter $p = 0.24$. Method A centers at 0.241 and Method B centers at 0.385. Method B is the biased one, and the two numbers that show it are its mean of 0.385 against the true 0.24.

(b) Size, mechanism, direction. The bias is about $0.385 - 0.240 = 0.145$, or 14.5 percentage points. The mechanism is undercoverage: students living more than a mile away were never on Method B's list. Distance is tied to the variable being measured, since a student two miles out is far likelier to ride the bus than to bike, so the reachable group bikes more than the school does. The estimate overstates the proportion who bike.

(c) Is 0.001 evidence of anything? No, and here is the check. The mean of 200 independent sample proportions has standard deviation $\frac{0.060}{\sqrt{200}} \approx \frac{0.060}{14.142} \approx 0.0042$, so a miss of 0.001 is about a quarter of one standard deviation. That is ordinary chance. Method B's miss, by contrast, is $\frac{0.145}{0.069 / \sqrt{200}} \approx \frac{0.145}{0.0049} \approx 30$ standard deviations, which chance does not produce.

(d) Predict the four numbers. Quadrupling the sample size from 50 to 200 divides the standard deviation of $\hat{p}$ by $\sqrt{4} = 2$, and it does nothing at all to the center. Method A: mean still about 0.240, standard deviation about $\frac{0.060}{2} = 0.030$. Method B: mean still about 0.385, standard deviation about $\frac{0.069}{2} = 0.0345$. Confirm both from the formula: $\sqrt{\frac{0.24(0.76)}{200}} = \sqrt{\frac{0.1824}{200}} = \sqrt{0.000912} \approx 0.0302$ and $\sqrt{\frac{0.385(0.615)}{200}} = \sqrt{\frac{0.236775}{200}} = \sqrt{0.001183875} \approx 0.0344$.

Note what part (d) demonstrates. After quadrupling $n$, Method B's results cluster more tightly than ever around 0.385, so its bias has become easier to see and no smaller. A larger sample sharpens a wrong answer.

(e) One sentence. Bias is the distance from the center of a method's sampling distribution to the parameter and depends only on the method, while sampling variability is the spread of that distribution around its own center and shrinks as $n$ grows.

(a) Name and direction. Selection was never the issue, so the lean is in the answers: this is response bias from question wording. The clause before the question hands the parent a reason to say yes and offers nothing on the other side, and it arrives while they are deciding. Support for the later start time will be overstated.

(b) Rewrite it. Strip the argument, put both sides in the question, and offer a way out for parents with no view: "Do you support, oppose, or have no opinion about moving the high school start time from 7:20 a.m. to 8:15 a.m.?" The wording now names the change and nothing else, and neither answer is presented as the sensible one.

(c) Compute both. Draft: $\hat{p} = \frac{148}{200} = 0.74$, or 74%. Rewrite: $\hat{p} = \frac{106}{200} = 0.53$, or 53%. The gap is $0.74 - 0.53 = 0.21$, 21 percentage points, produced entirely by the sentence in front of the question.

(d) Why the random split matters. The two groups of 200 came from one random sample and were formed by chance, so they are alike on average in every way except the wording each one heard. The wording is therefore the only systematic difference between them, and the 21 point gap can be attributed to it rather than to two different kinds of parent. Randomly assigning the wording turns this into an experiment about the question itself.

(e) What survives. Response bias does, in a second form. The caller introduces herself as calling on behalf of the school board, which is the body proposing the change, and parents shade their answers toward whatever the caller seems to want. That pushes the reported support for the board's proposal upward, so 53% is best read as an upper estimate rather than as the parameter.

(a) Two fractions. The response rate is $\frac{180}{500} = 0.36$, or 36%, and among the answerers $\hat{p} = \frac{126}{180} = 0.70$, or 70%.

(b) Name it. The 500 were chosen by a random mechanism and 320 of them never answered, so this is nonresponse bias. A student who got the shot has a quick, creditable answer to give a health center that is plainly hoping to hear it, while a student who skipped it has a reason to leave the envelope unopened. The answerers lean vaccinated, so 0.70 overstates the proportion among the 500.

(c) Bound it with no assumptions. 126 students definitely got the shot. If none of the 320 non-answerers did, the proportion among the 500 is $\frac{126}{500} = 0.252$. If all 320 did, it is $\frac{126 + 320}{500} = \frac{446}{500} = 0.892$. So the parameter for these 500 lies somewhere in 0.252 to 0.892, a band $0.892 - 0.252 = 0.640$ wide.

(d) The reported margin of error. Using $\hat{p} = 0.70$ and $n = 180$, the margin of error is $1.96\sqrt{\frac{0.70(0.30)}{180}} = 1.96\sqrt{\frac{0.21}{180}} = 1.96\sqrt{0.0011667} = 1.96(0.03416) \approx 0.0669$, about 6.7 percentage points. That would be reported as 70% plus or minus 6.7 points.

Compare the two numbers. A 13.4 point interval sits inside a 64 point band of genuine uncertainty. The margin of error measures only the chance variation of a random sample of size 180; it assumes those 180 are a random sample from the population, which is exactly the assumption nonresponse breaks. No formula on the sheet widens an interval to cover bias, which is why a low response rate has to be reported alongside the estimate rather than buried.

(e) The repair. Take a random subsample of the 320 non-answerers, say 60 of them, and pursue those 60 hard with phone calls or a visit until nearly all respond. That subsample represents the missing group, so combining it with the original 180 gives an estimate for all 500. Mailing the survey to 1,500 students instead would leave the response rate, and therefore the lean, exactly where it is. The margin of error arithmetic itself is Unit 3 work, and the margin of error calculator checks it.

(a) Two separate stages, two separate problems. First, selection: nobody drew these 600 people. They were downtown on a Saturday morning and they chose to walk over to the table, so the sample is built from volunteers. That is voluntary response bias, layered on a frame that reaches only residents who visit a weekend farmers market. Second, the data: the question itself carries the words "free" and "instead of fighting for parking," which advertise the shuttle inside the question. That is response bias from question wording. The two are independent flaws, and repairing either one leaves the other in place.

(b) Both push the same way. The people who stop at a table about a beach shuttle are the people who like the idea, and residents who go downtown on weekends are already the ones who travel for leisure, so the volunteer selection overstates support. The wording names a benefit and a pain point and mentions no cost or wait, so it also overstates support. The errors compound rather than cancel, and the estimate is inflated twice.

(c) Compute both. Market: $\hat{p} = \frac{462}{600} = 0.77$, or 77%. Proper survey: $\hat{p} = \frac{129}{430} = 0.30$, or 30%, on a response rate of $\frac{430}{500} = 0.86$. The gap is $0.77 - 0.30 = 0.47$, 47 percentage points.

(d) Price out the staffer's idea. Even pretending the 600 were a random draw from the market crowd, the standard deviation of $\hat{p}$ is $\sqrt{\frac{0.77(0.23)}{600}} = \sqrt{\frac{0.1771}{600}} = \sqrt{0.00029517} \approx 0.0172$. At 6,000 it is $\sqrt{\frac{0.1771}{6000}} = \sqrt{0.000029517} \approx 0.0054$.

Say what that buys. Ten times the work tightens the spread by about 1.2 percentage points, around a number that is 47 percentage points away from the answer the random sample produced. Both flaws are properties of the table and the question, not of the count of people standing at it, so 6,000 market responses would be a more precise measurement of the wrong quantity.

(a) Two populations. The claim is about all Americans. The method can only describe people who had installed this fitness app and were still using it, which is a self-selected slice of the country, not a random sample of it.

(b) Three mechanisms, all pushing up. Undercoverage, and severe: no American outside the app's user list could be selected, and people who install and keep a fitness app exercise far more than people who do not, so the frame itself sits above the national rate. Voluntary response: among users, answering the pop-up was optional, and a user proud of a consistent week taps through while a user who has not opened the app in a month dismisses it, so the answerers lean toward the more active users. Response bias: exercise is self-reported to an app whose whole purpose is encouraging exercise, and people round their own effort up. Every one of the three pushes the 72% higher than the truth, so no cancellation is available.

(c) Response rate. $\frac{38400}{320000} = 0.12$, or 12%. Nearly nine of every ten users said nothing.

(d) Check the margin of error. With $\hat{p} = 0.72$ and $n = 38400$, $1.96\sqrt{\frac{0.72(0.28)}{38400}} = 1.96\sqrt{\frac{0.2016}{38400}} = 1.96\sqrt{0.00000525} = 1.96(0.002291) \approx 0.0045$, about 0.45 percentage points. The arithmetic is right, which is what makes the claim dangerous.

Say what it covers. That 0.45 points is the spread you would see if this same procedure were repeated, and nothing more. It answers the question "how much would the answer bounce around from run to run," not "how far is the center from the truth." A method leaning 20 or 30 points high reports the same tiny margin of error every time it runs, because the formula contains $\hat{p}$ and $n$ and no term at all for the quality of the sampling method. A large $n$ with a broken frame produces a confident number, not a correct one.

(e) A defensible sentence. "Among the 38,400 users who chose to answer an in-app survey, 72% reported exercising at least three times a week." It names the group actually measured, flags that they chose to answer, and says "reported" rather than treating self-report as measured behavior. It makes no claim about Americans, which is the only claim the app wanted and the one its data cannot support. What a study does and does not license is worked through in can you generalize results.