Scope of inference practice problems (8 solved)

(a) Both randomizations are present. Treatments were randomly assigned, so cause and effect is supported, and the 300 patients were randomly selected from the 18,000-patient registry, so the conclusion generalizes to that registry. This is the top-left cell, the strongest position a study can be in.

(b) Selection is random, assignment is absent. Nobody assigned dog ownership; the firm recorded what already existed, so this is an observational study and only an association can be reported. The 1,000 were randomly selected from the voter file, so that association generalizes to the people on that file. Note the population is the registered voters on the file, not all adults in the state, since unregistered adults could not be selected.

(c) Assignment is random, selection is not. The routines were randomly assigned, so a difference in sprint times can be attributed to the routine. The 40 people volunteered from one gym's sign-up sheet, so the conclusion covers those 40 and people like them, not gym members in general.

(d) Neither randomization is present. Coffee intake was not assigned, so association only. The 900 chose to reply to an open invitation, so the result describes people similar to the subscribers who chose to reply, and even within her subscriber list the volunteers may not resemble the quiet majority.

Line up the four. Reading the grid, (a) is cause and effect plus generalization, (b) is generalization without cause, (c) is cause without generalization, and (d) is neither. The first question is always what was assigned; the second is always who was selected.

(a) Two rates. App users: $\frac{1820}{40000} = 0.0455$, or 4.55%. Non-users: $\frac{3900}{60000} = 0.065$, or 6.50%. The ratio is $\frac{0.0455}{0.065} = 0.70$, so the app group's claim rate is 70% of the other group's, a 30% lower rate and a difference of $0.065 - 0.0455 = 0.0195$, or 1.95 percentage points.

(b) Study type. Customers decided for themselves whether to install the app, so no treatment was imposed and the insurer recorded what happened. That makes it an observational study, and an observational study can establish an association but not a cause.

(c) Name a confounder. Driving caution, meaning the habits a driver already had before hearing about the app. First link: cautious drivers are far more willing to let an insurer watch their braking and speed, because they expect the recording to flatter them, so caution is associated with installing the app. Second link: cautious drivers crash less regardless of any app, so caution affects the claim rate directly. Because the careful drivers piled into the app group, the effect of the app and the effect of habits those drivers already had cannot be separated, and the 1.95 point gap could belong to either one.

Check what does not qualify. The lower premium the insurer gives app users is a consequence of installing the app, not something sitting behind the choice, so it is part of how the app might work rather than a rival explanation. A confounding variable has to be tied to the explanatory variable and also affect the response on its own.

(d) The defensible conclusion. Causation: because customers chose the app rather than being randomly assigned to it, this study shows only that app use and claim rate are associated, and the association may be explained by the kind of driver who opts in. Generalization: the 100,000 customers are this insurer's whole book of business rather than a random sample of drivers, so the association describes this insurer's customers and should not be extended to drivers generally. To defend the causal claim the insurer would have to randomly assign customers to install the app or not.

(a) The difference. $21.8 - 12.4 = 9.4$ minutes, so the curfew group fell asleep 9.4 minutes faster on average.

(b) Causation. Because the 90 volunteers were randomly assigned to the two conditions, the two groups were alike on average with respect to age, caffeine habits, workload, and every other variable at the start. A 9.4 minute difference larger than chance variation would produce is therefore evidence that the screen curfew caused the faster sleep onset.

(c) Generalization. The 90 were volunteers from one university's psychology participant pool, not a random sample of any population, so the conclusion applies to people similar to these volunteers: university students who sign up for sleep studies. It does not extend to adults in general, whose evening screen use, work schedules, and ages differ from a student pool's.

(d) What would justify "adults." Only random selection of the participants from the population of adults would support that word, since generalization comes from how the units entered the study, not from how treatments were assigned. That change is hard because an experiment requires people to follow a two-week protocol, and a randomly selected adult can refuse. Anyone who agrees has volunteered, which puts the study back where it started. The realistic middle path is to recruit volunteers from several settings, say a hospital, a workplace, and a retirement community, so that the group at least resembles a wider range of adults, and to say plainly that the sample was not random.

(e) Why the two questions stay separate. Random assignment works inside the group you already have: it distributes the extraneous variables carried by those 90 people evenly across the two conditions, so the only systematic difference between the groups is the curfew. That argument never mentions the rest of the world, so it holds no matter how the 90 arrived. Random selection is what connects the 90 to a larger population, and it is missing, so the causal finding is real but local.

(a) Three rates. Near sunflowers: $\frac{204}{240} = 0.85$, or 85%. Not near: $\frac{429}{660} = 0.65$, or 65%. Overall: $\frac{204 + 429}{900} = \frac{633}{900} \approx 0.7033$, about 70.3%. The gap is $0.85 - 0.65 = 0.20$, 20 percentage points.

(b) Locate it in the grid. The 900 were randomly selected from the 12,000 registered beekeepers, so random selection is present. Nobody assigned hive locations; keepers had already put their hives where they wanted, so random assignment is absent. This is the top-right cell: generalization yes, causation no.

(c) The two halves. Generalization: because the 900 were a simple random sample of the state's 12,000 registered beekeepers, the 20 point difference generalizes to that population of registered beekeepers. Causation: because hive placement was not assigned, the study shows an association between sunflower proximity and winter survival, not that proximity causes survival.

(d) A confounder. The keeper's experience and management practice. First link: keepers who run larger, more deliberate operations are the ones who negotiate placements next to sunflower fields, while a hobbyist puts two hives in the back garden, so experience is associated with proximity. Second link: experienced keepers treat for mites on schedule, feed in autumn, and insulate for winter, all of which raise survival regardless of what is growing nearby. The two effects arrive together, so neither the sunflowers nor the management can be credited with the 20 points.

(e) A design that would earn the causal claim. Take the same random sample of registered keepers, ask each to supply two comparable hives, then randomly assign one hive of each pair to a sunflower-adjacent site the department arranges and the other to a site with no sunflowers nearby. Random assignment inside each pair balances the keeper's own management across the two conditions, because the same keeper tends both hives. With random selection and random assignment both present, the study would support a causal conclusion for the state's registered beekeepers. The catch is that participation is voluntary, so keepers who decline would drag it back toward a volunteer sample.

(a) Counts. $\frac{60}{480} = 0.125$, so 12.5% of the state's middle schools were selected, and the 60 split into $\frac{60}{2} = 30$ schools per group.

(b) The units are schools. A schedule was assigned to a school, not to a student, so the experimental unit is the school and the response is the school's mean gain. That controls the wording: the study compares 30 schools with 30 schools, and every conclusion is a statement about schools adopting a schedule. Writing it as a statement about students would treat thousands of students as separately assigned units, which they were not.

(c) The two halves. Causation: because the 60 schools were randomly assigned to the two schedules, the two groups of schools were alike on average with respect to size, funding, and prior performance, so a $6.2 - 2.9 = 3.3$ point difference in mean gain larger than chance variation is evidence that the reading block caused the higher gain. Generalization: because the 60 schools were randomly selected from the state's 480 middle schools, that causal conclusion extends to the state's middle schools.

(d) The neighboring state. Not supported by this study. Random selection was from one state's 480 middle schools, so the population the result covers is that state's middle schools. A neighboring state may differ in curriculum, school day length, and student population, and nothing in this design speaks to it. The principal is making a judgment that his schools resemble the ones studied, which may be sensible but is not a statistical conclusion. Replicate in his own state to support it.

(e) Fix the reporter's sentence. Two things are wrong: the unit and the certainty. The 3.3 points is a difference between two group means of schools, not a promise for any one student or even any one school, and school means vary around it. Rewrite: "Among the middle schools in the study, those randomly assigned the 90-minute reading block gained 3.3 more points on average than those keeping the current schedule." That names the units, keeps the claim on means, and keeps the random assignment visible.

(a) Two calculations. The difference is $3.28 - 3.09 = 0.19$ GPA points. The overall mean is the weighted average $\frac{185(3.28) + 315(3.09)}{500} = \frac{606.8 + 973.35}{500} = \frac{1580.15}{500} = 3.1603$, which rounds to 3.16 as reported.

(b) Two separate overreaches. The word "raises" claims cause. Nobody assigned students to study in groups; the researcher recorded a choice students had already made, so this is an observational study and only an association is supported. The word "nationwide" claims a population. Random selection was from one university's list of 21,000 students, so the sample represents that university and nothing beyond it. One flaw is about assignment, the other about selection, and fixing one leaves the other untouched.

(c) A headline that fits. "Students at this university who study in groups have higher GPAs on average." It reports an association, names the population that was actually sampled, and drops any suggestion that group study produced the difference. Adding the size is fair: the gap was 0.19 GPA points.

(d) A confounder. Course load difficulty, or more precisely the major a student is in. First link: group study is normal in engineering and pre-med sequences with weekly problem sets and rare in majors built on solo reading and essays, so major is associated with group study. Second link: majors differ in how strictly they grade, so major affects GPA on its own. Prior academic preparation works the same way: students who arrive with stronger study habits both seek out study groups and earn higher grades. Either variable would produce a GPA gap with no contribution from group study at all.

(e) Two different repairs. To earn the word "raises," randomly assign the sampled students to a group study requirement or to solo study for a term and compare GPAs, since assignment is what balances major and preparation across the groups. To earn the word "nationwide," draw the sample randomly from a national list of enrolled college students rather than from one university's roster. A study with both changes would support the original headline; a study with either one alone supports only half of it.

(a) Counts and rates. Analyzed: $100 - 34 = 66$ exercise patients and $100 - 6 = 94$ usual-care patients. Dropout rates: $\frac{34}{100} = 0.34$, or 34%, against $\frac{6}{100} = 0.06$, or 6%, a difference of 28 percentage points. The reported gap is $462 - 418 = 44$ meters.

(b) With everyone measured. Treatments were randomly assigned, so a difference larger than chance variation would support the conclusion that the exercise program caused the longer walk distance. Generalization would still be limited, because the 200 came from one clinic's waiting list rather than a random sample, so the conclusion would cover patients similar to these.

(c) Why the finishers are not the randomized groups. Random assignment created two groups of 100 that were alike on average. The groups being compared are no longer those: they are 66 patients and 94 patients who were filtered by an event that happened after assignment and depended on the treatment received. Quitting a supervised exercise program is not a coin flip; it happens to the patients whose knees hurt most and who are least able to keep up. The two analyzed groups were therefore formed partly by the patients themselves, which is exactly what random assignment existed to prevent, so the comparison has become observational.

(d) The direction. The 34 who quit the exercise arm are, on average, the ones doing worst on it, and removing them leaves the fitter, more tolerant two thirds. Almost nobody was filtered out of the usual-care arm by comparison, since only 6 left. The exercise group's mean is therefore lifted by the removal in a way the usual-care group's is not, so 44 meters overstates the program's effect. Note also that a 34% dropout rate is itself a result: a program most patients cannot finish is worth less than one they can.

(e) What to report and what to change. Defensibly: among patients who completed 12 weeks, the exercise finishers walked 44 meters farther on average than the usual-care finishers, an association that may reflect who was able to finish rather than the program itself, and the exercise program had a 34% dropout rate against 6% for usual care. Next time, measure the 6-minute walk on all 200 patients at week 12 whether or not they stuck with the program, and analyze them in the groups they were assigned to, which keeps the randomization intact and keeps the causal conclusion available.

(a) Rates first. Before: $\frac{84}{2100000} = 0.00004$ per trip, which is $0.00004 \times 100000 = 4.00$ collisions per 100,000 trips. After: $\frac{63}{2800000} = 0.0000225$ per trip, which is $2.25$ collisions per 100,000 trips. The decrease is $4.00 - 2.25 = 1.75$ per 100,000 trips, and $\frac{1.75}{4.00} = 0.4375$, a 43.75% decrease in the rate.

(b) Why counts mislead. The denominator changed. Trips rose from 2.1 million to 2.8 million, an increase of $\frac{2.8 - 2.1}{2.1} \approx 0.333$, or about 33%, so more riding was happening in the second year. A count of 63 against 84 understates the improvement, because those 63 collisions are spread over a third more cycling. The rate per trip is the comparable quantity.

(c) Classify the evidence. No treatment was randomly assigned to anything: the city changed its streets and watched. No random selection took place either, since this is one city's downtown observed at two points in time. This is an observational before-and-after study, in the bottom-right cell, so it supports an association between the year and the collision rate for this city's downtown and nothing more.

(d) The speed limit as a confounder. First link: it changed at exactly the same time as the bike lanes, so it is perfectly associated with the explanatory variable, which here is before versus after. Second link: lower vehicle speeds give drivers more stopping time and reduce collisions on their own. Because the two changes are locked together, no amount of data from these two years can say which one produced the drop, or how much each contributed. A city-wide downward trend in collisions, or better weather in the second year, would work the same way.

(e) A design that would earn the causal claim. Suppose the city has 260 comparable downtown corridors. Randomly select 40 of them, randomly assign 20 to receive protected bike lanes and 20 to keep their current layout, hold the speed limit at 25 miles per hour on all 40, and compare collisions per 100,000 trips after one year. Random assignment makes the two sets of corridors alike on average in traffic volume, width, and weather, so a difference in rate can be attributed to the lanes; random selection from the 260 corridors extends that conclusion to the city's comparable downtown corridors. Note that the experimental units are corridors, so the conclusion is about corridors, and the study covers this one city rather than cities in general.