Generalizability vs Scope of Inference

Generalizability is a question about the audience for a result, and it is settled by how the sample was selected. Randomly selecting units from a population licenses a conclusion about that population. Without random selection, the conclusion reaches only individuals similar to the ones actually studied. Random assignment settles a different question, whether the difference can be credited to the treatment, and the two choices are made independently, so there are four combinations.

A university randomly selects 300 of its 18,000 enrolled students from the registrar's list, then randomly assigns 150 to a new advising portal and 150 to the existing walk-in system. Both dials are set, so a difference in credits completed can be credited to the portal and extended to all 18,000 students at that university. Not to college students in general: you may generalize to the population the sample was drawn from, and nobody sampled another campus.

The sentence that costs the most points is "the sample was large, so the results generalize." Size buys precision, never representativeness, which is why a bigger sample does not fix bias. Its mirror image is "the experiment was randomized, so it applies to everyone." That cites the wrong randomization. Random assignment builds comparable groups out of whoever is already in the study and says nothing about who that is.

Random selection can also be undone after the fact. If a third of the selected students never reply, the people you measured were no longer randomly selected, and the reach of the conclusion shrinks with them. Generalizing also stops at the sampling frame rather than at the population you had in mind.

Scope of inference is two answers, not one: what kind of relationship the study supports, and about whom. The first is settled by whether treatments were randomly assigned to the units. The second is settled by whether the units were randomly selected from a stated population. Both are fixed by the design before any data exist, and nothing done during the analysis widens either one.

A state association has 1,200 licensed electricians. A researcher randomly selects 120 of them from the membership roll, randomly assigns 60 to a pre-inspection checklist and 60 to current practice, and records faults per 100 inspections: 4.1 for the checklist group against 6.3 for current practice, a difference of 2.2. Both randomizations are present, so the conclusion has both halves. Because the checklist was randomly assigned, it caused the lower fault rate. Because the 120 were randomly selected from the roll, that reaches the association's 1,200 electricians and stops there.

"The result was significant at $\alpha = 0.01$, so the effect holds for electricians generally" is the sentence to kill. Evidence is not scope. A p-value says how surprising the data would be if the null were true; it knows nothing about who was recruited or how treatments were handed out. A smaller p-value, a narrower interval, and a fancier model all leave the audience and the causal permission exactly where the design left them.

Missing a randomization is a smaller claim rather than a broken study, and the four combinations of the two are laid out under generalizability. The narrow cells are the ordinary ones: almost every experiment runs on volunteers, and almost every survey assigns nothing to anybody.

Scope of inference surfaces twice in the course, in topic 1.10 on data collection and again in topic 1.13 on experimental design. Write it as two sentences, each naming the randomization that licenses it.