Generalizability
By Jude Wallis · Updated
Generalizability is whether a study's results extend to a larger population, and it depends on how the sample was selected, not on random assignment.
Generalizability is a question about the audience for a result, and it is settled by how the sample was selected. Randomly selecting units from a population licenses a conclusion about that population. Without random selection, the conclusion reaches only individuals similar to the ones actually studied. Random assignment settles a different question, whether the difference can be credited to the treatment, and the two choices are made independently, so there are four combinations.
| How the study was built | Treatments randomly assigned | No random assignment |
|---|---|---|
| Units randomly selected | Cause and effect, for the whole population | Association only, for the whole population |
| Units not randomly selected | Cause and effect, for units like those studied | Association only, for units like those studied |
A university randomly selects 300 of its 18,000 enrolled students from the registrar's list, then randomly assigns 150 to a new advising portal and 150 to the existing walk-in system. Both dials are set, so a difference in credits completed can be credited to the portal and extended to all 18,000 students at that university. Not to college students in general: you may generalize to the population the sample was drawn from, and nobody sampled another campus.
The sentence that costs the most points is "the sample was large, so the results generalize." Size buys precision, never representativeness, which is why a bigger sample does not fix bias. Its mirror image is "the experiment was randomized, so it applies to everyone." That cites the wrong randomization. Random assignment builds comparable groups out of whoever is already in the study and says nothing about who that is.
Random selection can also be undone after the fact. If a third of the selected students never reply, the people you measured were no longer randomly selected, and the reach of the conclusion shrinks with them. Generalizing also stops at the sampling frame rather than at the population you had in mind.
More collecting data and study design terms, or browse the full statistics glossary.