Replication vs Random Assignment

Replication is one of the four elements of a well-designed experiment, alongside comparison, random assignment and direct control, and it means one thing: more than one experimental unit receives each treatment. Count the units per treatment and you have counted the replication.

Why it is needed: every measured difference is a treatment effect plus the ordinary difference between units. Put the new fertilizer on one plot and the standard on one plot, and a 3 kilogram gap in yield could be the fertilizer or could be that the first plot drains better. Nothing in the data separates them. With many units per treatment, unit-to-unit variation averages out of each group mean. If the response has standard deviation $\sigma = 6$ points, a group of 4 units has a mean with standard error $6/\sqrt{4} = 3$ points, while a group of 36 has $6/\sqrt{36} = 1$. Nine times the units, one third the noise.

Two sentences to reject. "We replicated the experiment by running it again the next year" describes a replication study, which is a good thing to do and is not this design element; the count here happens inside one experiment. "Each subject was measured three times, so the treatment was replicated three times" is worse, because three readings on one person are one unit measured three times, and those readings are not independent of each other.

Watch what the unit actually is. If a treatment is applied to a whole classroom, the classroom is the experimental unit, so 4 classrooms of 25 students per treatment is a replication of 4, not 100. Counting the students makes the comparison look far more precise than it is.

Replication buys precision, not protection from bias. More units per treatment shrink the standard error and do nothing about a confounded design; that job belongs to random assignment. Both are listed in topic 1.13 Experimental Design.

Random assignment means a chance device, not the experimenter, decides which unit gets which treatment, and the device's probabilities do not depend on anything about the unit. That is the whole requirement. It does not demand equal group sizes and it does not promise that the finished groups will look alike. What it buys is that every variable other than the treatment was spread across the groups by that same chance mechanism, so a difference in the response has only two explanations left, the treatment or chance, and the p-value measures the second one.

Balance is a tendency, not a guarantee. Take 20 subjects, 10 of them women, split into two groups of 10 by shuffling names. The expected split is 5 and 5, but the chance of landing exactly there is $\frac{\binom{10}{5}^2}{\binom{20}{10}} = \frac{63504}{184756} = 0.344$. A group holding 8 or more of the 10 women turns up about 2.3 percent of the time, and all 10 landing together about once in 92,000 assignments.

So "the treatment group came out older on average, so the randomization failed" reads the wrong thing. Random assignment is judged on the procedure used, not on the split it produced, and imbalance of exactly that size already sits inside the reference distribution the p-value comes from. Redrawing until the groups look even destroys that: the assignment is no longer random and the stated error rate no longer holds.

The boundary is who is in the study at all, and random assignment says nothing about it. Forty volunteers randomly assigned support a causal claim about people like those volunteers and about no one else. Widening the audience takes random selection, a separate act on a separate list; see scope of inference.

Topic 1.13 lists random assignment beside comparison, replication, and direct control as the four elements of a well-designed experiment.