Randomization test

If the treatment does nothing, each subject's response would have been the same number in either group, so the split you observed is only one of many your random assignment could have produced. A randomization test lists the others: deal the responses back into groups of the original sizes, recompute the statistic, and repeat. Those values form the randomization distribution, and the p-value is the fraction at least as extreme as the observed one.

Ten volunteers, five per group. Treatment scores 22, 25, 27, 29, 32 average 27; control scores 18, 20, 21, 24, 26 average 21.8; the observed difference is 5.2. There are $\binom{10}{5} = 252$ ways to choose which five carried the treatment label, few enough to list instead of simulate. Seven of the 252 reach 5.2 or more, a one-sided p-value of $7/252 = 0.0278$; 14 reach 5.2 or more in absolute value, a two-sided p-value of $14/252 = 0.0556$.

"None of my 10,000 shuffles were that extreme, so the p-value is 0" is the report to refuse. A simulation can only bound a probability by the run's resolution, so the honest line is under 0.0001. The quieter counting slip is dropping your own result: at least as extreme means at or beyond, so ties count, and the comparison is against the observed statistic, not any cutoff.

A simulated p-value carries error of its own, and the source is the number of shuffles, not of subjects. Near $p = 0.0556$ with 1,000 shuffles the standard error is $\sqrt{0.0556(0.9444)/1000} = 0.0072$, so such runs scatter by roughly a hundredth: harmless at $p = 0.30$, decisive beside $\alpha$. Enumeration stops being an option fast, since ten per group already gives $\binom{20}{10} = 184{,}756$ assignments.

The Fall 2026 course defines the randomization distribution in topic 2.12, Sampling Distributions and the Central Limit Theorem: the distribution of a statistic generated by simulation from repeatedly reassigning the response values to treatment groups. Topic 2.3 is Estimating Probabilities Using Simulation. The tests you are asked to carry out are still the named $z$, $t$, and chi-square procedures of Units 3 and 4.