Randomization Test vs Hypothesis Test

If the treatment does nothing, each subject's response would have been the same number in either group, so the split you observed is only one of many your random assignment could have produced. A randomization test lists the others: deal the responses back into groups of the original sizes, recompute the statistic, and repeat. Those values form the randomization distribution, and the p-value is the fraction at least as extreme as the observed one.

Ten volunteers, five per group. Treatment scores 22, 25, 27, 29, 32 average 27; control scores 18, 20, 21, 24, 26 average 21.8; the observed difference is 5.2. There are $\binom{10}{5} = 252$ ways to choose which five carried the treatment label, few enough to list instead of simulate. Seven of the 252 reach 5.2 or more, a one-sided p-value of $7/252 = 0.0278$; 14 reach 5.2 or more in absolute value, a two-sided p-value of $14/252 = 0.0556$.

"None of my 10,000 shuffles were that extreme, so the p-value is 0" is the report to refuse. A simulation can only bound a probability by the run's resolution, so the honest line is under 0.0001. The quieter counting slip is dropping your own result: at least as extreme means at or beyond, so ties count, and the comparison is against the observed statistic, not any cutoff.

A simulated p-value carries error of its own, and the source is the number of shuffles, not of subjects. Near $p = 0.0556$ with 1,000 shuffles the standard error is $\sqrt{0.0556(0.9444)/1000} = 0.0072$, so such runs scatter by roughly a hundredth: harmless at $p = 0.30$, decisive beside $\alpha$. Enumeration stops being an option fast, since ten per group already gives $\binom{20}{10} = 184{,}756$ assignments.

The Fall 2026 course defines the randomization distribution in topic 2.12, Sampling Distributions and the Central Limit Theorem: the distribution of a statistic generated by simulation from repeatedly reassigning the response values to treatment groups. Topic 2.3 is Estimating Probabilities Using Simulation. The tests you are asked to carry out are still the named $z$, $t$, and chi-square procedures of Units 3 and 4.

A hypothesis test, also called a significance test, assumes a claim about a population parameter and asks how unusual the observed sample would be if that claim held. The machinery is fixed: state $H_0$ and $H_a$, check the conditions, compute a test statistic, turn it into a p-value using the null distribution, and compare that p-value with a significance level $\alpha$ (alpha) fixed beforehand. The output is one of two verdicts, reject $H_0$ or fail to reject $H_0$, and never a probability that either hypothesis is true.

Test $H_0: \mu = 100$ against $H_a: \mu > 100$ with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is $15/\sqrt{25} = 3$. A sample mean of 106.2 gives $z = \frac{106.2 - 100}{3} = 2.07$ and a p-value of 0.0194. At $\alpha = 0.05$ that is a rejection: samples this far above 100 turn up under 2 percent of the time when 100 is the truth.

The misreading that costs points: "the p-value was 0.16, so we accept the null hypothesis and conclude the mean is 100." A test never accepts $H_0$. A sample mean of 103 in that same setup gives $z = 1.00$ and a p-value of 0.1587, and a 95 percent interval of $103 \pm 1.96(3)$, or 97.12 to 108.88. Every value in that range is as consistent with the sample as 100 is, so failing to reject rules nothing in. The asymmetry is deliberate: evidence can contradict one specific value, and no sample can confirm one.

A test also answers only the question its hypotheses posed and does not audit its own scope. Whether a conclusion reaches a population depends on random sampling, and whether it supports a cause depends on random assignment: a rejection from a convenience sample is still a rejection and still worth very little. Significant is not the same as large, the subject of statistical vs practical significance.

In the Fall 2026 AP Statistics course, one-proportion tests are Unit 3 topics 3.5 through 3.7, and tests for means are Unit 4 topics 4.4 and 4.5.