What is a randomization test? Shuffle and count

What a randomization test does

Every hypothesis test needs one thing: a distribution of the statistic assuming the null hypothesis is true. A $t$ test gets that distribution from a formula and a named curve. A randomization test gets it by rebuilding it out of your own data.

The procedure is short enough to state in a sentence. Take the response values you collected, throw the group labels back in a hat, deal them out at random, recompute the statistic, and write it down. Do that a few thousand times and the numbers you wrote down form a picture of what the statistic looks like when the labels mean nothing. Then see where your real result falls in that picture.

Nothing in there requires a normal population, a large sample, or a memorized standard error. The only thing it requires is that the labels were assigned at random in the first place, which is exactly what a randomized experiment guarantees. That is the trade: you give up a closed-form formula and you get a test whose validity rests on the design of the study rather than on assumptions about a population. Where a $t$ procedure is robust to mild non-normality but still leaning on the condition, a randomization test never leaned on it at all.

The logic: under the null, the labels are arbitrary

Here is the idea the whole method turns on, and it is worth reading twice.

Suppose the null hypothesis is true and the treatment does nothing at all. Then each subject's response would have been exactly the same number no matter which group that subject landed in. The response belongs to the subject; the label was assigned by a coin flip. So the split you actually observed is just one of the many splits the randomization could have produced, and every other split was equally likely.

That makes the reshuffling honest rather than a trick. You are not inventing data. You are listing the other outcomes your own random assignment could have delivered, under the assumption that the treatment changed nothing. If your real difference sits comfortably inside that crowd of possible differences, the randomization alone explains it. If almost none of the reshuffles reach as far as your result, then either something unlikely happened or the assumption of no effect is wrong.

Notice what the assumption buys and what it does not. Random assignment of labels is what licenses the shuffle, so a randomization test on an experiment supports a conclusion about cause. Generalizing beyond the subjects still needs random selection, exactly as in scope of inference.

The four steps

1. Compute the statistic on the real data. Usually a difference in means or a difference in proportions. Call it the observed value and keep its sign.
2. Pool and reshuffle. Put every response into one pile, ignoring which group it came from. Deal the responses back out into groups of the original sizes, at random.
3. Recompute and record. Work out the same statistic on the shuffled groups and record it. Repeat a few thousand times. The recorded values are the randomization distribution, and it will be centred near zero, because you built it under the assumption of no effect.
4. Count. Count how many recorded values are at least as extreme as the observed one, and divide by the number of shuffles. For a two-sided alternative, count both tails by using absolute values.

Step 4 is where students slip. "At least as extreme" means at or beyond the observed value, so the count includes ties, and the comparison is against the observed value itself rather than against any cutoff. The direction comes from the alternative hypothesis, the same way it does in a formula-based test. You can build the same habit on ordinary probability problems in how to design and run a simulation.

It is the definition of a p-value, run literally

A p-value is the probability of getting a result at least as extreme as the observed one, assuming the null hypothesis is true. Students memorize that sentence and then compute p-values with a calculator that hides every word of it.

A randomization test hides nothing. Read the definition and the procedure side by side:

The p-value stops being a number that appears on a screen and becomes a fraction you could count on your fingers if you had the patience. That is the real reason to learn the method even if you never run one on an exam: it makes what a p-value means concrete, and it makes the classic misreadings visibly wrong. The fraction is computed inside a world where the null hypothesis is assumed true, so it can never be the probability that the null hypothesis is true.

Small studies: you can list every possible shuffle

With a small experiment you do not have to simulate at all. You can enumerate every reassignment and get an exact answer.

Take five subjects in each group, ten in total. The number of ways to choose which five subjects get the treatment label is

So the randomization distribution has exactly 252 members, one for each possible assignment, and the p-value is a fraction with 252 in the denominator. In the worked example below, 7 of those 252 assignments produce a difference in means of 5.2 or more, giving a one-sided p-value of $7/252 = 0.0278$, and 14 reach 5.2 or more in absolute value, giving a two-sided p-value of $14/252 = 0.0556$.

The distribution built this way has two features worth noticing. It is centred exactly at 0, because the null hypothesis of no effect is what you assumed while building it. And because the two groups are the same size here, it is symmetric: every assignment has a mirror image that swaps the groups and flips the sign of the difference. That symmetry is why the two-sided count is exactly double the one-sided count here. With groups of different sizes the mirror image is not another assignment of the required shape, and the distribution need not be symmetric.

Enumeration stops being possible fast. Ten subjects per group already gives $\binom{20}{10} = 184{,}756$ assignments, and 15 per group gives more than 155 million. That is why the practical version is simulation: a random sample of the possible shuffles rather than all of them.

How many shuffles is enough

A simulated p-value is an estimate, so it carries its own error, and the source of that error is the number of shuffles rather than the number of subjects.

On the same ten-subject experiment, three runs of 1,000 shuffles each gave two-sided p-values of 0.052, 0.051 and 0.065, against the exact answer of 0.0556. Two of the estimates land 0.0036 and 0.0046 below the exact 0.0556 and the third lands 0.0094 above it, so the scatter runs to about a hundredth, not a thousandth. That does not matter when the p-value is 0.30 or 0.001, and it matters a great deal when the p-value is near 0.05 and someone is about to decide which side of the line it falls on.

The rule of thumb: the standard error of a simulated p-value near $p$ is $\sqrt{p(1-p)/N}$ for $N$ shuffles, so it shrinks like $1/\sqrt{N}$. At $N = 1000$ with $p$ near 0.0556 that is $\sqrt{\frac{0.0556(0.9444)}{1000}} = 0.0072$, which is the size of the scatter above. A thousand shuffles is enough for a classroom illustration, ten thousand for a result you intend to report, and if the answer sits near your significance level, run more rather than argue about the third decimal.

One more caution about counting. A simulated p-value can come out as 0 out of 10,000, and it is wrong to report it as zero. No number of shuffles can prove a probability is zero. Report it as less than 0.0001, which is what the simulation actually established.

What the Fall 2026 AP course does and does not ask for

This section sticks to what the Fall 2026 course framework lists, because the difference between what is taught and what is tested matters here.

The course has five units. Its topic list contains exactly one topic on simulation: Unit 2 topic 2.3, Estimating Probabilities Using Simulation. No topic in the five-unit list is titled randomization test or randomization distribution.

The tests you are asked to carry out are named for their formulas. Topic 3.7 is Carrying Out a Test for a Population Proportion, 3.13 is Carrying Out a Test for the Difference Between Two Population Proportions, 3.15 is Carrying Out a Chi-Square Test for Homogeneity or Independence, 4.5 is Carrying Out a Test for a Population Mean or Population Mean Difference, and 4.10 is Carrying Out a Test for the Difference Between Two Population Means. Free-response question 3 on the exam is an inference question, a hypothesis test or a confidence interval.

So, honestly: build and read simulated distributions, because that is topic 2.3 and it is where a p-value stops being a formula and becomes a count. Learn to look at a dotplot of simulated results, count the dots at or beyond the observed value, and divide. But the topic list gives no basis for expecting a question to ask you to run a randomization test in place of one of the named procedures. Treat this as the idea behind the p-value rather than as a procedure you will be asked to execute, and use the named test when the question says test.