Experimental design classroom activity: memory test
By Jude Wallis · Published
Students are randomly assigned one of two sheets holding the same 20 words, grouped by category or scrambled, study for 60 seconds, and write down what they recall. The class compares mean recall, then reshuffles the scores to see how large a difference random assignment alone produces.
AP Statistics: Unit 1 (topics 1.13 Experimental Design, 2.12 Sampling Distributions and the Central Limit Theorem). The experiment itself is topic 1.13: treatments, experimental units, random assignment, control and replication. The reshuffled differences form a randomization distribution, which the framework introduces in topic 2.12.
What the activity shows
This is a completely randomized experiment with two treatments that fits in one period. The treatments are two arrangements of the same 20 words. The experimental units are the students. The response is the number of words recalled, from 0 to 20.
Students experience each design principle as something they did: comparison of two treatments, random assignment by a shuffled stack of sheets, replication across many students, and control of everything the teacher can hold fixed. They then analyze the result with index cards, asking how often a difference as large as theirs appears when the scores are dealt into two groups by chance.
Memory research has long reported that organized material is recalled better than scrambled material, so the grouped sheet is the likely winner. The size of the difference in your room is unknown, and the analysis must not assume one.
Materials and timing
- Half sheets of paper, half printed with version A and half with version B, one per student. Write a small A or B in a top corner.
- A timer.
- One index card or sticky note per student for the score.
- Small slips of paper, enough for each group of four to copy every score in the class.
- Two number lines on the board, one above the other, both marked from 0 to 20.
Version A, grouped. Four rows of five words:
| Row | Words |
|---|---|
| 1 | horse, tiger, rabbit, eagle, salmon |
| 2 | hammer, wrench, ladder, shovel, pliers |
| 3 | bread, cheese, carrot, apple, butter |
| 4 | jacket, sweater, glove, scarf, boot |
Version B, scrambled. The same 20 words:
| Row | Words |
|---|---|
| 1 | hammer, bread, jacket, tiger, shovel |
| 2 | apple, eagle, glove, wrench, carrot |
| 3 | scarf, horse, pliers, butter, rabbit |
| 4 | ladder, boot, cheese, salmon, sweater |
Do not print category names on either version. Plan on 45 minutes: 3 to hand out sheets, 5 for study and recall, 7 to score and plot, 10 to discuss the design, 15 for the reshuffling, and 5 to conclude.
Steps
- Before class, stack equal numbers of version A and version B. With an odd class size, one version gets one extra sheet.
- In front of the students, shuffle the stack thoroughly and deal one sheet face down to each student. Say only that the shuffle decides who gets which sheet.
- On your signal students turn the sheet over and study the words in silence for 60 seconds.
- Students turn the sheet face down and have 2 minutes to write every word they recall on the back.
- Students turn the sheet over, count the words they recalled correctly, and write that score and their version letter on an index card.
- Students place their cards on the board as dots: version A on the top number line, version B on the bottom.
- Compute the mean score for each version and the difference, mean of A minus mean of B. Decide this direction before looking at the numbers.
- Show both versions side by side and name the parts of the design: treatments, experimental units, response, random assignment, and what was held the same for everyone.
- Reshuffle. Each group of four copies every score onto slips with no version letter, shuffles the slips, deals them into two piles the same sizes as the real groups, and computes the difference in means, first pile minus second pile. Only the first pile needs adding up, because the second pile's total is the class total minus the first. Each group does this 5 times and reports its 5 differences.
- Plot every reshuffled difference on a new number line. Count how many are at least as large as the real difference from step 7, and divide by the number of reshuffles.
What students should see
Random assignment spreads other variables by chance. It does not equalize them. With 30 students split 15 and 15, the shuffle picks one of equally likely assignments. Suppose 12 of the 30 students slept badly last night. The chance they split exactly 6 and 6 is 0.290. The chance of a 6 and 6 or a 7 and 5 split is 0.736. That leaves a 0.264 chance that one group gets 8 or more of the 12. So the two groups will differ somewhat in sleep, vocabulary, and everything else, and the size of those differences is governed by chance alone.
That leaves two explanations for a difference in mean recall. One is the arrangement of the words. The other is the luck of the shuffle. The reshuffling in steps 9 and 10 measures the second explanation directly.
The reshuffled differences center on 0. If the arrangement made no difference, each student would have scored the same on either sheet, and the version letters would be arbitrary labels. Dealing the scores into two piles at random then shows the differences that labels alone produce. With enough reshuffles their dotplot is centered near 0 and roughly symmetric.
The fraction in step 10 is an estimated p-value. If the real difference sits far out in the right tail of the reshuffled differences, chance is a poor explanation and the class has evidence that the grouped arrangement caused higher recall for these students. If the real difference sits in the middle of the plot, the data are consistent with no effect. With about 35 reshuffles the estimate is coarse: each reshuffle is worth about 0.03.
Scope. The students were randomly assigned, so a convincing difference supports a cause-and-effect conclusion. They were not randomly selected from any larger population, so the conclusion applies to this class.
Discussion questions
- Why did a shuffled stack decide who got which sheet? What could go wrong if the front half of the room got version A and the back half got version B?
- Both versions used the same 20 words, the same 60 seconds and the same room. What would we be unable to conclude if version B had used harder words?
- Did the shuffle make the two groups equal in memory ability? No. It gave every student the same chance of each sheet, so any difference in ability between the groups is due to chance, and the reshuffling shows how big such chance differences get.
- You did not know how the two versions differed while you studied. Why does that help?
- What does one dot on the reshuffled plot represent? One random way of splitting the class's actual scores into two groups, summarized by its difference in means.
- To whom do our results apply? What would we need to do to apply them to all students at the school?
- How would you redesign this so each student is compared with themselves? Each student would study one grouped list and one scrambled list of different words, with a coin flip deciding which comes first.
Extension
Compare with a two-sample t test. Enter each group's mean, standard deviation and size into the two-sample t test calculator and compare its one-sided p-value with the fraction from the reshuffling. Before trusting the t result, check each group's dotplot for strong skew or outliers, since the groups are small. The two p-values will usually be in the same neighborhood, and seeing that gives students a reason to believe the formula.
Change the response. Score the same sheets a second way, such as the number of words recalled from the first two rows only, and ask whether the conclusion changes. This shows that the response variable is a design choice made before the data are seen.
Read more. The logic of steps 9 and 10 is laid out in what a randomization test is, and the design vocabulary in how to design an experiment.
Example analysis by reshuffling
Example data, not from a real class, kept small so every possible reshuffle can be counted. Six students with the grouped sheet recalled 14, 11, 16, 12, 15 and 13 words. Six students with the scrambled sheet recalled 10, 12, 9, 13, 11 and 8 words. Find the difference in means and the exact fraction of reshuffles that give a difference at least as large.
Grouped mean: words.
Scrambled mean: words.
Observed difference: words.
There are ways to deal the 12 scores into a first pile of 6 and a second pile of 6. If the arrangement has no effect, each is equally likely.
All 12 scores total 144. A first pile with total gives a difference of , so the difference is at least 3.0 exactly when .
Counting every deal: 2 give a first-pile total of 83, 4 give 82, and 10 give 81. That is 16 deals with a difference of 3.0 or more.
Fraction: .
The grouped sheet's mean is 3.0 words higher, and only 16 of the 924 possible deals, about 1.7%, produce a difference that large. Chance assignment alone rarely does this, so these example data give convincing evidence that the grouped arrangement increased recall for these 12 students. A two-sample t test on the same example gives t of about 2.78 with 10 degrees of freedom and a one-sided p-value of about 0.010.
How balanced does random assignment make the groups?
Thirty students are randomly assigned, 15 to each sheet. Twelve of the 30 slept badly last night. Find the probability that exactly 6 of them land in each group, and the probability that one group gets 8 or more.
Count all assignments: ways to choose the 15 students who get version A.
Count assignments with exactly 6 of the 12 in group A: choose 6 of the 12, then 9 of the other 18. That is .
Probability of an exact 6 and 6 split: .
The same count for 5 or 7 in group A gives probability 0.223 each, so a split of 6 and 6 or 7 and 5 has probability .
Everything else is a split of 8 and 4 or more lopsided: .
An exact even split happens 29% of the time, and about 26% of the time one group gets at least 8 of the 12 tired students. Random assignment does not make the groups alike. It makes the imbalance a matter of chance, which the analysis can then account for.
Frequently asked questions
What if the difference comes out near zero, or the scrambled group wins?
Run the reshuffling anyway. A real difference that sits in the middle of the reshuffled differences is a correct result: the data are consistent with no effect. A class that fails to find an effect has still run and analyzed a sound experiment, and can discuss whether 15 students per group was enough to detect a modest difference.
My class has an odd number of students. Does that break the design?
No. Use one extra sheet of either version and shuffle as usual. When reshuffling, deal the slips into piles of the same two sizes as the real groups.
How many reshuffles does the class need?
Seven groups doing 5 each gives 35, enough to see whether the real difference is typical or far in the tail. The estimated p-value moves in steps of 1 over the number of reshuffles, so 35 reshuffles cannot report anything between 0 and about 0.03. More reshuffles give a finer estimate.
Is this a completely randomized design or matched pairs?
Completely randomized. Every student receives one treatment, and a single shuffle assigns all the students at once with no pairing or blocking. The last discussion question asks students to turn it into a matched pairs design.
Can I skip the printing?
The two versions are 20 short words each. Writing each version once by hand and photocopying it onto half sheets takes a few minutes. The sheets must be handed out by shuffling, because reading a list aloud or projecting it gives everyone the same treatment.