Random sampling classroom activity: judgment vs SRS

By Jude Wallis · Published

Students estimate the mean word length of a 100-word passage twice: first from 5 words they choose as typical, then from a simple random sample of 5. The true mean is exactly 4 letters. Random-sample means center on 4, and the class sees whether its judgment samples run high.

AP Statistics: Unit 1 (topics 1.11 Random Sampling, 1.12 Potential Problems with Sampling). Students carry out a simple random sample with a chance device (topic 1.11) and compare it with a judgment sample to see how a selection method can produce bias (topic 1.12).

What the activity shows

Selection bias is a property of a sampling method: the method tends to miss in one direction. One sample cannot show that, but thirty samples drawn the same way can. In this activity every student samples the same small population twice, once by judgment and once by chance, and the class compares two dotplots of sample means against a population mean the teacher already knows.

The population is the 100 words in the table below. Its mean word length is exactly 4 letters. More than half the words have three letters or fewer, and short words such as a, of and is are easy to pass over when a person is asked to pick typical words, so judgment samples commonly come out too long. Random samples have no such tilt.

This is the same design as the well-known version with 100 rectangles of different areas. Words are used here because the whole population prints from this page with no drawing.

Materials and timing

  • One printed copy of the word table per student or pair, or the table projected.
  • A source of random two-digit numbers from 00 to 99. A calculator command such as randInt(0,99) works. So does a deck of cards with the face cards removed: count ace as 1 and ten as 0, draw a card for the first digit, put it back, shuffle, and draw again for the second digit.
  • Two number lines on the board, one directly above the other, both marked from 1 to 9 letters in steps of 0.2. Label the top one judgment and the bottom one random.
  • Sticky notes or markers for the dots.

Plan on 40 minutes: 5 for the judgment sample, 5 to plot it, 10 for the random sample, 5 to plot it, and 15 to compare and discuss. Do not announce the population mean until both dotplots are complete.

The population: 100 words

Read the table left to right, top to bottom, and it is one passage. Each word has a two-digit label: the row number, then the column number. The word in row 1, column 5 is word 15, students. The word in row 0, column 0 is word 00, every.

Row0123456789
0everymorningthebusstopsatthecornernearour
1schoolandacrowdofstudentsclimbsonwithheavy
2backpackssomeofusreadsomesleepandafew
3finishhomeworkthatwasdueyesterdaythedriverisa
4quietmanwhoknowseverybodybynamehewaitsfor
5thekidwhoisalwayslateandhenevercomplains
6aboutitonrainydaysthewindowsfogupimmediately
7andwedrawontheglassiliketheride
8becauseitistheonlypartofmydaywhen
9nobodyisaskingmetodoanythingextraordinaryatall

Facts for the teacher, computed from all 100 words:

  • Total letters: 400. Mean word length: exactly 4.00 letters. Median: 3 letters.
  • Population standard deviation: 2.25 letters.
  • Word counts by length: 4 words of 1 letter, 24 of 2, 25 of 3, 15 of 4, 12 of 5, 8 of 6, 3 of 7, 3 of 8, 4 of 9, 1 of 11 and 1 of 13.
  • 53 of the 100 words have three letters or fewer. Only 12 have seven letters or more.

Steps

  1. Hand out the table face down. Tell students they will have 20 seconds to look at a passage and circle 5 words that they judge to be typical of the words in it. Give no other guidance.
  2. Turn the pages over, time 20 seconds, and stop.
  3. Each student counts the letters in the 5 circled words, computes the mean, and places a dot at that value on the judgment number line.
  4. Explain the labels: row digit, then column digit, from 00 to 99.
  5. Each student generates two-digit random numbers until there are 5 different labels, skipping any repeat, and writes down the 5 words those labels point to.
  6. Each student computes the mean length of the 5 random words and places a dot on the random number line.
  7. Ask the class to describe the center and spread of each dotplot before knowing the truth.
  8. Announce the population mean, 4 letters, and draw one vertical line at 4 through both dotplots.
  9. Compute the average of the dots on each line and write both numbers next to the plots.

What students should see

The random dotplot. A simple random sample of 5 gives every word the same chance, 5/1005/100, of being chosen, whatever its length. The sample mean xˉ\bar{x} ("x-bar") is an unbiased estimator: averaged over all possible samples it equals the population mean, 4.00. Individual samples still vary. The standard deviation of xˉ\bar{x} for samples of 5 is about 0.99 letters, so dots from about 2 to 6 are ordinary. In a simulation of 400,000 random samples, about 74% of the sample means landed within 1 letter of 4, and about 42% landed above 4.

With 30 students, the average of the 30 random-sample means has a standard deviation of about 0.18, so it should fall between roughly 3.6 and 4.4.

The judgment dotplot. This guide cannot give a theoretical value for it, because it depends on what your students choose. The usual result is a dotplot centered above 4, often well above, with few or no dots below 4. If your class shows that, the vertical line at 4 will cut through the middle of the random dots and sit at the left edge of the judgment dots.

The comparison that matters. Some random dots will be farther from 4 than some judgment dots. That is variability. Bias is about where the whole dotplot is centered. A method can be unbiased and still give a poor estimate in one sample, and it can be biased and still land close once.

Discussion questions

  1. Which dotplot is centered closer to 4? What does that say about each method, as opposed to each sample?
  2. Find a random-sample dot that is far from 4. Does that dot show the random method is biased? No. One sample shows variability. Bias is a tendency across many samples.
  3. Which words did almost nobody circle in the judgment round? Students usually name the one-letter and two-letter words. Those 28 words are more than a quarter of the population.
  4. Suppose everyone had circled 10 typical words. Would the judgment dotplot move toward 4? A larger sample reduces spread. It does not remove a tilt in how the words are chosen.
  5. In the random round, what gave a word like a the same chance as a word like extraordinary? The label was chosen by a chance device that knows nothing about the word.
  6. Name a survey where the person choosing the sample could tilt it without meaning to, the way the eye tilts toward long words.

Extension

Double the sample size. Each student draws a simple random sample of 10 words and plots the mean on a third number line. The center stays at 4. The standard deviation of xˉ\bar{x} drops from about 0.99 to about 0.68, so the dots pull in toward the line. Students see that sample size controls spread while the selection method controls center.

A sampling method with bias built in. Suppose a word's chance of being chosen were proportional to its number of letters, so a 6-letter word was twice as likely as a 3-letter word. The expected length of a chosen word would be ∑L2/∑L=2106/400≈5.27\sum L^2 / \sum L = 2106/400 \approx 5.27 letters, well above the true mean of 4. Ask students what physical method would behave roughly like this. Pointing at a printed paragraph with eyes closed is one answer, since long words cover more of the page.

Connect to sampling distributions. The random dotplot is a small sampling distribution of a sample mean. The sampling distribution and CLT visualizer draws thousands of sample means from a population and shows the same two facts on screen: centered on the population mean, narrower as the sample size grows. For the vocabulary of what went wrong in the judgment round, see what selection bias is.

Example samples: one by judgment, one at random

Example data, not from a real class. A student circles morning, students, homework, driver and windows as typical words. The same student then generates the random labels 12, 45, 67, 80 and 29. Find both sample means and compare them with the population mean of 4.

  1. Judgment sample lengths: morning 7, students 8, homework 8, driver 6, windows 7. The mean is (7+8+8+6+7)/5=36/5=7.2(7 + 8 + 8 + 6 + 7)/5 = 36/5 = 7.2 letters.

  2. Random labels to words: 12 is row 1, column 2, the word a. 45 is by. 67 is fog. 80 is because. 29 is few.

  3. Random sample lengths: 1, 2, 3, 7, 3. The mean is (1+2+3+7+3)/5=16/5=3.2(1 + 2 + 3 + 7 + 3)/5 = 16/5 = 3.2 letters.

  4. Errors: the judgment sample is 7.2−4=3.27.2 - 4 = 3.2 letters too high. The random sample is 3.2−4=−0.83.2 - 4 = -0.8, which is 0.8 letters too low.

  5. Four of the five random words have three letters or fewer. Such words are 53% of the population, and the judgment sample contains none of them.

The judgment sample mean is 7.2 letters and the random sample mean is 3.2 letters, against a true mean of 4. One pair of samples does not establish bias. The class dotplots do that by showing where each method centers.

The spread of the random-sample means

The population of 100 word lengths has standard deviation 2.25 letters. Find the standard deviation of the sample mean for simple random samples of 5 words, and check whether the usual formula is close enough here.

  1. The usual formula gives σ/n=2.249/5≈1.006\sigma/\sqrt{n} = 2.249/\sqrt{5} \approx 1.006 letters.

  2. That formula assumes independent draws. Sampling 5 words without replacement from only 100 makes the draws slightly dependent. The sample is 5% of the population, under the 10% guideline, so the formula should still be close.

  3. The exact value for sampling without replacement multiplies by (100−5)/(100−1)≈0.9796\sqrt{(100 - 5)/(100 - 1)} \approx 0.9796, giving 1.0060(0.9796)≈0.9851.0060(0.9796) \approx 0.985 letters.

  4. The two answers, 1.006 and 0.985, differ by about 2%, which no classroom dotplot could detect.

  5. For samples of 10 the exact value is (2.249/10)90/99≈0.678(2.249/\sqrt{10})\sqrt{90/99} \approx 0.678 letters.

The sample mean for random samples of 5 words has a standard deviation of about 0.99 letters, and about 0.68 letters for samples of 10. Random dots between about 2 and 6 letters are ordinary for samples of 5.

Frequently asked questions

What if my class's judgment samples center on 4?

Report what happened. The activity still makes its point if you ask why a careful person might avoid the tilt and whether anyone could know that without the population mean in hand. A random sample needs no such luck or skill, and its lack of bias can be shown without looking at the data.

Do repeated random numbers count twice?

No. Skip a repeat and generate another number. A simple random sample of 5 is 5 different words, chosen so that every group of 5 words is equally likely.

Is the 100-word table a sample?

No. For this activity it is the entire population, so its mean of 4 letters is a parameter. Each student's mean of 5 words is a statistic. Keeping those two words attached to the right numbers is a useful side lesson.

Can I use rectangles of different areas instead?

Yes. The design is identical: a sheet of 100 numbered rectangles, a judgment sample of 5, a random sample of 5, and two dotplots of mean area. You need the true mean area of your own sheet. This guide uses words so that the population and its exact mean come ready to print.

Why samples of only 5 words?

Five keeps the arithmetic to one minute and leaves plenty of visible spread in the random dotplot, which is what lets students separate variability from bias. The extension with samples of 10 shows the spread shrinking.