What makes a statistical procedure robust?

Q: Are t procedures robust to outliers?

Much less than they are to skew. A t interval is built from x-bar and s, and neither is resistant, so one extreme value shifts the centre and inflates the width at once. The AP shape condition treats them separately for that reason: at n under 30 it asks for no strong skew and no outliers, and a single clear outlier is enough to fail it.

Q: Is the one-proportion z test robust?

Less robust than the t procedures. It replaces a discrete count with a continuous normal curve, and that approximation degrades quickly when a category is rare, which is what the Large Counts condition guards: at least 10 successes and 10 failures, counted as expected under p₀ for a test and as observed in the sample for an interval. When it fails, the actual Type I error rate can sit well away from your α.

By Jude Wallis · Published July 31, 2026

A procedure is robust when violating one of its conditions barely changes the answer. The $t$ procedures are robust to mild non-normality, and more so as $n$ grows, though strong skew at a small $n$ still costs real coverage. Nothing is robust to a biased sample, at any size.

AP Statistics: Unit 4 (topics 4.2 Constructing a Confidence Interval for a Population Mean or Population Mean Difference, 4.4 Setting Up a Test for a Population Mean or Population Mean Difference). Robustness is not a named topic in the Fall 2026 AP Statistics course. It is the reason the conditions are worded as they are, and it is assessed inside the setting-up topics for every procedure: 3.5 and 3.12 for proportions, 3.14 for chi-square, and 4.4 and 4.9 for means.

Robust means the conclusion survives a broken condition

Every inference procedure comes with conditions, and real data never satisfy them exactly. No population is precisely normal. No sample is a perfect random draw. If a broken condition automatically invalidated the answer, statistics would be unusable.

So the useful question is not whether a condition holds but how much it matters when it fails. Robustness is the answer to that question. A procedure is robust against a particular violation if the results stay close to correct when the violation is mild. It is not robust if a small departure produces a badly wrong answer.

Two things follow immediately, and both are ways people go wrong. Robustness is always robustness against something specific, so "the $t$ test is robust" is not a complete sentence: robust against mild non-normality, yes; against a voluntary response sample, not at all. And robustness is a property of the procedure, not of your data. Your data cannot be robust. The procedure can be robust to what your data do.

How you measure robustness: coverage

Robustness sounds like a judgment call, but for a confidence interval it can be measured exactly.

A 95% confidence interval makes a promise about the long run: across repeated random samples from the population, 95% of the intervals you build will contain the parameter. That percentage is called coverage, and it is the thing to check. Break a condition, generate a very large number of samples, build the interval each time, and count how often it actually captured the truth.

If coverage stays near 95%, the procedure is robust to that violation and your stated confidence level is close to honest. If coverage drops to 88%, it is not: you are saying 95% and delivering 88%, and one interval in eight misses instead of one in twenty. You can watch coverage accumulate sample by sample in the confidence interval coverage simulator.

The same idea applies to a hypothesis test, where the number to watch is the Type I error rate. A test is robust if the actual chance of a false alarm stays near $\alpha$ when a condition is violated.

The demonstration: t intervals on a badly skewed population

Here is the check, run on 200,000 simulated samples for each row. The population is an exponential distribution, which is strongly right skewed with a long tail, about as far from normal as anything you are likely to meet. Each sample gets a nominal 95% one-sample $t$ interval, $\bar{x} \pm t^* \frac{s}{\sqrt{n}}$ , and the table reports how often that interval actually contained the true population mean.

$n$	actual coverage of a nominal 95% interval
5	88.3%
10	90.0%
30	92.7%
100	94.2%
500	94.8%

Two things to take from it. The interval is too optimistic at every sample size, so strong skew does cost you something real. And the cost shrinks steadily as $n$ grows, which is the central limit theorem at work: the sampling distribution of $\bar{x}$ straightens out as the sample grows even when the population never does.

The comparisons make it sharper. On a normal population the same intervals cover 95.0% at $n = 5$ , 10 and 30, which is the promise kept. On a uniform population, non-normal but symmetric, coverage is 93.4% at $n = 5$ and 94.9% at $n = 30$ . On a moderately skewed population, a gamma distribution with skewness 1.0 against the exponential's 2.0, coverage is 93.2% at $n = 5$ and 94.4% at $n = 30$ . Symmetry is cheap to violate. Skew is what costs, and small samples are where it costs most, which is exactly why the AP condition for means gets stricter as $n$ falls.

Which AP procedures tolerate what

Procedure	Condition	Robust to a mild violation?
One-sample and paired $t$	population approximately normal	Yes, increasingly so as $n$ grows. Strong skew or an outlier at small $n$ is the case to avoid
Two-sample $t$	both populations approximately normal	Yes, and most reliably when the two sample sizes are similar
One-proportion and two-proportion $z$	Large Counts: at least 10 successes and 10 failures, expected under $p_0$ for a test and observed in the sample for an interval	Less so. This is a normal curve standing in for a discrete count, and it degrades quickly once a category is rare
Chi-square for homogeneity or independence	every expected count at least 5	Less so. A small expected count sits in a denominator and inflates the statistic
All of them	10% condition when sampling without replacement	Yes, and in your favour. Breaking it makes the standard error formula too large, so the interval comes out too wide rather than wrong
All of them	Random sample or random assignment	No. Not mildly, not at any sample size

The pattern behind the first two rows is worth naming. The $t$ procedures tolerate non-normality because they depend on the shape of the sampling distribution of $\bar{x}$ , not on the shape of the population, and averaging pulls that sampling distribution toward normal on its own. The proportion and chi-square rows are less forgiving because their approximations are about counts in categories, and no amount of averaging fixes a category that almost never occurs. The full condition list, with the exact wording to write down, is in the conditions for inference checklist.

The violation nothing survives: a biased sample

Now break only the sampling. Keep a population that is exactly normal, so the shape condition is perfect, and use a method that systematically never reaches the bottom 20% of the population: a volunteer panel, a survey of people who answer the phone, a study of patients who came back for the follow-up visit. The formula is correct, the arithmetic is correct, the data look fine. Here is the coverage of the true population mean:

$n$	coverage of the true mean
10	80.2%
30	30.3%
100	0.2%
1,000	0.0%

Coverage gets worse as the sample grows, and that is not a paradox once you see the mechanism. The method is centred on the wrong number: the group it can actually reach has a mean 0.35 standard deviations above the population mean. A larger sample estimates that wrong number more precisely, so the interval tightens around the wrong place and slides off the right one. Precision is not accuracy, and a narrow interval around a biased estimate is worse than a wide one, because it is confidently wrong.

This is the same lesson as does a bigger sample fix bias, stated as a coverage number. Bias lives in the method, so $n$ has no grip on it. Write the sentence down in this form: no statistical procedure is robust against a non-random sample, and there is no sample size that repairs one.

What robustness does not license

It is not permission to skip the condition check. On an AP free response, stating and verifying the conditions is scored on its own, whether or not the procedure would have survived. Robustness tells you how much to worry, not whether to write.
It is not general toughness. Each robustness claim is about one procedure against one violation. A $t$ interval that shrugs off mild skew is destroyed by a badly chosen sample.
It does not cover outliers the way it covers skew. A $t$ interval is built from $\bar{x}$ and $s$ , and neither is resistant, so one wild value moves the centre and inflates the width at the same time. At $n = 20$ , mild skew is fine and a clear outlier is not. See t-test with a small sample.
Robust and resistant are close cousins, not the same claim. Resistant describes a statistic: how much its value moves when a few observations are extreme, which is why the median is resistant and the mean is not. Robust describes a procedure: whether it still performs as advertised when one of its conditions is not quite met. See resistant statistic. That is the wider convention, but the AP course does not follow it: topic 1.7 writes "a resistant (or robust) measure of center" and calls the mean "nonresistant (or non-robust)", treating the two words as one label. So expect them interchangeable on the exam and distinct in a textbook, and read each from what its sentence is about.
It does not make a small sample big. Robustness is about whether the stated confidence level is honest, not about whether the interval is useful. A trustworthy interval that runs from $-3.7$ to $4.5$ has settled nothing, which is the other half of statistical versus practical significance.

How to write this on the AP exam

Condition checks are scored on evidence, not on vocabulary. Three habits cover almost every case.

Name how the data were collected, quoting the problem. "The 40 households were selected at random from the town directory" earns the Random condition. "The teacher used her own class" does not, and the correct move then is to say so and to limit the conclusion, not to run the procedure anyway and hope.

For a mean, satisfy the shape condition by one of its three routes: the population is stated to be approximately normal, or $n \geq 30$ , or a graph of the sample shows no strong skew and no outliers. Say which route you took, and if you took the graph route, describe what the graph shows.

When a condition is genuinely shaky, say what it costs. A sentence like "the sample is only 12 observations and the dotplot shows a clear high outlier, so the $t$ interval may not have its stated 95% coverage" is a better answer than pretending the condition holds. Practise the wording on the conditions checking set.

Mild skew at n = 20: does the t interval survive?

A random sample of 20 delivery times, in minutes: 12, 14, 15, 15, 16, 16, 17, 17, 18, 18, 18, 19, 19, 20, 21, 22, 23, 24, 25, 27. Decide whether a one-sample $t$ interval is trustworthy here, then build the 95% interval for the mean delivery time.

Check Random: the problem states a random sample, so that condition is met. Nothing later in the analysis can substitute for this one.
Check the shape condition. Here $n = 20$ , which is under 30, and the population is not stated to be normal, so the only route left is the graph of the sample.
Find the five-number summary (median-excluded quartiles, the TI-84 convention). Sorted, the lower half is 12, 14, 15, 15, 16, 16, 17, 17, 18, 18 with median $\frac{16+16}{2} = 16$ , so $Q_1 = 16$ . The upper half is 18, 19, 19, 20, 21, 22, 23, 24, 25, 27 with median $\frac{21+22}{2} = 21.5$ , so $Q_3 = 21.5$ . The overall median is $\frac{18+18}{2} = 18$ , the minimum is 12 and the maximum is 27.
Apply the 1.5 IQR rule: $IQR = 21.5 - 16 = 5.5$ , so the fences are $16 - 1.5(5.5) = 7.75$ and $21.5 + 1.5(5.5) = 29.75$ . Every value lies inside, so there are no outliers.
Take the graph route properly: plot the 20 values as a dotplot. It has one peak, at 18, with 12 of the 20 times packed between 15 and 20, a short left tail down to 12 and a slightly longer right tail out to 27, and no point cut off from the rest by a wide gap. That is mild right skew, not strong skew, and the numbers agree: the mean is $\frac{376}{20} = 18.8$ against a median of 18, a gap of 0.8 minutes on a range of 15. The shape condition is satisfied by the graph route.
Compute the interval. $s = 3.901417$ , so the standard error is $\frac{3.901417}{\sqrt{20}} = 0.8724$ .
Critical value: $t^* = 2.093$ for 95% confidence with $df = 19$ .
Margin of error: $2.093 \times 0.8724 = 1.8259$ .
Interval: $18.8 \pm 1.8259$ , which is 16.974 to 20.626 minutes.
Contrast: replace the two largest values 25 and 27 with 31 and 38, and the fences stay at 7.75 and 29.75 while both 31 and 38 fall outside them. Two flagged outliers in a sample of 20 is exactly the situation the $t$ procedure is not robust to, and the honest answer there is to say the condition fails.

The interval is trustworthy: mild right skew and no outliers at $n = 20$ is well inside what the $t$ procedure tolerates. The 95% confidence interval for the mean delivery time is 16.974 to 20.626 minutes. Write the justification in words as well: the sample is random, the dotplot has a single peak with only a mild right tail and no separated point, and the 1.5 IQR rule flags no outliers.

A tenfold sample that fixes nothing

A website posts an opinion poll. Of the 2,000 readers who choose to answer, 72% say yes. Build the 95% confidence interval. Then suppose 20,000 readers answer and the sample proportion is still 0.72. What changes, and what does not?

At $n = 2000$ with $\hat{p} = 0.72$ , the standard error is $\sqrt{\frac{0.72(0.28)}{2000}} = \sqrt{0.0001008} = 0.010040$ .
Margin of error: $1.96 \times 0.010040 = 0.019678$ , so the interval is $0.72 \pm 0.019678$ , or 0.7003 to 0.7397.
At $n = 20000$ : $\sqrt{\frac{0.72(0.28)}{20000}} = \sqrt{0.00001008} = 0.003175$ .
Margin of error: $1.96 \times 0.003175 = 0.006223$ , so the interval is 0.7138 to 0.7262.
Measure what improved: the margin fell from 0.019678 to 0.006223, a factor of $\sqrt{10} \approx 3.162$ , exactly as the $1/\sqrt{n}$ relationship predicts.
Now check the Random condition. Readers chose to answer, which is a voluntary response sample, so people with strong opinions are over-represented by the design.
Identify what $\hat{p}$ actually estimates: the proportion saying yes among readers who choose to answer polls, not the proportion among readers. Those are different populations, and the gap between them does not shrink with $n$ .
Conclude that no confidence level attaches to either interval, because the condition that the procedure is not robust to is the one that failed.

The intervals are 0.7003 to 0.7397 at $n = 2000$ and 0.7138 to 0.7262 at $n = 20000$ , narrower by a factor of about 3.162. Neither is a 95% confidence interval for the readership. The Random condition fails and no procedure is robust to that, so the tenfold sample bought precision about the wrong quantity. The fix is a random sample of readers, at any size.

Frequently asked questions

What does robust mean in statistics?

A procedure is robust against a specified violation if its results stay close to correct when that condition fails mildly. For a confidence interval, the test is coverage: if a nominal 95% interval still captures the parameter about 95% of the time under the violation, the procedure is robust to it. Robustness is always against something specific, never a general toughness.

Are t procedures robust to outliers?

Much less than they are to skew. A $t$ interval is built from $\bar{x}$ and $s$ , and neither is resistant, so one extreme value shifts the centre and inflates the width at once. The AP shape condition treats them separately for that reason: at $n$ under 30 it asks for no strong skew and no outliers, and a single clear outlier is enough to fail it.

Is the one-proportion z test robust?

Less robust than the $t$ procedures. It replaces a discrete count with a continuous normal curve, and that approximation degrades quickly when a category is rare, which is what the Large Counts condition guards: at least 10 successes and 10 failures, counted as expected under $p_0$ for a test and as observed in the sample for an interval. When it fails, the actual Type I error rate can sit well away from your $\alpha$ .

If a procedure is robust, can I skip the condition check?

No. On the AP exam, stating and verifying conditions is scored separately from the calculation, so skipping it loses points even when the procedure would have survived. Robustness tells you how much a violation matters; it never tells you not to look.

Is any procedure robust to a biased sample?

None. Bias moves the centre of the sampling distribution rather than its shape, so the whole interval is aimed at the wrong number. Simulation makes the point brutally: with a method that misses the bottom 20% of a population, coverage of a nominal 95% interval falls from about 80% at $n = 10$ to near 0% at $n = 100$ . A bigger sample makes it worse, not better.