Statistical vs practical significance explained

Q: Can a result be statistically significant but not important?

Yes, and it is common in studies with large samples. A test asks whether the data would be surprising under the null hypothesis, and a large sample makes small departures surprising. A 1 percentage point difference is not significant at n = 1000 and is significant at n = 10,000, with the difference itself unchanged. Report the confidence interval so a reader can see the size.

Q: Can a large effect fail to be statistically significant?

Yes. A small sample gives a large standard error, so even a big difference can produce a small test statistic. That is a Type II error risk, not evidence of no effect. When you fail to reject, build the interval: if it is wide, the honest conclusion is that the study was not able to settle the question.

Q: Does a smaller p-value mean a bigger effect?

No. A p-value depends on the effect size and the sample size together, so it cannot separate them. A p-value of 0.0001 from 100,000 observations can describe a smaller effect than a p-value of 0.04 from 50 observations. To compare sizes, compare the estimates and their intervals, not the p-values.

Q: How do you judge practical significance?

By the context, never by the arithmetic. Ask what the smallest effect inside your confidence interval would be worth to the person making the decision: what it costs, what it saves, what it risks. In medicine that judgment is called clinical significance and is set by clinicians, not by the statistics.

Q: Does AP Statistics test practical significance?

Not as a topic of its own. The Fall 2026 course tests it through interpretation: p-values in topic 3.6, and justifying a claim from a confidence interval in topics 3.4 and 4.3. Conclusions are scored on saying there is or is not convincing statistical evidence, never on calling an effect large or important.

By Jude Wallis · Published July 31, 2026

Statistical significance means the result would be surprising if the null hypothesis were true. Practical significance means the effect is big enough to matter. Sample size separates them: with $n$ large enough, a test will almost certainly detect any nonzero effect, however small it is.

AP Statistics: Unit 3 (topics 3.6 p-Values, 3.4 Justifying a Claim Based on a Confidence Interval for a Population Proportion). The Fall 2026 AP Statistics course has no topic named practical significance. The distinction is assessed inside topic 3.6 on p-values and inside the justify-a-claim topics, 3.4 for proportions and 4.3 for means, where you read an interval and say what it does and does not support.

The difference, one sentence each

Statistical significance is a statement about surprise. A result is statistically significant when its p-value falls at or below the significance level $\alpha$ ("alpha") you fixed before collecting data, which means data at least this extreme would be unlikely if the null hypothesis were true.

Practical significance is a statement about size. An effect is practically significant when it is large enough that someone should act differently because of it. No part of a hypothesis test can decide that for you. It is a judgment about context: a 2-point gain on a 100-point test, a 0.4 mmHg drop in blood pressure, a 1 percentage point edge in a poll.

The two come apart because a p-value is not a measure of size. It answers one question, would this result be surprising if the null hypothesis were true, and the surprise depends on two things: how big the effect is and how much data you collected. Freeze the effect and pour in data, and the surprise grows on its own.

Why sample size decides statistical significance

Take the simplest case, a two-sided one-proportion z-test of $H_0: p = 0.50$ . The standardized test statistic is

z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1 - p_0)}{n}}}

where $\hat{p}$ ("p-hat") is the sample proportion and $p_0$ ("p-naught") is the value the null hypothesis claims. Now freeze the effect at one percentage point: $\hat{p} = 0.51$ in every sample, no matter how big the sample is.

The numerator is stuck at $0.51 - 0.50 = 0.01$ forever. The denominator is $\sqrt{0.25/n}$ , which equals $0.5/\sqrt{n}$ , so the whole statistic collapses to

z = \frac{0.01}{0.5/\sqrt{n}} = 0.02\sqrt{n}

Everything about the data is held fixed except $n$ , and $z$ still grows without limit. It crosses the two-sided 0.05 cutoff of 1.96 when $0.02\sqrt{n} = 1.96$ , that is when $\sqrt{n} = 98$ and $n = 9604$ . Below that sample size, one percentage point is not significant. Above it, the same one percentage point is. Nothing about the effect changed. Only the amount of data did.

The standard error is the mechanism. It shrinks like $1/\sqrt{n}$ , and the test statistic is the fixed gap measured in standard errors, so shrinking the ruler makes the same gap read larger.

The demonstration: one percentage point at eight sample sizes

Every row below is the same two-sided test of $H_0: p = 0.50$ with the identical sample proportion, $\hat{p} = 0.51$ . Only $n$ changes.

$n$	successes	standard error	$z$	p-value	significant at 0.05?
100	51	0.0500	0.20	0.8415	no
500	255	0.0224	0.45	0.6547	no
1,000	510	0.0158	0.63	0.5271	no
2,500	1,275	0.0100	1.00	0.3173	no
5,000	2,550	0.0071	1.41	0.1573	no
10,000	5,100	0.0050	2.00	0.0455	yes
25,000	12,750	0.0032	3.16	0.0016	yes
100,000	51,000	0.0016	6.32	under 0.0001	yes

Read down the p-value column. The evidence walks from "nothing at all" to "overwhelming" while the thing being measured, a 51 to 49 split, never moves an inch. The crossing happens at $n = 9604$ , between the fifth and sixth rows.

The last row is the one worth sitting with. At $n = 100{,}000$ the p-value is smaller than 0.0001, the kind of number people describe as highly significant, and the finding is still that 51 people out of 100 clicked instead of 50. A p-value that small, off a sample that large, is telling you about the amount of data at least as much as about the size of the effect. That is why "how good is my p-value" is the wrong question to ask of it.

The same trap with a mean

Nothing here is special to proportions. Suppose a review program raises the mean score by 0.4 points on a 100-point test, with a sample standard deviation of $s = 10$ points. As a standardized effect that is $0.4 / 10 = 0.04$ standard deviations, which is almost nothing. The one-sample t statistic is

t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} = \frac{0.4}{10/\sqrt{n}}

$n$	standard error	$t$	$df$	two-sided p-value
25	2.0000	0.20	24	0.8432
100	1.0000	0.40	99	0.6900
400	0.5000	0.80	399	0.4242
2,500	0.2000	2.00	2,499	0.0456
10,000	0.1000	4.00	9,999	0.00006

The crossing sits at $n = 2404$ , the smallest sample size for which this 0.4-point gain has a two-sided p-value below 0.05. A district that tests 2,500 students gets a significant result. A district that tests 400 students, measuring the exact same program with the exact same effect, does not.

What to report instead of a verdict

The fix is not a better p-value. It is to report the size of the effect and how precisely you pinned it down, which is exactly what a confidence interval does.

Go back to the proportion at $n = 10{,}000$ , where the test rejected $H_0$ with $p = 0.0455$ . The 95% confidence interval for $p$ uses $\hat{p}$ in the standard error, giving $\sqrt{0.51(0.49)/10{,}000} = 0.004999$ and a margin of error of $1.96 \times 0.004999 = 0.0098$ :

0.51 \pm 0.0098 \rightarrow (0.5002,\ 0.5198)

Read the interval and the argument ends. The true proportion is somewhere between 0.02 and 1.98 percentage points above half. The bottom of the plausible range is indistinguishable from a coin flip. The test told you the effect is probably not exactly zero; the interval tells you it might be almost exactly zero, and that is the information a decision needs.

Do the same for the mean. At $n = 2500$ the 95% interval for the mean gain is $0.4 \pm 1.9609(0.2)$ , which is 0.008 to 0.792 points. The entire plausible range sits below eight tenths of one point on a 100-point test. So report three things every time: the estimate, the interval, and the units it is measured in. Then say in plain words whether a difference that size would change anyone's behaviour. More on reading intervals in how to interpret a confidence interval for a proportion.

The mirror image: not significant does not mean no effect

The trap runs the other way too, and it is the half students forget.

Test the same 0.4-point gain on 25 students. You get $t = 0.20$ , $p = 0.8432$ , and you fail to reject $H_0$ . It is tempting to write "the program does not work." Build the interval instead: $t^* = 2.0639$ with $df = 24$ and a standard error of 2.0 gives a margin of error of 4.1278, so the 95% interval runs from $-3.728$ to $4.528$ points. The data are consistent with a 3.7-point loss and equally consistent with a 4.5-point gain. They have settled nothing.

That is what low power looks like from the inside: a small study fails to detect a real effect and the failure is indistinguishable from the effect being absent. Failing to reject the null hypothesis is never evidence that the null hypothesis is true, and a wide interval is how you prove that to yourself. You can watch the relationship between effect size, sample size and detection rate directly in the power and errors explorer.

How this shows up on the AP exam

The Fall 2026 course has no topic called practical significance, so it is never a question by itself. It is tested through wording, in two places.

First, the conclusion of a test. The scored sentence is that there is or is not convincing statistical evidence that the alternative hypothesis is true. Words like large, important, meaningful, or substantial are not in the conclusion and can cost you the point, because the test never measured any of those. See how to write a hypothesis test conclusion for the exact templates.

Second, justifying a claim from a confidence interval, which is topic 3.4 for proportions and 4.3 for means. Those questions hand you an interval and ask what it supports. An interval that excludes the null value but hugs it describes an effect the data can separate from zero but that is too small to act on, and saying so in context is the answer they want.

A useful habit for both: after you write your conclusion, ask what the smallest effect inside your interval would mean for the person who has to make the decision. If your conclusion would read the same whether that number were 0.02 percentage points or 20, it is not finished.

The same click rate, two sample sizes, two conclusions

A website tests a checkout button. In both runs, 51% of visitors click. Test $H_0: p = 0.50$ against $H_a: p \neq 0.50$ at $\alpha = 0.05$ using 5,000 visitors (2,550 clicks), then again using 10,000 visitors (5,100 clicks). Compare the conclusions, then build the 95% confidence interval for the larger sample.

Confirm the effect is identical in both runs: $\hat{p} = 2550/5000 = 0.51$ and $\hat{p} = 5100/10000 = 0.51$ .
For a test, the standard error uses the null value $p_0 = 0.50$ , not $\hat{p}$ . At $n = 5000$ : $\sqrt{\frac{0.50(0.50)}{5000}} = \sqrt{0.00005} = 0.007071$ .
Standardize: $z = \frac{0.51 - 0.50}{0.007071} = 1.41$ .
Two-sided p-value: $2P(Z \geq 1.41) = 0.1573$ . Since $0.1573 > 0.05$ , fail to reject $H_0$ .
At $n = 10000$ : $\sqrt{\frac{0.50(0.50)}{10000}} = \sqrt{0.000025} = 0.005000$ .
Standardize: $z = \frac{0.51 - 0.50}{0.005} = 2.00$ , and the two-sided p-value is $2P(Z \geq 2.00) = 0.0455$ . Since $0.0455 < 0.05$ , reject $H_0$ .
Now the interval at $n = 10000$ . An interval has no null value, so the standard error uses $\hat{p}$ : $\sqrt{\frac{0.51(0.49)}{10000}} = 0.004999$ .
Margin of error: $1.96 \times 0.004999 = 0.009798$ .
Interval: $0.51 \pm 0.009798$ , which is 0.5002 to 0.5198.

At $n = 5000$ , $z = 1.41$ and $p = 0.1573$ , so you fail to reject. At $n = 10000$ , $z = 2.00$ and $p = 0.0455$ , so you reject. The click rate is 51% in both. The 95% interval at $n = 10000$ runs from 0.5002 to 0.5198, so the true edge over half could be as small as 0.02 of a percentage point, which is 2 extra clicks in every 10,000 visitors.

A statistically significant gain of 0.4 points

A review program is tested on 2,500 students. Their mean score gain is $\bar{x} = 0.4$ points on a 100-point test, with $s = 10$ points. Test $H_0: \mu = 0$ against $H_a: \mu \neq 0$ at $\alpha = 0.05$ , then build a 95% confidence interval for the mean gain and decide whether a school should buy the program.

Standard error of the sample mean: $\frac{s}{\sqrt{n}} = \frac{10}{\sqrt{2500}} = \frac{10}{50} = 0.2$ .
Test statistic: $t = \frac{0.4 - 0}{0.2} = 2.00$ with $df = n - 1 = 2499$ .
Two-sided p-value for $t = 2.00$ with $df = 2499$ : $p = 0.0456$ . Since $0.0456 < 0.05$ , reject $H_0$ . There is convincing statistical evidence that the mean gain is not zero.
Critical value for 95% confidence with $df = 2499$ : $t^* = 1.9609$ .
Margin of error: $1.9609 \times 0.2 = 0.3922$ .
Interval: $0.4 \pm 0.3922$ , which is 0.008 to 0.792 points.
Judge the size. The largest gain the data support is 0.792 of a point out of 100. As a standardized effect the estimate is $0.4/10 = 0.04$ standard deviations.

$t = 2.00$ with $df = 2499$ and $p = 0.0456$ , so the gain is statistically significant at $\alpha = 0.05$ . The 95% interval, 0.008 to 0.792 points, puts the entire plausible range below one point on a 100-point test. Statistically significant and practically worthless: the school should not buy it on this evidence.

Frequently asked questions

Can a result be statistically significant but not important?

Yes, and it is common in studies with large samples. A test asks whether the data would be surprising under the null hypothesis, and a large sample makes small departures surprising. A 1 percentage point difference is not significant at $n = 1000$ and is significant at $n = 10{,}000$ , with the difference itself unchanged. Report the confidence interval so a reader can see the size.

Can a large effect fail to be statistically significant?

Yes. A small sample gives a large standard error, so even a big difference can produce a small test statistic. That is a Type II error risk, not evidence of no effect. When you fail to reject, build the interval: if it is wide, the honest conclusion is that the study was not able to settle the question.

Does a smaller p-value mean a bigger effect?

No. A p-value depends on the effect size and the sample size together, so it cannot separate them. A p-value of 0.0001 from 100,000 observations can describe a smaller effect than a p-value of 0.04 from 50 observations. To compare sizes, compare the estimates and their intervals, not the p-values.

How do you judge practical significance?

By the context, never by the arithmetic. Ask what the smallest effect inside your confidence interval would be worth to the person making the decision: what it costs, what it saves, what it risks. In medicine that judgment is called clinical significance and is set by clinicians, not by the statistics.

Does AP Statistics test practical significance?

Not as a topic of its own. The Fall 2026 course tests it through interpretation: p-values in topic 3.6, and justifying a claim from a confidence interval in topics 3.4 and 4.3. Conclusions are scored on saying there is or is not convincing statistical evidence, never on calling an effect large or important.