Effect Size vs Statistical Significance

An effect size is a measure of magnitude. In its plainest form it is the raw difference the study estimated: 3 points, 0.4 mmHg, 5 percentage points. A standardized version divides that difference by a standard deviation, which strips the units so effects measured on different scales can be lined up against each other. Either form answers how much. A p-value answers a different question and cannot be converted into this one.

Suppose a review program raises a mean score by 0.4 points on a 100-point test, with $s = 10$ points. The raw effect is 0.4 points and the standardized effect is $0.4/10 = 0.04$ standard deviations. Now hold both of those fixed and change only the sample size. At $n = 400$ the one-sample $t$ statistic is $0.4/(10/\sqrt{400}) = 0.80$, with a two-sided p-value of 0.424. At $n = 10{,}000$ the same 0.4 points gives $t = 4.00$ and a p-value of 0.00006. The effect size is 0.04 in both rows.

That is why "the p-value was smaller, so the effect is bigger" cannot be rescued. Those two p-values, 0.424 and 0.00006, describe the identical 0.4-point gain. A p-value blends the size of the effect with the amount of data into a single number, and nothing unblends it, so p-values can never rank two findings by size. Compare the estimates and their intervals instead.

An effect size is a size, and only that. Whether 0.4 points is worth acting on is practical significance, a judgment about context that no formula produces. How firmly the study pinned 0.4 down is the confidence interval. A standardized effect also hides its units, so 0.04 standard deviations means nothing until someone says what a standard deviation of test scores is worth.

No topic in the Fall 2026 topic list is named effect size. The same work is asked for under different wording, in the justify-a-claim topics: 3.11 for a difference between two proportions and 4.8 for a difference between two means hand you a confidence interval for that difference and ask what claim it supports.

A result is statistically significant at level $\alpha$ (alpha) when its p-value is at or below $\alpha$, with $\alpha$ fixed before the data arrive. The whole claim is about surprise under the null hypothesis: data at least this extreme would be unlikely if $H_0$ were true. It is not a claim about how big the effect is, whether it matters, or how likely $H_0$ is to be true.

Sample size is what makes that a narrow claim rather than an impressive one. Test $H_0: p = 0.50$ against $H_a: p \ne 0.50$ and hold the sample result fixed at $\hat{p} = 0.51$ (p-hat), one percentage point off. At $n = 1000$ the standard error is $\sqrt{0.50(0.50)/1000} = 0.0158$, so $z = 0.01/0.0158 = 0.63$ and the p-value is 0.527, which is nothing at all. At $n = 100{,}000$ the standard error falls to 0.00158, $z = 6.32$, and the p-value is under 0.0001. The 51 to 49 split never moved. Only the amount of data did, and the crossing happens at $n = 9604$.

So the sentence to distrust is "the result was significant, so the effect is large." Significance measures a gap in standard errors, and the standard error shrinks like $1/\sqrt{n}$, which means a fixed gap reads as stronger and stronger evidence as data pile up. The mirror error is "the result was not significant, so there is no effect." A small study that fails to reject often could not have detected an effect of ordinary size in the first place.

Significance is also a verdict at one chosen threshold, so a p-value of 0.08 is significant at $\alpha = 0.10$ and not at $\alpha = 0.05$, with nothing about the data changing in between. Report the estimate and a confidence interval next to the verdict: the size of the effect is its effect size, and whether that size is worth acting on is practical significance.