Statistical significance

A result is statistically significant at level $\alpha$ (alpha) when its p-value is at or below $\alpha$, with $\alpha$ fixed before the data arrive. The whole claim is about surprise under the null hypothesis: data at least this extreme would be unlikely if $H_0$ were true. It is not a claim about how big the effect is, whether it matters, or how likely $H_0$ is to be true.

Sample size is what makes that a narrow claim rather than an impressive one. Test $H_0: p = 0.50$ against $H_a: p \ne 0.50$ and hold the sample result fixed at $\hat{p} = 0.51$ (p-hat), one percentage point off. At $n = 1000$ the standard error is $\sqrt{0.50(0.50)/1000} = 0.0158$, so $z = 0.01/0.0158 = 0.63$ and the p-value is 0.527, which is nothing at all. At $n = 100{,}000$ the standard error falls to 0.00158, $z = 6.32$, and the p-value is under 0.0001. The 51 to 49 split never moved. Only the amount of data did, and the crossing happens at $n = 9604$.

So the sentence to distrust is "the result was significant, so the effect is large." Significance measures a gap in standard errors, and the standard error shrinks like $1/\sqrt{n}$, which means a fixed gap reads as stronger and stronger evidence as data pile up. The mirror error is "the result was not significant, so there is no effect." A small study that fails to reject often could not have detected an effect of ordinary size in the first place.

Significance is also a verdict at one chosen threshold, so a p-value of 0.08 is significant at $\alpha = 0.10$ and not at $\alpha = 0.05$, with nothing about the data changing in between. Report the estimate and a confidence interval next to the verdict: the size of the effect is its effect size, and whether that size is worth acting on is practical significance.