Critical Value vs P-Value

A critical value is a quantile of a reference distribution, chosen so that a stated area falls beyond it. For a confidence interval at level $C$ it is the number $z^*$ (z-star) or $t^*$ (t-star) with the middle $C$ of the curve lying between $-z^*$ and $z^*$. For a significance test it is the point cutting off $\alpha$ (alpha) in whichever tail the alternative hypothesis points to. One idea, two jobs: it sets a width, or it sets a boundary.

The standard normal values, listed in full on the z-table:

At 95%, 1.960 is the value leaving 0.025 in each tail, so 0.95 of the standard normal curve sits between $-1.960$ and $1.960$. With the population standard deviation unknown you switch to $t^*$ at the right degrees of freedom, and it is larger than the matching $z^*$ at every finite df. At 95% the t-table gives 2.262 at 9 df, 2.064 at 24 df, and 1.984 at 99 df, closing on 1.960 from above without reaching it.

The error worth naming is "the critical value for 95% confidence is 1.96." It is 1.96 only when the reference curve is the standard normal. A mean from a sample of 12 uses $t$ with 11 df, where the critical value is 2.201, and with a normal population, reaching out only 1.96 standard errors captures the true mean 92.4% of the time rather than 95%. The interval is about 11% too narrow and it undercovers by more than two points.

Not every critical value comes in a $\pm$ pair. The chi-square distribution is right skewed and its tests use the upper tail alone, so there is a single cutoff: 7.815 at 3 degrees of freedom and $\alpha = 0.05$, from the chi-square table. A critical value also marks only the boundary. How far past it you landed is what a p-value reports, which is why two tests can share a critical value and carry very different evidence.

A p-value is a conditional probability computed while the null hypothesis is taken as true. Written out, it is $P(\text{a result at least as extreme as the observed one} \mid H_0 \text{ is true})$, where extreme means in whichever direction the alternative hypothesis points.

Suppose you flip a coin 100 times, get 60 heads, and test $H_0: p = 0.5$ against $H_a: p \ne 0.5$. Under the null, $\hat{p}$ (p hat) is centered at 0.5 with standard deviation 0.05, so the observed $\hat{p} = 0.60$ sits $z = \frac{0.60 - 0.50}{0.05} = 2.00$ standard deviations out. Under the normal model the z-test uses, the two-sided p-value is 0.0455. Said aloud: if the coin really were fair, that model puts about 4.6 percent of samples of 100 flips at least 0.10 away from 0.5 in one direction or the other. Against the one-sided $H_a: p > 0.5$ the same model gives 0.0228, half as much, because only the upper tail counts.

Two readings of that 0.0455 are wrong and both are everywhere. It is not the probability that the coin is fair, and it is not the probability that the result happened by chance. Both of those are claims about the hypothesis, while the p-value is a claim about the data that takes the hypothesis as given. The conditioning runs one way only, and reversing it is the single most repeated error in applied statistics.

A large p-value is not evidence for $H_0$ either. It means the data would not be surprising if $H_0$ were true, which is a much weaker statement than $H_0$ being true. Small samples have little power to detect anything, so a large p-value from one may mean only that the test could not have found an effect of ordinary size, which is why a p-value of 0.4 from 12 observations tells you very little.

The decision comes from comparing the p-value against a significance level $\alpha$ (alpha) fixed before the data are seen: reject $H_0$ when the p-value is at or below $\alpha$, and fail to reject it otherwise. The p-value itself is a measure of evidence on a continuous scale, so 0.049 and 0.051 carry nearly identical information despite falling on opposite sides of the usual cutoff.