What Is a Good P-Value? There Is No Such Thing

Why "good p-value" is the wrong question

A p-value is not a grade. It does not arrive good or bad on its own. The p-value, written $p$, is a probability: the chance of getting a result at least as extreme as the one in your sample, computed as if the null hypothesis were exactly true.

What turns that probability into a decision is a second number you supply. The significance level, written $\alpha$ ("alpha"), is the threshold you pick before you collect data. Then the rule is mechanical:

• If $p \leq \alpha$, reject the null hypothesis.
• If $p > \alpha$, fail to reject it.

So "is 0.03 a good p-value?" has no answer until someone says what $\alpha$ is. At $\alpha = 0.05$, a p-value of 0.03 rejects. At $\alpha = 0.01$, the same 0.03 does not. Same data, same p-value, opposite decisions, because the threshold moved.

The ordering matters as much as the numbers. You fix $\alpha$ first, then compute $p$, then compare. Reading the p-value first and then deciding what counts as small is not a test at all, because you can always pick a threshold that gives you the answer you wanted. Choosing a significance level is a separate decision made for separate reasons.

What the number actually measures

Read the definition slowly, because every misuse of p-values comes from misreading one clause of it. The p-value is:

The vertical bar means "given that," so the whole quantity is conditional on the null hypothesis $H_0$ ("H-naught") being true. Three consequences follow.

• Small $p$ means the data are surprising under the null. It does not measure how big an effect is, only how badly the data fit the null model.
• $p$ is not the probability that the null hypothesis is true. That would be $P(H_0 \mid \text{data})$, which is a different conditional probability and is not what any AP procedure computes. See what a p-value means for the full unpacking.
• $p$ is not the probability you made a mistake. The long-run Type I error rate is $\alpha$, the threshold, not $p$.

The conditional bar is also the reason a p-value exists at all: the null supplies one specific distribution to compute the probability against. That is worth understanding on its own, and it is covered in why we assume the null hypothesis is true.

The 0.05 threshold is inherited, not derived

Nothing in probability theory changes at 0.05. The threshold became standard through early twentieth century practice, because it was a convenient round number for published tables, and it stuck. No mathematical result singles it out.

You can see how arbitrary it is by asking what $\alpha = 0.05$ actually buys you. It says: if the null is true, you accept a 5% chance of rejecting it anyway. That is a Type I error. Lower $\alpha$ to 0.01 and you cut false alarms to 1%, but you also make it harder to detect a real effect, so Type II errors go up and power goes down. At a fixed sample size there is no setting of $\alpha$ that reduces both, which is why the choice belongs to the context and not to tradition. Collecting more data is the one move that lowers the Type II error rate without raising the Type I rate. Type I vs Type II errors walks through the tradeoff.

Different fields settled on different numbers for exactly this reason. Screening tests that must not miss a real problem often run at $\alpha = 0.10$. Settings where a false alarm is expensive run at $\alpha = 0.01$. On the AP exam, the problem usually hands you $\alpha$, and if it does not, 0.05 is the safe default to state and use.

p = 0.049 and p = 0.051 are nearly identical evidence

This is the point that matters most. Two p-values on opposite sides of 0.05 lead to opposite written conclusions, but they represent almost exactly the same amount of evidence.

Put them on the test-statistic scale, where the comparison is honest. For a two-sided z-test:

The entire gap between "reject" and "fail to reject" is $1.9686 - 1.9515 = 0.0171$ standard errors. That is under two hundredths of one standard error. Nothing in the data changed meaningfully; the sample statistic moved by an amount too small to matter, and the sentence you write flipped.

So treat the p-value as a continuous measure of evidence, and treat the reject or fail-to-reject line as an administrative convention on top of it. A result at $p = 0.06$ is not "no effect." It is evidence that did not clear the bar you set. Say that, and say how close it came.

What you must not do is respond to a near miss by moving the bar, adding data until the p-value drops, or running extra tests and reporting the one that worked. Each of those changes the true error rate far above the $\alpha$ you claimed, so the p-value you report stops meaning what it says. The honest move is to report the p-value you got, note that it is close, and treat the question as unresolved.

Why "highly significant" is sloppy

Once $\alpha$ is fixed, the decision is binary. A result either clears the threshold or it does not. There is no third category of extra-cleared.

"Highly significant" tries to smuggle two different ideas into one phrase:

• How strong the evidence against the null is. A p-value of 0.0001 does say the data are more surprising under the null than a p-value of 0.04 does. Report the actual p-value and let the reader see that.
• How big or important the effect is. The p-value cannot tell you this. With a large enough sample, an effect too small to care about will produce a tiny p-value, because the standard error shrinks toward zero. This is the difference between statistical significance and practical significance.

A related habit worth dropping is "approaching significance" for $p = 0.07$. The p-value is not moving toward anything. It is the number you got.

Write the conclusion the way the exam wants it: name the decision, name $\alpha$, give the p-value, and state the conclusion in context. How to write a hypothesis test conclusion has the sentence template.

What to report instead of a verdict

Give the reader enough to judge for themselves. A complete inference write-up carries four things:

1. The test statistic and, for a $t$ or $\chi^2$ procedure, its degrees of freedom, so the calculation can be checked. Two or three decimals is plenty.
2. The p-value itself, not just "less than 0.05." Three or four decimals, and write $p < 0.0001$ rather than $p = 0$ when it underflows. How many decimal places to keep covers the rounding rules and why they change the answer.
3. A confidence interval for the same parameter. The interval shows the size of the plausible effects, which a p-value never does. See confidence interval vs hypothesis test.
4. The conclusion in context, naming the population and the parameter in the words of the problem.

Done that way, a p-value of 0.049 and a p-value of 0.051 read as what they are: the same result, sitting on either side of a line someone drew.