P-value vs alpha (significance level)

P-value vs alpha: the short answer

A hypothesis test ends by comparing two numbers, and these are the two. Alpha ($\alpha$), the significance level, is a threshold you choose before you see the data, most often 0.05. The p-value is a number the data hand you afterward, measuring how surprising your result would be if the null hypothesis were true.

The test is the comparison between them. If the p-value is at or below $\alpha$, the result is too surprising to credit chance, so you reject the null hypothesis; if the p-value is above $\alpha$, you fail to reject it. Alpha is the fixed bar you set in advance, and the p-value is what you measure and hold up against that bar.

What alpha (the significance level) is

The significance level $\alpha$ is the predetermined probability of rejecting the null hypothesis when it is actually true. You set it before collecting data, typically at 0.05, sometimes 0.01 or 0.10, based on how costly a false alarm would be.

Because rejecting a true null is exactly a Type I error, $\alpha$ is also the probability of a Type I error, a link explored in Type I versus Type II errors. Choosing $\alpha$ is a judgment call about risk: a smaller $\alpha$ such as 0.01 makes you more cautious about rejecting the null, cutting false alarms but making a true effect harder to detect. Whatever value you pick, fix it before you look at the data, so the threshold cannot be bent to fit the result.

What the p-value is

The p-value is the probability, computed assuming the null hypothesis is true, of getting a test statistic at least as extreme as the one your sample produced. It is a measure of evidence against the null: a small p-value means your result would be unusual if the null were true, which is what points toward the alternative.

The lower the p-value, the stronger the evidence for the alternative hypothesis. A p-value that is not small means the result would not be surprising under the null, so it fails to provide convincing evidence for the alternative; note that this does not prove the null true. The p-value comes entirely from the data through the test statistic, so unlike $\alpha$ you cannot know it in advance. The what a p-value means guide works through the interpretation in depth.

The differences side by side

Alpha is the standard you commit to in advance; the p-value is the reading you take from the data and compare against it.

How the decision works

The formal decision is a single comparison. If the p-value is less than or equal to $\alpha$, reject the null hypothesis and call the result statistically significant. If the p-value is greater than $\alpha$, fail to reject the null hypothesis, since the evidence is not strong enough to clear the bar you set.

Two cautions matter for the AP exam. "Fail to reject" is not "accept": a large p-value means you lack convincing evidence for the alternative, not that the null is true. And always state the conclusion in context and tie it to the comparison, for example "because the p-value of 0.036 is less than $\alpha = 0.05$, we reject the null and conclude there is convincing evidence that..." You can check a p-value against your threshold with the p-value calculator.

The classic mix-up and how to avoid it

The core confusion is treating the p-value and $\alpha$ as the same kind of thing, or reading the p-value as "the probability the null hypothesis is true." It is neither. The p-value is computed by assuming the null is true, so it cannot also be the chance that the null is true, and it is a result from the data, not a threshold you chose.

Keep them apart with three reminders. First, $\alpha$ comes before the data and never changes; the p-value comes from the data and changes with every sample. Second, the p-value is a conditional probability about the data given the null, not a probability about the null itself. Third, do the comparison in one fixed direction: reject only when the p-value is at or below $\alpha$. Setting the null and alternative correctly first, as in null versus alternative hypothesis, keeps the whole comparison pointed the right way.