How to Choose a Significance Level (Alpha)

What alpha is, and when you pick it

The significance level $\alpha$ (the Greek letter alpha) is the probability of rejecting a true null hypothesis. It is the false alarm rate you agree to run before any data exist. The AP course defines it that way exactly: the predetermined probability of rejecting the null hypothesis given that it is true.

Two parts of that definition carry the weight. Predetermined means the number is fixed in advance, not read off the results. **Probability of rejecting a true $H_0$** means $\alpha$ is the Type I error rate, so setting $\alpha$ is deciding how often you are willing to raise an alarm that turns out to be nothing.

The decision rule that follows is mechanical. If the p-value is less than or equal to $\alpha$, reject $H_0$; otherwise fail to reject. The two numbers are different objects that happen to share a scale: $\alpha$ is a threshold you choose, and the p-value is what the data hand back (p-value vs alpha).

Why 0.05 is a convention and not a law

0.05 became the default for historical reasons rather than mathematical ones. Nothing about the world changes between a p-value of 0.049 and one of 0.051, yet the reject-or-not label flips. The AP course lists 0.01, 0.05, and 0.10 as typical values set before collecting data, which is a clear signal that 0.05 is one option among several.

On an exam the practical rule is short. If the problem states a significance level, use it. If it does not, 0.05 is the safe default, and you write it into your setup so the reader knows what you compared against.

Outside of exams, report the actual p-value next to the decision. Someone who sees 0.0007 learns something that "p < 0.05" hides, and someone who sees 0.048 learns something too. See what does a p-value mean for how much a single p-value can carry.

The tradeoff you accept when you lower alpha

Lowering $\alpha$ makes rejection harder, so you reject a true $H_0$ less often. The same change makes you fail to reject a false $H_0$ more often, which is a Type II error, with probability $\beta$ (the Greek letter beta).

Power is $1 - \beta$, the probability of correctly rejecting a false null hypothesis. The chain runs in one direction: lower $\alpha$, fewer Type I errors, more Type II errors, less power. Raising $\alpha$ runs the chain the other way, buying power with false alarms.

The AP course lists four changes that lower the Type II error rate and raise power, holding the others fixed: a larger sample size, a smaller standard error, a true parameter farther from the null value, and a larger significance level. Only two of those are yours to control, and the one that costs you nothing in credibility is sample size.

That gives a clean division of labor. The consequences of a Type I error should drive your choice of $\alpha$. The consequences of a Type II error should drive how large a sample you collect. Type I vs Type II errors works through the full error table.

A medical screening, where missing a real problem is worse

A county clinic screens a random sample of 400 children from its service area for elevated blood lead. The statewide baseline rate is 5%, so the test is $H_0: p = 0.05$ against $H_a: p > 0.05$, where $p$ is the true proportion of children in the clinic's service area above the threshold. Rejecting $H_0$ triggers a free follow-up testing and remediation program.

Weigh the two errors before choosing. A Type I error means launching the program in a service area that is actually at baseline, which costs money and causes some avoidable worry. A Type II error means missing a genuinely elevated rate, so children keep absorbing lead that a follow-up would have caught.

The second error is far worse, and raising $\alpha$ raises power, so set $\alpha = 0.10$ and say why in the write-up. The screen is a first filter, and a rejection here triggers individual follow-up testing rather than a diagnosis. A false alarm here buys another round of testing; a missed signal buys nothing.

With 26 of the 400 children above the threshold, the p-value comes out to 0.0838, worked below. At $\alpha = 0.10$ the clinic acts. At $\alpha = 0.05$ it does not, and identical data lead to opposite actions.

A manufacturing tolerance, where a false alarm is expensive

A plant tests whether a bearing line has drifted past its 2% defect tolerance, so $H_0: p = 0.02$ against $H_a: p > 0.02$. Rejecting means halting the line for a full tooling rebuild, a shutdown that costs days of output.

The errors now swap places. A Type I error stops a line that was running inside tolerance. A Type II error lets a slightly elevated defect rate continue, and downstream inspection catches most of those parts before they reach a customer.

Here the expensive error is the false alarm, so tighten the threshold to $\alpha = 0.01$ and demand strong evidence before pulling the line. To protect against the Type II error, sample more bearings rather than loosening $\alpha$, since sample size is the lever that improves both error rates at once.

The field does not decide the answer. A drug approval trial is also medical, and there a Type I error puts an ineffective treatment on the market, so those trials use 0.05 or stricter. If the bearing were a safety-critical aircraft part, the reasoning above would flip. Ask which error hurts more in this specific study.

Choosing alpha after seeing the p-value breaks the test

Suppose you see a p-value of 0.06 and then decide that $\alpha = 0.10$ was your level all along. Your real Type I error rate is no longer 0.10. It is whatever rate a rule like "reject whenever the p-value lands near the line" produces, and that rate is higher than any number you wrote down. The guarantee attached to $\alpha$ holds only when $\alpha$ is fixed independently of the data.

The same problem arrives in three disguises:

• Switching from a two-sided to a one-sided alternative after seeing which way $\hat{p}$ (read "p-hat", the sample proportion) came out, which halves the p-value at no cost in evidence.
• Collecting more data only when the first result missed the cutoff, then testing again.
• Running many tests and reporting only the ones that cleared 0.05.

Each of these inflates the true false alarm rate above the stated one. Fix $\alpha$ and $H_a$ in the setup before the data exist, and the arithmetic afterwards means what it claims. Reporting the p-value alongside the decision also lets a reader judge a borderline result for themselves.

How to state alpha in a written response

Put $\alpha$ in the setup, next to the hypotheses and before any calculation. One line of justification is enough when the problem lets you choose.

Template. Let $p$ be the true proportion of (population) that (response). $H_0: p = p_0$ versus $H_a: p > p_0$. We will use $\alpha = 0.05$.

When the problem names a significance level, use that one and do not substitute your own. When it does not, name your choice and give the reason in a sentence: "we use $\alpha = 0.10$ because failing to detect elevated lead is more damaging than an unnecessary round of follow-up testing."

Then close the loop at the end. Compare the p-value to the $\alpha$ you named, state reject or fail to reject, and finish with a conclusion in context. The proportion z-test calculator lets you see how a p-value lands against several thresholds, and the AP treatment of error rates sits at potential errors when performing tests.