Power vs significance level (alpha)
By Jude Wallis · Published
The significance level, alpha, is the probability that a test rejects the null hypothesis when the null is true. Power is the probability that it rejects when a specific alternative is true. Both are rejection probabilities; they differ in which truth is assumed. Raising alpha raises power.
AP Statistics: Unit 3 (topics 3.8 Potential Errors When Performing Tests). The significance level as the probability of a Type I error, and power as the probability of correctly rejecting a false null hypothesis, both belong to Unit 3 topic 3.8, Potential Errors When Performing Tests, in the Fall 2026 AP Statistics course. The examples use a z test with a known sigma so each power can be computed from the standard normal curve.
Power vs significance level: the short answer
Both numbers answer the question "how often does this test reject ?" under two different assumptions about reality.
The significance level (alpha) is the probability of rejecting when is true. A rejection in that case is a Type I error, so is an error rate, and you want it small.
Power is the probability of rejecting when a specific alternative value of the parameter is true. A rejection in that case is the correct decision, so power is a success rate, and you want it large. Power equals , where (beta) is the probability of a Type II error against that alternative.
The two are tied together. With the sample size and everything else held fixed, raising makes the test reject more readily in every situation, so power goes up along with the Type I error rate. Lowering lowers both.
What the significance level is
The significance level is chosen by the person running the test, before the data are collected. Typical values are 0.10, 0.05, and 0.01. It is the threshold for the p-value: reject when the p-value is at or below .
Because it is computed assuming is true, it needs no information about the alternative. A test has one significance level, and the sample size does not change it. A test at rejects a true null hypothesis in 5% of random samples whether is 25 or 2,500, provided the conditions hold.
The choice is a judgment about consequences. A smaller protects against false alarms and makes real effects harder to detect. That trade is discussed in how to choose a significance level.
What power is
Power is a result of the design. Nobody sets it directly. It follows from four things:
- The significance level .
- The sample size.
- The variability in the data, which together with the sample size sets the standard error.
- How far the true parameter value is from the null value.
The fourth item means a test has many powers, one for each possible true value. A true value far from the null is easy to detect and gives high power. A true value close to the null gives low power. Any statement of power has to name the alternative it was computed against.
A study planned with high power, often 0.80 or more against an alternative that matters, is likely to detect that alternative if it is real. A study with low power can easily fail to reject even when is false, so a failure to reject from such a study says little. The relationship between power and is the subject of power vs Type II error.
One test at three significance levels
Take a z test of against with known and , so the standard error is . The test rejects when ("x-bar") is at or above a cutoff of . The table gives the probability of rejecting when the true mean is 100, 103, and 105.
| Cutoff for | Rejects if | Rejects if | Rejects if | |
|---|---|---|---|---|
| 0.01 | 106.98 | 0.0100 | 0.0924 | 0.2547 |
| 0.05 | 104.93 | 0.0500 | 0.2595 | 0.5087 |
| 0.10 | 103.84 | 0.1000 | 0.3891 | 0.6499 |
The column for is the Type I error rate, and it reproduces in every row. The last two columns are powers. Reading across a row, the rejection probability climbs as the truth moves away from 100. Reading down a column, a larger lowers the cutoff and raises every rejection probability at once. You can move the same levers in the power and errors interactive.
The differences side by side
| Feature | Significance level | Power |
|---|---|---|
| Probability of | Rejecting | Rejecting |
| Computed assuming | is true | A specific alternative value is true |
| The rejection is | A Type I error | A correct decision |
| You want it | Small | Large |
| How it is determined | Chosen directly | Follows from , , variability, and the true value |
| How many values per test | One | One for each alternative value |
| Effect of a larger sample | None | Increases |
| Complement | , the chance of not rejecting a true | , the Type II error probability |
When to use which. Use when carrying out a test: it is the line the p-value is compared with. Use power when planning a study or judging one that failed to reject: it tells you whether the test had a reasonable chance of detecting an effect of a stated size. On a free-response question about increasing power, the available moves are a larger sample, a larger , or less variability, and only the larger costs you more Type I errors.
The classic mix-up and how to avoid it
The first error is writing power as . Power is . The two complements describe different situations: is the probability of correctly keeping a true , and power is the probability of correctly rejecting a false one. In the example above, and the power against is 0.5087, and nothing requires those to add to 1.
The second error is assuming a stricter significance level makes a test better in every respect. Lowering from 0.05 to 0.01 in the example cuts the Type I error rate to a fifth and cuts the power against from 0.5087 to 0.2547. The only way to lower and keep or raise power is to change something else, usually the sample size.
The third error is treating either number as a statement about a hypothesis. is not the probability that is true, and power is not the probability that is true. Each is the probability of the decision "reject," computed after assuming which hypothesis holds.
To keep them straight, state the assumed truth first. "If the null is true, we reject with probability ." "If the mean is really 105, we reject with probability equal to the power."
Raising alpha raises power and the Type I error rate together
A z test uses against with known and . Find the power against a true mean of 105 at , , and . Use technology and carry unrounded values.
Standard error: .
Find each cutoff for from the null distribution, centered at 100. The upper-tail critical values are 2.326, 1.645, and 1.282, so the cutoffs are , , and .
Now assume the true mean is 105. Sample means are then centered at 105 with the same standard error of 3.
At : the cutoff 106.98 is standard errors above 105. The area to the right is 0.2547.
At : the cutoff 104.93 is standard errors from 105. The area to the right is 0.5087.
At : the cutoff 103.84 is standard errors from 105. The area to the right is 0.6499.
Compare. Moving from 0.01 to 0.10 multiplies the Type I error rate by 10 and raises the power against 105 from 0.2547 to 0.6499.
The power against is 0.2547 at , 0.5087 at , and 0.6499 at . A larger lowers the cutoff, so the test rejects more often whether is true or false. The powers were computed with unrounded cutoffs; working from the rounded z-scores shown gives values within 0.002 of these.
Lowering alpha and raising power at once with a larger sample
Keep the same test, against with , but increase the sample size to . Find the power against at and compare it with the power at and , which was 0.5087.
New standard error: .
Cutoff at : .
Assume the true mean is 105. The cutoff is standard errors from 105.
Power: the area to the right of on the standard normal curve, which technology gives as 0.8430 with unrounded values.
Compare with the original design. At and the Type I error rate was 0.05 and the power was 0.5087. At and the Type I error rate is 0.01 and the power is 0.8430.
For reference, keeping at gives a cutoff of and a power of 0.9543.
With and , the power against is 0.8430, up from 0.5087, while the Type I error rate falls from 0.05 to 0.01. The significance level is set by choice and is unaffected by sample size. Power depends on sample size, so more data can improve it without raising .
Frequently asked questions
Is power equal to 1 minus alpha?
No. Power is , where is the probability of a Type II error against a specific alternative. The quantity is the probability of not rejecting when is true. A test at can have a power of 0.26, 0.51, or 0.95 depending on the sample size and the alternative.
Does increasing the significance level increase power?
Yes, with everything else fixed. A larger widens the rejection region, so the test rejects more often under every true value of the parameter. The cost is a higher probability of a Type I error, since that probability is itself.
Does a larger sample size change the significance level?
No. The significance level is chosen by the researcher and stays where it is set. A larger sample reduces the standard error, which raises power against any given alternative and lowers the Type II error probability, while the Type I error rate remains .