Power vs significance level (alpha)

By Jude Wallis · Published

The significance level, alpha, is the probability that a test rejects the null hypothesis when the null is true. Power is the probability that it rejects when a specific alternative is true. Both are rejection probabilities; they differ in which truth is assumed. Raising alpha raises power.

AP Statistics: Unit 3 (topics 3.8 Potential Errors When Performing Tests). The significance level as the probability of a Type I error, and power as the probability of correctly rejecting a false null hypothesis, both belong to Unit 3 topic 3.8, Potential Errors When Performing Tests, in the Fall 2026 AP Statistics course. The examples use a z test with a known sigma so each power can be computed from the standard normal curve.

Power vs significance level: the short answer

Both numbers answer the question "how often does this test reject H0H_0?" under two different assumptions about reality.

The significance level α\alpha (alpha) is the probability of rejecting H0H_0 when H0H_0 is true. A rejection in that case is a Type I error, so α\alpha is an error rate, and you want it small.

Power is the probability of rejecting H0H_0 when a specific alternative value of the parameter is true. A rejection in that case is the correct decision, so power is a success rate, and you want it large. Power equals 1−β1 - \beta, where β\beta (beta) is the probability of a Type II error against that alternative.

The two are tied together. With the sample size and everything else held fixed, raising α\alpha makes the test reject more readily in every situation, so power goes up along with the Type I error rate. Lowering α\alpha lowers both.

What the significance level is

The significance level is chosen by the person running the test, before the data are collected. Typical values are 0.10, 0.05, and 0.01. It is the threshold for the p-value: reject H0H_0 when the p-value is at or below α\alpha.

Because it is computed assuming H0H_0 is true, it needs no information about the alternative. A test has one significance level, and the sample size does not change it. A test at α=0.05\alpha = 0.05 rejects a true null hypothesis in 5% of random samples whether nn is 25 or 2,500, provided the conditions hold.

The choice is a judgment about consequences. A smaller α\alpha protects against false alarms and makes real effects harder to detect. That trade is discussed in how to choose a significance level.

What power is

Power is a result of the design. Nobody sets it directly. It follows from four things:

  1. The significance level α\alpha.
  2. The sample size.
  3. The variability in the data, which together with the sample size sets the standard error.
  4. How far the true parameter value is from the null value.

The fourth item means a test has many powers, one for each possible true value. A true value far from the null is easy to detect and gives high power. A true value close to the null gives low power. Any statement of power has to name the alternative it was computed against.

A study planned with high power, often 0.80 or more against an alternative that matters, is likely to detect that alternative if it is real. A study with low power can easily fail to reject H0H_0 even when H0H_0 is false, so a failure to reject from such a study says little. The relationship between power and β\beta is the subject of power vs Type II error.

One test at three significance levels

Take a z test of H0:μ=100H_0: \mu = 100 against Ha:μ>100H_a: \mu > 100 with σ=15\sigma = 15 known and n=25n = 25, so the standard error is 1525=3\frac{15}{\sqrt{25}} = 3. The test rejects when xˉ\bar{x} ("x-bar") is at or above a cutoff of 100+z∗×3100 + z^* \times 3. The table gives the probability of rejecting when the true mean is 100, 103, and 105.

α\alphaCutoff for xˉ\bar{x}Rejects if μ=100\mu = 100Rejects if μ=103\mu = 103Rejects if μ=105\mu = 105
0.01106.980.01000.09240.2547
0.05104.930.05000.25950.5087
0.10103.840.10000.38910.6499

The column for μ=100\mu = 100 is the Type I error rate, and it reproduces α\alpha in every row. The last two columns are powers. Reading across a row, the rejection probability climbs as the truth moves away from 100. Reading down a column, a larger α\alpha lowers the cutoff and raises every rejection probability at once. You can move the same levers in the power and errors interactive.

The differences side by side

FeatureSignificance level α\alphaPower
Probability ofRejecting H0H_0Rejecting H0H_0
Computed assumingH0H_0 is trueA specific alternative value is true
The rejection isA Type I errorA correct decision
You want itSmallLarge
How it is determinedChosen directlyFollows from α\alpha, nn, variability, and the true value
How many values per testOneOne for each alternative value
Effect of a larger sampleNoneIncreases
Complement1−α1 - \alpha, the chance of not rejecting a true H0H_0β\beta, the Type II error probability

When to use which. Use α\alpha when carrying out a test: it is the line the p-value is compared with. Use power when planning a study or judging one that failed to reject: it tells you whether the test had a reasonable chance of detecting an effect of a stated size. On a free-response question about increasing power, the available moves are a larger sample, a larger α\alpha, or less variability, and only the larger α\alpha costs you more Type I errors.

The classic mix-up and how to avoid it

The first error is writing power as 1−α1 - \alpha. Power is 1−β1 - \beta. The two complements describe different situations: 1−α1 - \alpha is the probability of correctly keeping a true H0H_0, and power is the probability of correctly rejecting a false one. In the example above, α=0.05\alpha = 0.05 and the power against μ=105\mu = 105 is 0.5087, and nothing requires those to add to 1.

The second error is assuming a stricter significance level makes a test better in every respect. Lowering α\alpha from 0.05 to 0.01 in the example cuts the Type I error rate to a fifth and cuts the power against μ=105\mu = 105 from 0.5087 to 0.2547. The only way to lower α\alpha and keep or raise power is to change something else, usually the sample size.

The third error is treating either number as a statement about a hypothesis. α\alpha is not the probability that H0H_0 is true, and power is not the probability that HaH_a is true. Each is the probability of the decision "reject," computed after assuming which hypothesis holds.

To keep them straight, state the assumed truth first. "If the null is true, we reject with probability α\alpha." "If the mean is really 105, we reject with probability equal to the power."

Raising alpha raises power and the Type I error rate together

A z test uses H0:μ=100H_0: \mu = 100 against Ha:μ>100H_a: \mu > 100 with σ=15\sigma = 15 known and n=25n = 25. Find the power against a true mean of 105 at α=0.01\alpha = 0.01, 0.050.05, and 0.100.10. Use technology and carry unrounded values.

  1. Standard error: 1525=3\frac{15}{\sqrt{25}} = 3.

  2. Find each cutoff for xˉ\bar{x} from the null distribution, centered at 100. The upper-tail critical values are 2.326, 1.645, and 1.282, so the cutoffs are 100+2.326(3)=106.98100 + 2.326(3) = 106.98, 100+1.645(3)=104.93100 + 1.645(3) = 104.93, and 100+1.282(3)=103.84100 + 1.282(3) = 103.84.

  3. Now assume the true mean is 105. Sample means are then centered at 105 with the same standard error of 3.

  4. At α=0.01\alpha = 0.01: the cutoff 106.98 is 106.98−1053=0.66\frac{106.98 - 105}{3} = 0.66 standard errors above 105. The area to the right is 0.2547.

  5. At α=0.05\alpha = 0.05: the cutoff 104.93 is 104.93−1053=−0.02\frac{104.93 - 105}{3} = -0.02 standard errors from 105. The area to the right is 0.5087.

  6. At α=0.10\alpha = 0.10: the cutoff 103.84 is 103.84−1053=−0.39\frac{103.84 - 105}{3} = -0.39 standard errors from 105. The area to the right is 0.6499.

  7. Compare. Moving α\alpha from 0.01 to 0.10 multiplies the Type I error rate by 10 and raises the power against 105 from 0.2547 to 0.6499.

The power against μ=105\mu = 105 is 0.2547 at α=0.01\alpha = 0.01, 0.5087 at α=0.05\alpha = 0.05, and 0.6499 at α=0.10\alpha = 0.10. A larger α\alpha lowers the cutoff, so the test rejects more often whether H0H_0 is true or false. The powers were computed with unrounded cutoffs; working from the rounded z-scores shown gives values within 0.002 of these.

Lowering alpha and raising power at once with a larger sample

Keep the same test, H0:μ=100H_0: \mu = 100 against Ha:μ>100H_a: \mu > 100 with σ=15\sigma = 15, but increase the sample size to n=100n = 100. Find the power against μ=105\mu = 105 at α=0.01\alpha = 0.01 and compare it with the power at n=25n = 25 and α=0.05\alpha = 0.05, which was 0.5087.

  1. New standard error: 15100=1.5\frac{15}{\sqrt{100}} = 1.5.

  2. Cutoff at α=0.01\alpha = 0.01: 100+2.326(1.5)=103.49100 + 2.326(1.5) = 103.49.

  3. Assume the true mean is 105. The cutoff is 103.49−1051.5=−1.01\frac{103.49 - 105}{1.5} = -1.01 standard errors from 105.

  4. Power: the area to the right of −1.01-1.01 on the standard normal curve, which technology gives as 0.8430 with unrounded values.

  5. Compare with the original design. At n=25n = 25 and α=0.05\alpha = 0.05 the Type I error rate was 0.05 and the power was 0.5087. At n=100n = 100 and α=0.01\alpha = 0.01 the Type I error rate is 0.01 and the power is 0.8430.

  6. For reference, keeping α=0.05\alpha = 0.05 at n=100n = 100 gives a cutoff of 100+1.645(1.5)=102.47100 + 1.645(1.5) = 102.47 and a power of 0.9543.

With n=100n = 100 and α=0.01\alpha = 0.01, the power against μ=105\mu = 105 is 0.8430, up from 0.5087, while the Type I error rate falls from 0.05 to 0.01. The significance level is set by choice and is unaffected by sample size. Power depends on sample size, so more data can improve it without raising α\alpha.

Frequently asked questions

Is power equal to 1 minus alpha?

No. Power is 1−β1 - \beta, where β\beta is the probability of a Type II error against a specific alternative. The quantity 1−α1 - \alpha is the probability of not rejecting H0H_0 when H0H_0 is true. A test at α=0.05\alpha = 0.05 can have a power of 0.26, 0.51, or 0.95 depending on the sample size and the alternative.

Does increasing the significance level increase power?

Yes, with everything else fixed. A larger α\alpha widens the rejection region, so the test rejects more often under every true value of the parameter. The cost is a higher probability of a Type I error, since that probability is α\alpha itself.

Does a larger sample size change the significance level?

No. The significance level is chosen by the researcher and stays where it is set. A larger sample reduces the standard error, which raises power against any given alternative and lowers the Type II error probability, while the Type I error rate remains α\alpha.