Power of a test explained (statistics)
By Jude Wallis · Published
The power of a test is the probability that it correctly rejects a false null hypothesis. Power equals , where is the probability of a Type II error. Power rises with a larger sample, a larger significance level, less variability, and a true value farther from the null.
AP Statistics: Unit 3 (topics 3.8 Potential Errors When Performing Tests). Power is part of Unit 3, topic 3.8 (Potential Errors When Performing Tests) in the Fall 2026 AP Statistics course, alongside Type I and Type II errors.
What the power of a test measures
A hypothesis test can miss. The null hypothesis can be false and the sample can still fail to provide convincing evidence against it. That miss is a Type II error.
Power is the probability of not missing. It is the chance that the test detects a difference that is really there. A test with power 0.90 against some alternative will reject in about 90% of random samples when that alternative is the truth. A test with power 0.30 will usually fail to notice.
Power is always calculated against one specific alternative value of the parameter. "The power of this test" is incomplete until you say power to detect what. The same test has low power against a true value close to the null value and high power against one far away.
The formula for power
- is the null hypothesis.
- The vertical bar means "given that".
- ("beta") is the probability of a Type II error: failing to reject when a specific alternative value is true.
Power and are complements. If power is 0.64, then .
Power is not the complement of ("alpha"), the significance level. is the probability of rejecting when is true. Power is the probability of rejecting when is false. They are computed under different assumptions about the truth. The four outcomes of a test are laid out in Type I and Type II errors explained.
What increases power
Four things raise the power of a test. The numbers come from the worked example, a test of against with baseline power of about 0.64.
| Change | Effect on power | In the worked example |
|---|---|---|
| Larger sample size | Increases | to : about 0.64 to about 0.89 |
| Larger significance level | Increases | to : about 0.64 to about 0.77 |
| True value farther from the null value | Increases | True against true : about 0.64 against about 0.26 |
| Less variability in the data | Increases | A smaller standard error separates the two cases more clearly |
Each lever has a cost or a limit:
- A larger sample costs time and money, but it raises power without raising the chance of a Type I error.
- A larger makes false alarms more likely. Dropping to in the example cuts power to about 0.37.
- The true parameter value is not something the researcher controls. Researchers decide what size of difference is worth detecting and plan for that.
- Variability can sometimes be reduced by better measurement or by design choices such as blocking or pairing.
How to interpret power in context
An interpretation names the specific alternative, gives the probability, and says what the test would conclude.
"If the true proportion of voters who support the measure is 0.60, there is about a 0.64 probability that this test will find convincing evidence that the proportion is greater than 0.50."
The matching Type II error statement: "If the true proportion is 0.60, there is about a 0.36 probability that the test will fail to find convincing evidence that the proportion is greater than 0.50."
Both sentences begin with "if the true proportion is 0.60". Power is a conditional probability. It does not say how likely it is that is false.
Why power matters when planning a study
Power is mostly a planning tool. Before collecting data, a researcher asks: if the effect I care about is real, how likely is my study to detect it?
If the answer is low, a result of "fail to reject " will be hard to read. It could mean there is no effect, or it could mean the study was too small to see one. A test with low power cannot tell those apart. That is one reason failing to reject is never treated as evidence that is true.
A common convention in research is to aim for power of at least 0.80 against the smallest difference that would matter in practice, and to choose the sample size that reaches it. In the worked example, 100 voters gives power of about 0.64 against , and 200 voters gives about 0.89.
Very high power also has a side effect. With a huge sample, a test will detect differences too small to matter. See statistical vs practical significance.
Common mistakes
- Writing power as . Power is .
- Calling power "the probability of rejecting ". It is the probability of rejecting when is false, against a stated alternative.
- Treating power as one fixed number for a test. It changes with the true parameter value. Power against and power against are different.
- Thinking a smaller makes a test better in every way. A smaller lowers the Type I error probability and also lowers power.
- Reading power as the probability that is true. Power assumes a particular alternative is true and reports how often the test would detect it.
- Computing power after the fact from the observed result and using it to explain a non-significant test. Power belongs to the planning stage, against a difference chosen in advance.
- Saying "increase power by increasing the effect size". A researcher chooses which difference to plan for, not what the truth is.
Power in AP Statistics
Power belongs with errors in testing: Unit 3, topic 3.8, Potential Errors When Performing Tests, in the Fall 2026 AP Statistics course. Unit 3 carries 15-25% of the multiple-choice section, and the idea applies equally to tests for means in Unit 4.
Be ready with the definition, the relationship power , an interpretation in context against a specific alternative, and the direction in which sample size, , the true parameter value, and variability each move power.
The power and errors interactive shows the null and alternative sampling distributions side by side, with , , and power as shaded areas you can change. Follow it with the Type I and Type II error practice set. For side-by-side comparisons, see power vs Type II error and power vs significance level.
Power of a one-proportion z-test
A campaign will survey a random sample of 100 voters in a large city to test against at , where is the proportion of all voters in the city who support a ballot measure. If the true proportion is 0.60, what is the power of the test? What happens with 200 voters?
Find which sample proportions lead to rejecting . If is true, is approximately normal with mean 0.50 and standard deviation . The expected counts, 50 and 50, are both at least 10.
For a one-sided test at , reject when , which is when .
Now assume the truth is . Then is approximately normal with mean 0.60 and standard deviation .
Power is the probability that lands in the rejection region under that truth: . Standardize: .
. So power is about 0.64 and .
Repeat with . The cutoff becomes , and under the standard deviation of is .
, and .
With 100 voters, power is about 0.64: if the true proportion of supporters is 0.60, there is about a 0.64 probability the test finds convincing evidence that it is greater than 0.50. With 200 voters, power rises to about 0.89.
Frequently asked questions
What is the power of a test in statistics?
It is the probability that a hypothesis test rejects the null hypothesis when the null hypothesis is false, calculated against a specific alternative value of the parameter. It equals , where is the probability of a Type II error.
How do you increase the power of a test?
Increase the sample size, increase the significance level , or reduce variability through better measurement or design. Power is also higher when the true parameter value is farther from the null value. Of these, only increasing raises the probability of a Type I error.
What is a good power for a test?
A common research convention is to plan for power of at least 0.80 against the smallest difference that matters in practice. That is a convention, not a rule. Higher power needs a larger sample, so the target is a balance between cost and the risk of missing a real effect.
What is the relationship between power and Type II error?
They are complements. Power is , so any change that lowers the probability of a Type II error raises power by the same amount. A test with against some alternative has power 0.64 against it.