Power vs Confidence Level

Power is $1 - \beta$ (beta), where $\beta$ is the probability of a Type II error, and both numbers are computed while assuming one specific alternative value is the truth. A test therefore does not have a power. It has one power for every value the parameter could take, and a power quoted without naming that value is not a number anyone can use.

Take the z test behind the site's power visualizer: $H_0: \mu = 100$ against $H_a: \mu > 100$, with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is $15/\sqrt{25} = 3$. At $\alpha = 0.05$ the test rejects once $\bar{x}$ (x-bar) clears $100 + 1.645(3) = 104.93$. If the true mean is 105, sample means center at 105 with that same standard error of 3, and the chance of clearing 104.93 is 0.5087. That is the power against $\mu = 105$, near a coin flip.

The misreading to kill: "power is 0.5087, so there is about a 51 percent chance the null hypothesis is false." Power says nothing about whether $H_0$ is false. It assumes a stated alternative is already true and reports how often the test would notice. It is also not the probability that this particular conclusion is correct, since it is a long-run rate for a procedure aimed at one alternative.

Two boundaries are worth holding. Against $\mu = 100$, where the null is true, power equals $\alpha$ exactly, 0.0500, because the only rejections left are Type I errors. And power can sit below $\alpha$: this one-sided test looks upward only, so against a true mean of 95 its power is 0.0005.

Name the lever before claiming a tradeoff. Raising $n$ from 25 to 100 takes power against $\mu = 105$ from 0.5087 to 0.9543 and leaves $\alpha$ at 0.05. Raising $\alpha$ from 0.05 to 0.10 takes power to 0.6499 and doubles the Type I error rate in the same move. AP Statistics covers power in topic 3.8, Potential Errors When Performing Tests.

A confidence level is chosen before the data are collected and it does two jobs at once. It fixes the critical value the interval formula uses, and it names the rate at which that formula succeeds across repeated random samples of the same size. Write it $C$. The parameter never moves. The interval does, because a fresh sample produces fresh endpoints.

Suppose 240 of 400 randomly sampled adults say yes, so $\hat{p} = 0.60$ (p-hat) and the standard error is $\sqrt{\frac{0.60(0.40)}{400}} = 0.0245$. At 90% the critical value is $z^* = 1.645$ and the interval runs 0.560 to 0.640. At 95% it is $z^* = 1.960$ and the interval runs 0.552 to 0.648. At 99% it is $z^* = 2.576$ and the interval runs 0.537 to 0.663. One data set, three intervals. Nothing about the sample changed, only the level.

The sentence to stop writing is "there is a 95% probability that the true proportion is between 0.552 and 0.648." Once those two numbers exist they are fixed, the proportion was always fixed, and so that probability is 0 or 1 with no way to tell which. The randomness was spent when the sample was drawn. The 95% belongs to the procedure: it is a statement about the samples you did not take. The coverage simulator makes that visible, with the true $p$ drawn as a line and each new sample laying down one more interval to count.

A confidence level is what the method promises, not always what it delivers. For the proportion interval above the two come apart: at $n = 50$ with a true $p$ of 0.60, its exact long-run capture rate is 94.1% against a nominal 95%, because the standard error is built from $\hat{p}$ rather than from the unknown $p$. The rate a method actually achieves is its coverage probability.

Raising the level buys reliability with width and nothing else. At a fixed $n$ the 99% interval above is 0.126 wide against 0.096 at 95%, and a 100% interval would be every value from 0 to 1: certain to capture $p$ and worth nothing.