Significance Level vs Power

$\alpha$ (alpha) does two jobs with one number. It is the cutoff the p-value is compared against, and it is the probability the test rejects $H_0$ when $H_0$ is true. Those coincide because the rejection region is built to hold exactly that much area under the null distribution, which is also why $\alpha$ has to be chosen before the data arrive.

Take $H_0: \mu = 100$ against $H_a: \mu > 100$, with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is 3. Setting $\alpha = 0.05$ puts the cutoff at $100 + 1.645(3) = 104.93$: if the true mean really is 100, 5 percent of samples of 25 land above 104.93 through nothing but sampling variability. Set $\alpha = 0.01$ instead and the cutoff moves out to 106.98.

The misreading is subtle and common: "I test at $\alpha = 0.05$, so 5 percent of the results I call significant are wrong." It is not 5 percent of your rejections. It is 5 percent of the tests you run on null hypotheses that happen to be true. What fraction of your significant findings are false alarms depends on how often the nulls you test are true in the first place, and $\alpha$ carries no information about that.

A fixed threshold also does not make 0.049 and 0.051 different evidence. They sit either side of an agreed line, nothing more, which is why the p-value gets reported alongside the decision rather than swallowed by it.

Say which lever you mean. Holding $n$, the true effect, and the spread fixed, lowering $\alpha$ lowers power: in the test above, $\alpha$ of 0.10, 0.05, and 0.01 gives power 0.6499, 0.5087, and 0.2547 against a true mean of 105. That is a trade between the two error rates, not a law binding power to $\alpha$, because raising $n$ from 25 to 100 lifts power to 0.9543 with $\alpha$ still 0.05. AP Statistics covers the errors this threshold governs in topic 3.8, Potential Errors When Performing Tests.

Power is $1 - \beta$ (beta), where $\beta$ is the probability of a Type II error, and both numbers are computed while assuming one specific alternative value is the truth. A test therefore does not have a power. It has one power for every value the parameter could take, and a power quoted without naming that value is not a number anyone can use.

Take the z test behind the site's power visualizer: $H_0: \mu = 100$ against $H_a: \mu > 100$, with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is $15/\sqrt{25} = 3$. At $\alpha = 0.05$ the test rejects once $\bar{x}$ (x-bar) clears $100 + 1.645(3) = 104.93$. If the true mean is 105, sample means center at 105 with that same standard error of 3, and the chance of clearing 104.93 is 0.5087. That is the power against $\mu = 105$, near a coin flip.

The misreading to kill: "power is 0.5087, so there is about a 51 percent chance the null hypothesis is false." Power says nothing about whether $H_0$ is false. It assumes a stated alternative is already true and reports how often the test would notice. It is also not the probability that this particular conclusion is correct, since it is a long-run rate for a procedure aimed at one alternative.

Two boundaries are worth holding. Against $\mu = 100$, where the null is true, power equals $\alpha$ exactly, 0.0500, because the only rejections left are Type I errors. And power can sit below $\alpha$: this one-sided test looks upward only, so against a true mean of 95 its power is 0.0005.

Name the lever before claiming a tradeoff. Raising $n$ from 25 to 100 takes power against $\mu = 105$ from 0.5087 to 0.9543 and leaves $\alpha$ at 0.05. Raising $\alpha$ from 0.05 to 0.10 takes power to 0.6499 and doubles the Type I error rate in the same move. AP Statistics covers power in topic 3.8, Potential Errors When Performing Tests.