Power

Power is $1 - \beta$ (beta), where $\beta$ is the probability of a Type II error, and both numbers are computed while assuming one specific alternative value is the truth. A test therefore does not have a power. It has one power for every value the parameter could take, and a power quoted without naming that value is not a number anyone can use.

Take the z test behind the site's power visualizer: $H_0: \mu = 100$ against $H_a: \mu > 100$, with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is $15/\sqrt{25} = 3$. At $\alpha = 0.05$ the test rejects once $\bar{x}$ (x-bar) clears $100 + 1.645(3) = 104.93$. If the true mean is 105, sample means center at 105 with that same standard error of 3, and the chance of clearing 104.93 is 0.5087. That is the power against $\mu = 105$, near a coin flip.

The misreading to kill: "power is 0.5087, so there is about a 51 percent chance the null hypothesis is false." Power says nothing about whether $H_0$ is false. It assumes a stated alternative is already true and reports how often the test would notice. It is also not the probability that this particular conclusion is correct, since it is a long-run rate for a procedure aimed at one alternative.

Two boundaries are worth holding. Against $\mu = 100$, where the null is true, power equals $\alpha$ exactly, 0.0500, because the only rejections left are Type I errors. And power can sit below $\alpha$: this one-sided test looks upward only, so against a true mean of 95 its power is 0.0005.

Name the lever before claiming a tradeoff. Raising $n$ from 25 to 100 takes power against $\mu = 105$ from 0.5087 to 0.9543 and leaves $\alpha$ at 0.05. Raising $\alpha$ from 0.05 to 0.10 takes power to 0.6499 and doubles the Type I error rate in the same move. AP Statistics covers power in topic 3.8, Potential Errors When Performing Tests.