Confidence Level vs Significance Level

A confidence level is chosen before the data are collected and it does two jobs at once. It fixes the critical value the interval formula uses, and it names the rate at which that formula succeeds across repeated random samples of the same size. Write it $C$. The parameter never moves. The interval does, because a fresh sample produces fresh endpoints.

Suppose 240 of 400 randomly sampled adults say yes, so $\hat{p} = 0.60$ (p-hat) and the standard error is $\sqrt{\frac{0.60(0.40)}{400}} = 0.0245$. At 90% the critical value is $z^* = 1.645$ and the interval runs 0.560 to 0.640. At 95% it is $z^* = 1.960$ and the interval runs 0.552 to 0.648. At 99% it is $z^* = 2.576$ and the interval runs 0.537 to 0.663. One data set, three intervals. Nothing about the sample changed, only the level.

The sentence to stop writing is "there is a 95% probability that the true proportion is between 0.552 and 0.648." Once those two numbers exist they are fixed, the proportion was always fixed, and so that probability is 0 or 1 with no way to tell which. The randomness was spent when the sample was drawn. The 95% belongs to the procedure: it is a statement about the samples you did not take. The coverage simulator makes that visible, with the true $p$ drawn as a line and each new sample laying down one more interval to count.

A confidence level is what the method promises, not always what it delivers. For the proportion interval above the two come apart: at $n = 50$ with a true $p$ of 0.60, its exact long-run capture rate is 94.1% against a nominal 95%, because the standard error is built from $\hat{p}$ rather than from the unknown $p$. The rate a method actually achieves is its coverage probability.

Raising the level buys reliability with width and nothing else. At a fixed $n$ the 99% interval above is 0.126 wide against 0.096 at 95%, and a 100% interval would be every value from 0 to 1: certain to capture $p$ and worth nothing.

$\alpha$ (alpha) does two jobs with one number. It is the cutoff the p-value is compared against, and it is the probability the test rejects $H_0$ when $H_0$ is true. Those coincide because the rejection region is built to hold exactly that much area under the null distribution, which is also why $\alpha$ has to be chosen before the data arrive.

Take $H_0: \mu = 100$ against $H_a: \mu > 100$, with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is 3. Setting $\alpha = 0.05$ puts the cutoff at $100 + 1.645(3) = 104.93$: if the true mean really is 100, 5 percent of samples of 25 land above 104.93 through nothing but sampling variability. Set $\alpha = 0.01$ instead and the cutoff moves out to 106.98.

The misreading is subtle and common: "I test at $\alpha = 0.05$, so 5 percent of the results I call significant are wrong." It is not 5 percent of your rejections. It is 5 percent of the tests you run on null hypotheses that happen to be true. What fraction of your significant findings are false alarms depends on how often the nulls you test are true in the first place, and $\alpha$ carries no information about that.

A fixed threshold also does not make 0.049 and 0.051 different evidence. They sit either side of an agreed line, nothing more, which is why the p-value gets reported alongside the decision rather than swallowed by it.

Say which lever you mean. Holding $n$, the true effect, and the spread fixed, lowering $\alpha$ lowers power: in the test above, $\alpha$ of 0.10, 0.05, and 0.01 gives power 0.6499, 0.5087, and 0.2547 against a true mean of 105. That is a trade between the two error rates, not a law binding power to $\alpha$, because raising $n$ from 25 to 100 lifts power to 0.9543 with $\alpha$ still 0.05. AP Statistics covers the errors this threshold governs in topic 3.8, Potential Errors When Performing Tests.