Significance Level vs P-Value

$\alpha$ (alpha) does two jobs with one number. It is the cutoff the p-value is compared against, and it is the probability the test rejects $H_0$ when $H_0$ is true. Those coincide because the rejection region is built to hold exactly that much area under the null distribution, which is also why $\alpha$ has to be chosen before the data arrive.

Take $H_0: \mu = 100$ against $H_a: \mu > 100$, with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is 3. Setting $\alpha = 0.05$ puts the cutoff at $100 + 1.645(3) = 104.93$: if the true mean really is 100, 5 percent of samples of 25 land above 104.93 through nothing but sampling variability. Set $\alpha = 0.01$ instead and the cutoff moves out to 106.98.

The misreading is subtle and common: "I test at $\alpha = 0.05$, so 5 percent of the results I call significant are wrong." It is not 5 percent of your rejections. It is 5 percent of the tests you run on null hypotheses that happen to be true. What fraction of your significant findings are false alarms depends on how often the nulls you test are true in the first place, and $\alpha$ carries no information about that.

A fixed threshold also does not make 0.049 and 0.051 different evidence. They sit either side of an agreed line, nothing more, which is why the p-value gets reported alongside the decision rather than swallowed by it.

Say which lever you mean. Holding $n$, the true effect, and the spread fixed, lowering $\alpha$ lowers power: in the test above, $\alpha$ of 0.10, 0.05, and 0.01 gives power 0.6499, 0.5087, and 0.2547 against a true mean of 105. That is a trade between the two error rates, not a law binding power to $\alpha$, because raising $n$ from 25 to 100 lifts power to 0.9543 with $\alpha$ still 0.05. AP Statistics covers the errors this threshold governs in topic 3.8, Potential Errors When Performing Tests.

A p-value is a conditional probability computed while the null hypothesis is taken as true. Written out, it is $P(\text{a result at least as extreme as the observed one} \mid H_0 \text{ is true})$, where extreme means in whichever direction the alternative hypothesis points.

Suppose you flip a coin 100 times, get 60 heads, and test $H_0: p = 0.5$ against $H_a: p \ne 0.5$. Under the null, $\hat{p}$ (p hat) is centered at 0.5 with standard deviation 0.05, so the observed $\hat{p} = 0.60$ sits $z = \frac{0.60 - 0.50}{0.05} = 2.00$ standard deviations out. Under the normal model the z-test uses, the two-sided p-value is 0.0455. Said aloud: if the coin really were fair, that model puts about 4.6 percent of samples of 100 flips at least 0.10 away from 0.5 in one direction or the other. Against the one-sided $H_a: p > 0.5$ the same model gives 0.0228, half as much, because only the upper tail counts.

Two readings of that 0.0455 are wrong and both are everywhere. It is not the probability that the coin is fair, and it is not the probability that the result happened by chance. Both of those are claims about the hypothesis, while the p-value is a claim about the data that takes the hypothesis as given. The conditioning runs one way only, and reversing it is the single most repeated error in applied statistics.

A large p-value is not evidence for $H_0$ either. It means the data would not be surprising if $H_0$ were true, which is a much weaker statement than $H_0$ being true. Small samples have little power to detect anything, so a large p-value from one may mean only that the test could not have found an effect of ordinary size, which is why a p-value of 0.4 from 12 observations tells you very little.

The decision comes from comparing the p-value against a significance level $\alpha$ (alpha) fixed before the data are seen: reject $H_0$ when the p-value is at or below $\alpha$, and fail to reject it otherwise. The p-value itself is a measure of evidence on a continuous scale, so 0.049 and 0.051 carry nearly identical information despite falling on opposite sides of the usual cutoff.