Type I Error vs P-Value

A Type I error requires two conditions at once: $H_0$ is true, and the test rejects it anyway. Its long-run rate under a true null is exactly the significance level $\alpha$ (alpha) you chose, which makes it the one error probability you set directly rather than inherit.

Take $H_0: \mu = 100$ against $H_a: \mu > 100$, with $\sigma$ (sigma) known to be 15 and $n = 25$, so the standard error is 3 and an $\alpha = 0.05$ test rejects once $\bar{x}$ (x-bar) clears 104.93. If the true mean really is 100, that happens in 5 percent of samples. Those 5 percent are Type I errors, and there is nothing wrong with the data or the arithmetic in any of them: the sample was unlucky, not mistaken. Drop $\alpha$ to 0.01 and the cutoff moves to 106.98, so the rate falls to 1 percent.

The sentence to retire: "my p-value was 0.03, so there is a 3 percent chance this rejection is a Type I error." Once the decision is made, either $H_0$ is true and you erred or it is false and you did not; the coin has already landed. $\alpha$ describes the procedure across many repetitions, not the single conclusion in front of you, and 0.03 is the p-value, computed under a null that may well be false.

The error is also unavailable when $H_0$ is false, no matter how badly the test performs. Rejecting a false null is the correct decision, and failing to reject it is a Type II error. Which of the two mistakes you could even make depends on a truth you never observe.

Lowering $\alpha$ cuts this error and, with $n$ and the true effect held fixed, raises the other: at $\alpha = 0.01$ the test above has $\beta = 0.7453$ against a true mean of 105, where $\alpha = 0.05$ gave 0.4913. A larger sample cuts both at once. Which error costs more is a judgment about consequences, and AP Statistics topic 3.8, Potential Errors When Performing Tests, asks for that judgment in context.

A p-value is a conditional probability computed while the null hypothesis is taken as true. Written out, it is $P(\text{a result at least as extreme as the observed one} \mid H_0 \text{ is true})$, where extreme means in whichever direction the alternative hypothesis points.

Suppose you flip a coin 100 times, get 60 heads, and test $H_0: p = 0.5$ against $H_a: p \ne 0.5$. Under the null, $\hat{p}$ (p hat) is centered at 0.5 with standard deviation 0.05, so the observed $\hat{p} = 0.60$ sits $z = \frac{0.60 - 0.50}{0.05} = 2.00$ standard deviations out. Under the normal model the z-test uses, the two-sided p-value is 0.0455. Said aloud: if the coin really were fair, that model puts about 4.6 percent of samples of 100 flips at least 0.10 away from 0.5 in one direction or the other. Against the one-sided $H_a: p > 0.5$ the same model gives 0.0228, half as much, because only the upper tail counts.

Two readings of that 0.0455 are wrong and both are everywhere. It is not the probability that the coin is fair, and it is not the probability that the result happened by chance. Both of those are claims about the hypothesis, while the p-value is a claim about the data that takes the hypothesis as given. The conditioning runs one way only, and reversing it is the single most repeated error in applied statistics.

A large p-value is not evidence for $H_0$ either. It means the data would not be surprising if $H_0$ were true, which is a much weaker statement than $H_0$ being true. Small samples have little power to detect anything, so a large p-value from one may mean only that the test could not have found an effect of ordinary size, which is why a p-value of 0.4 from 12 observations tells you very little.

The decision comes from comparing the p-value against a significance level $\alpha$ (alpha) fixed before the data are seen: reject $H_0$ when the p-value is at or below $\alpha$, and fail to reject it otherwise. The p-value itself is a measure of evidence on a continuous scale, so 0.049 and 0.051 carry nearly identical information despite falling on opposite sides of the usual cutoff.