Null Hypothesis vs P-Value

The null hypothesis, written $H_0$ and read H-naught (or H-zero), is the skeptical claim that nothing is going on: no effect, no difference, no association. It is a statement about the population, never about the sample in hand. When it concerns a parameter such as $p$ or $\mu$ (mu) it pins that parameter to a single number, which is why it carries an equals sign while the alternative $H_a$ carries $<$, $>$ or $\ne$.

That single number is what makes a test possible. Suppose you flip a coin 100 times to test whether it is fair. Setting $H_0: p = 0.5$ fixes the sampling distribution of $\hat{p}$ (p hat) completely: it is centered at 0.5 with standard deviation $\sqrt{\frac{0.5(0.5)}{100}} = 0.05$. An alternative such as $H_a: p \ne 0.5$ names no particular value, so no distribution can be built from it. Every test compares the observed data against a distribution built from the null, which is why the null has to be specific enough to determine one, and that is the whole content of the phrase assume the null is true.

Stating the hypothesis about the wrong quantity is the most common setup error. $H_0: \hat{p} = 0.5$ is not a hypothesis, because $\hat{p}$ is something you measured rather than something unknown. You already know what it equals; there is nothing left to test.

The second error lives in the conclusion. A test either rejects $H_0$ or fails to reject it, and failing to reject is not the same as accepting. A large p-value says the data are consistent with $H_0$, but they are also consistent with a whole range of nearby parameter values, which is precisely what a confidence interval puts on display. Rejection is not proof either: a Type I error is the case where a true null gets rejected, and its long-run rate is the significance level $\alpha$ (alpha) chosen before the data arrive. The null is a benchmark to measure evidence against, not a claim the test ever settles.

A p-value is a conditional probability computed while the null hypothesis is taken as true. Written out, it is $P(\text{a result at least as extreme as the observed one} \mid H_0 \text{ is true})$, where extreme means in whichever direction the alternative hypothesis points.

Suppose you flip a coin 100 times, get 60 heads, and test $H_0: p = 0.5$ against $H_a: p \ne 0.5$. Under the null, $\hat{p}$ (p hat) is centered at 0.5 with standard deviation 0.05, so the observed $\hat{p} = 0.60$ sits $z = \frac{0.60 - 0.50}{0.05} = 2.00$ standard deviations out. Under the normal model the z-test uses, the two-sided p-value is 0.0455. Said aloud: if the coin really were fair, that model puts about 4.6 percent of samples of 100 flips at least 0.10 away from 0.5 in one direction or the other. Against the one-sided $H_a: p > 0.5$ the same model gives 0.0228, half as much, because only the upper tail counts.

Two readings of that 0.0455 are wrong and both are everywhere. It is not the probability that the coin is fair, and it is not the probability that the result happened by chance. Both of those are claims about the hypothesis, while the p-value is a claim about the data that takes the hypothesis as given. The conditioning runs one way only, and reversing it is the single most repeated error in applied statistics.

A large p-value is not evidence for $H_0$ either. It means the data would not be surprising if $H_0$ were true, which is a much weaker statement than $H_0$ being true. Small samples have little power to detect anything, so a large p-value from one may mean only that the test could not have found an effect of ordinary size, which is why a p-value of 0.4 from 12 observations tells you very little.

The decision comes from comparing the p-value against a significance level $\alpha$ (alpha) fixed before the data are seen: reject $H_0$ when the p-value is at or below $\alpha$, and fail to reject it otherwise. The p-value itself is a measure of evidence on a continuous scale, so 0.049 and 0.051 carry nearly identical information despite falling on opposite sides of the usual cutoff.