AP Stats 3.8: Type I and Type II Errors

What topic 3.8 covers

Topic 3.8 examines the two ways a hypothesis test can reach a wrong decision. Because a test decides based on sample data, it can point to the alternative when the null is actually true, or miss the alternative when it is actually true. This topic names those errors, defines power, and lists what changes their probabilities. It applies to every significance test in the course, not just tests for proportions.

The two errors and power

A Type I error happens when the test finds convincing evidence for the alternative (a small p-value) but the alternative is not true. A Type II error happens when the test does not find convincing evidence (a large p-value) but the alternative is true. The power of a test is the probability it correctly rejects a false null. The table below pairs each decision with reality.

Probabilities of the errors

The probability of a Type I error equals the significance level $\alpha$, which you set before collecting data, often 0.01, 0.05, or 0.10. The probability of a Type II error is $1 - \text{power}$. A well-designed study aims for a small Type II error probability, and therefore high power, often at least 0.80. Power is always measured against a specific alternative value, because a false null that is far from the truth is easier to detect than one that is barely false.

What raises power

Holding everything else fixed, the Type II error probability falls and power rises when any one of the following happens:

• The sample size increases.
• The standard error decreases.
• The true parameter is farther from the null value.
• The significance level $\alpha$ increases.

Because $\alpha$ is the Type I error probability, and sample size drives the Type II error probability, the real-world consequences of each error should guide how you choose $\alpha$ and how large a sample to collect. When a Type I error is costly, use a smaller $\alpha$; when a Type II error is costly, collect a larger sample to raise power.