Least-squares regression line

The least-squares regression line is the one line that the least-squares criterion selects out of all the straight lines you could draw: the line minimizing $\sum (y - \hat{y})^2$, the total of the squared vertical residuals. It is written $\hat{y} = a + bx$ (read y-hat for $\hat{y}$), and its two coefficients come from the summary statistics, with slope $b = r \cdot \frac{s_y}{s_x}$ and intercept $a = \bar{y} - b\bar{x}$, where $\bar{x}$ is x-bar, the mean of the explanatory values.

Six students study 1, 2, 3, 4, 5 and 6 hours and score 60, 72, 65, 78, 71 and 86. Here $\bar{x} = 3.5$, $\bar{y} = 72$, $s_x = 1.8708$, $s_y = 9.2304$ and $r = 0.8107$, so $b = 0.8107 \times \frac{9.2304}{1.8708} = 4.00$ points per hour and $a = 72 - 4(3.5) = 58$. The fitted line is $\hat{y} = 58 + 4x$, its residuals are -2, 6, -5, 4, -7 and 4, and their squares total 146. No other straight line drives that total below 146.

The picture to correct is the one where the best-fit line threads through as many points as it can. This line passes through none of the six, and its smallest miss is 2 points. The point it does pass through is the point of averages $(\bar{x}, \bar{y}) = (3.5, 72)$, since $58 + 4(3.5) = 72$, and that holds for every least-squares line because it is the intercept formula rearranged. Quality of fit is a statement about all the residuals at once, never about how many points were hit.

Two limits come with the line. The formulas return one as long as the $x$-values are not all identical, curved data included, so the existence of a line is no evidence that a line belongs on the data. And squaring the misses leaves the fit with no resistance, so a single observation far out in $x$ can move $b$ a long way.

Least-squares regression is topic 5.5 in Unit 5, Regression Analysis.

A physics lab runs this procedure almost every week. Data is rearranged so the relationship plots as a straight line, a least-squares line is fitted to it, and a physical constant is read off the slope: linearization.