Least-Squares Regression Line vs Slope of a Regression Line

The least-squares regression line is the one line that the least-squares criterion selects out of all the straight lines you could draw: the line minimizing $\sum (y - \hat{y})^2$, the total of the squared vertical residuals. It is written $\hat{y} = a + bx$ (read y-hat for $\hat{y}$), and its two coefficients come from the summary statistics, with slope $b = r \cdot \frac{s_y}{s_x}$ and intercept $a = \bar{y} - b\bar{x}$, where $\bar{x}$ is x-bar, the mean of the explanatory values.

Six students study 1, 2, 3, 4, 5 and 6 hours and score 60, 72, 65, 78, 71 and 86. Here $\bar{x} = 3.5$, $\bar{y} = 72$, $s_x = 1.8708$, $s_y = 9.2304$ and $r = 0.8107$, so $b = 0.8107 \times \frac{9.2304}{1.8708} = 4.00$ points per hour and $a = 72 - 4(3.5) = 58$. The fitted line is $\hat{y} = 58 + 4x$, its residuals are -2, 6, -5, 4, -7 and 4, and their squares total 146. No other straight line drives that total below 146.

The picture to correct is the one where the best-fit line threads through as many points as it can. This line passes through none of the six, and its smallest miss is 2 points. The point it does pass through is the point of averages $(\bar{x}, \bar{y}) = (3.5, 72)$, since $58 + 4(3.5) = 72$, and that holds for every least-squares line because it is the intercept formula rearranged. Quality of fit is a statement about all the residuals at once, never about how many points were hit.

Two limits come with the line. The formulas return one as long as the $x$-values are not all identical, curved data included, so the existence of a line is no evidence that a line belongs on the data. And squaring the misses leaves the fit with no resistance, so a single observation far out in $x$ can move $b$ a long way.

Least-squares regression is topic 5.5 in Unit 5, Regression Analysis.

A physics lab runs this procedure almost every week. Data is rearranged so the relationship plots as a straight line, a least-squares line is fitted to it, and a physical constant is read off the slope: linearization.

In $\hat{y} = a + bx$ the slope $b$ is the change in the predicted response for a one-unit increase in the explanatory variable. The word predicted is load-bearing: $b$ describes how the line moves, not what happens to any individual. The slope also carries units, always units of $y$ per unit of $x$, so a bare number is an incomplete answer.

Six students study 1, 2, 3, 4, 5 and 6 hours and score 60, 72, 65, 78, 71 and 86. The least-squares line is $\hat{y} = 58 + 4x$, and $b = 4$ points per hour. Said in context: for each additional hour of study, the model predicts a score about 4 points higher. It comes from $b = r \cdot \frac{s_y}{s_x}$, here $0.8107 \times \frac{9.2304}{1.8708}$, which is why $b$ and the correlation $r$ always share a sign but rarely share a size.

The interpretation that loses credit is "each extra hour of study raises a student's score by 4 points." It makes two claims the line cannot support. Raises is causal, and these are observational data where a student who studies more may differ in other ways. And it promises 4 points to an individual, while the six real students sit as much as 7 points off the line. Predicted change, not guaranteed change.

The slope is not symmetric in the two variables, which surprises people. Regress hours on score instead of score on hours and the slope is 0.1643 hours per point, not the reciprocal $1/4 = 0.25$. The two slopes multiply to $4 \times 0.1643 = 0.657$, which is $r^2$. The correlation, by contrast, is 0.8107 whichever variable you call explanatory. Rescaling changes the slope too: measure study time in minutes and $b$ becomes $4/60$ points per minute while $r$ does not budge.