Linear Model vs Least-Squares Regression Line

A linear model claims that a response is well described by a straight line plus scatter around it. It is written $\hat{y} = a + bx$, where $\hat{y}$ (read y-hat) is the value the line returns, $a$ is the y-intercept, $b$ is the slope, and each observed $y$ differs from $\hat{y}$ by a residual. Any straight line through a scatterplot is a linear model; the least-squares regression line is the particular one that minimizes the sum of the squared residuals.

Six students study 1, 2, 3, 4, 5 and 6 hours and score 60, 72, 65, 78, 71 and 86, and the least-squares model is $\hat{y} = 58 + 4x$. Drop the hat and write $y = 58 + 4x$ and the equation becomes false for all six of them: it asserts scores of 62, 66, 70, 74, 78 and 82 when the students actually scored 60, 72, 65, 78, 71 and 86. The hat is what turns a false equation into an honest prediction.

So the misreading to name is "the data follow the equation." They do not, and no data ever do. A linear model is a summary of the pattern, and the leftover scatter is part of the description rather than a defect in it. A second version of the same slip is treating the model as a fact about the world instead of about these observations: refit on a different sample and $a$ and $b$ come out different.

Nothing in the arithmetic checks whether a line belongs on the data. Feed a clean parabola into a regression routine and it returns a slope, an intercept and a correlation without complaint. The form of the scatterplot and the shape of the residual plot are what justify the model, and the model speaks only for the range of $x$ the data covered.

Linear regression models are topic 5.3 in Unit 5, Regression Analysis.

The least-squares regression line is the one line that the least-squares criterion selects out of all the straight lines you could draw: the line minimizing $\sum (y - \hat{y})^2$, the total of the squared vertical residuals. It is written $\hat{y} = a + bx$ (read y-hat for $\hat{y}$), and its two coefficients come from the summary statistics, with slope $b = r \cdot \frac{s_y}{s_x}$ and intercept $a = \bar{y} - b\bar{x}$, where $\bar{x}$ is x-bar, the mean of the explanatory values.

Six students study 1, 2, 3, 4, 5 and 6 hours and score 60, 72, 65, 78, 71 and 86. Here $\bar{x} = 3.5$, $\bar{y} = 72$, $s_x = 1.8708$, $s_y = 9.2304$ and $r = 0.8107$, so $b = 0.8107 \times \frac{9.2304}{1.8708} = 4.00$ points per hour and $a = 72 - 4(3.5) = 58$. The fitted line is $\hat{y} = 58 + 4x$, its residuals are -2, 6, -5, 4, -7 and 4, and their squares total 146. No other straight line drives that total below 146.

The picture to correct is the one where the best-fit line threads through as many points as it can. This line passes through none of the six, and its smallest miss is 2 points. The point it does pass through is the point of averages $(\bar{x}, \bar{y}) = (3.5, 72)$, since $58 + 4(3.5) = 72$, and that holds for every least-squares line because it is the intercept formula rearranged. Quality of fit is a statement about all the residuals at once, never about how many points were hit.

Two limits come with the line. The formulas return one as long as the $x$-values are not all identical, curved data included, so the existence of a line is no evidence that a line belongs on the data. And squaring the misses leaves the fit with no resistance, so a single observation far out in $x$ can move $b$ a long way.

Least-squares regression is topic 5.5 in Unit 5, Regression Analysis.

A physics lab runs this procedure almost every week. Data is rearranged so the relationship plots as a straight line, a least-squares line is fitted to it, and a physical constant is read off the slope: linearization.