Least-Squares Regression Line vs Scatterplot

The least-squares regression line is the one line that the least-squares criterion selects out of all the straight lines you could draw: the line minimizing $\sum (y - \hat{y})^2$, the total of the squared vertical residuals. It is written $\hat{y} = a + bx$ (read y-hat for $\hat{y}$), and its two coefficients come from the summary statistics, with slope $b = r \cdot \frac{s_y}{s_x}$ and intercept $a = \bar{y} - b\bar{x}$, where $\bar{x}$ is x-bar, the mean of the explanatory values.

Six students study 1, 2, 3, 4, 5 and 6 hours and score 60, 72, 65, 78, 71 and 86. Here $\bar{x} = 3.5$, $\bar{y} = 72$, $s_x = 1.8708$, $s_y = 9.2304$ and $r = 0.8107$, so $b = 0.8107 \times \frac{9.2304}{1.8708} = 4.00$ points per hour and $a = 72 - 4(3.5) = 58$. The fitted line is $\hat{y} = 58 + 4x$, its residuals are -2, 6, -5, 4, -7 and 4, and their squares total 146. No other straight line drives that total below 146.

The picture to correct is the one where the best-fit line threads through as many points as it can. This line passes through none of the six, and its smallest miss is 2 points. The point it does pass through is the point of averages $(\bar{x}, \bar{y}) = (3.5, 72)$, since $58 + 4(3.5) = 72$, and that holds for every least-squares line because it is the intercept formula rearranged. Quality of fit is a statement about all the residuals at once, never about how many points were hit.

Two limits come with the line. The formulas return one as long as the $x$-values are not all identical, curved data included, so the existence of a line is no evidence that a line belongs on the data. And squaring the misses leaves the fit with no resistance, so a single observation far out in $x$ can move $b$ a long way.

Least-squares regression is topic 5.5 in Unit 5, Regression Analysis.

A physics lab runs this procedure almost every week. Data is rearranged so the relationship plots as a straight line, a least-squares line is fitted to it, and a physical constant is read off the slope: linearization.

A scatterplot shows one point $(x, y)$ per individual, with the explanatory variable on the horizontal axis and the response variable on the vertical. Both numbers must come from the same individual: two lists of equal length measured on different units do not make a scatterplot. You read it for direction, form, strength, and any point that departs from the pattern.

Five students study 1, 2, 3, 4 and 5 hours and score 62, 68, 74, 76 and 85. The cloud rises from left to right and looks close to linear. The least-squares line is $\hat{y} = 56.8 + 5.4x$ (y-hat) and the correlation is $r = 0.986$.

The misreading is "the points look tight, so $r$ must be about 0.99." Strength cannot be read off reliably by eye, because how tight the cloud looks depends on the axis scaling and $r$ does not. Multiply every score by 3 and the plot stretches into a steeper, apparently tighter band, and $r$ is still 0.9859 to four places. Record hours in minutes instead and the plot flattens, and $r$ is unchanged again. Correlation is computed from standardized values, so multiplying either variable by a positive constant, or adding one to it, leaves the number alone while changing the picture completely.

A scatterplot also cannot report how many observations sit at one spot. Add four more students who each studied 3 hours and scored 74 and the plot looks identical, one dot at $(3, 74)$, while $r$ moves from 0.986 to 0.982. Overplotting like that can bury a whole subgroup in a dense region.

A correlation of zero is not the same as no relationship. The five points $(-2, 4)$, $(-1, 1)$, $(0, 0)$, $(1, 1)$ and $(2, 4)$ lie exactly on $y = x^2$ and give $r = 0$. The scatterplot shows that curve plainly, which is the reason to look at it before trusting any single summary number. Scatterplots are topic 5.1.