Research Methods · Simplified
Why It Is Called Least Squares
The squares are not a metaphor. Move the line by hand and watch the area you are trying to get rid of.
Your turn
Fit the line yourself first
Twelve observations, and a line you control with two numbers. Every observation gets a square whose side is how far the line missed it. Make the total area as small as you can, then find out how close you got.
Key terms
- Intercept
- Where the line crosses the vertical axis: the predicted value of Y when X is zero.
- Slope
- How much the predicted Y changes for a one-unit rise in X.
- Residual
- The vertical distance from an observation to the line. The side of its square.
- R squared
- How much better your line does than simply predicting the average of Y for everybody. It is 0 for a line no better than the average and it can go negative for one that is worse.
Each square has its side equal to that observation's residual, so its area is the residual squared. Adding up those twelve areas gives the number below. Regression is the line that makes that total as small as it can possibly be.
What this shows
One line, and no argument about it
There is exactly one pair of numbers that makes the total area smallest, and it can be worked out directly rather than searched for. That is all a regression does.
Key idea: Squaring the residuals is a choice, not a law, and it has consequences. It makes the arithmetic solvable in one step, and it means an observation twice as far from the line contributes four times as much to the total, so the fitted line is pulled hard towards distant points. R squared then compares your line with the laziest possible prediction, the average of Y for everybody: it is 0 for a line that does no better than that average, and negative for one that does worse, which is worth seeing at least once because it makes clear that R squared is a comparison rather than a percentage of something absolute.
What the best-fitting line is not. It is not evidence that a straight line is the right model: a line can be fitted to anything, including data with an obvious curve, and least squares will happily return one. It is not a causal claim about X and Y. And a good fit here says nothing about prediction for new cases, because the line was chosen to suit these twelve observations specifically and will do somewhat worse on any others. Note also that the intercept is only meaningful when X of zero is a value that could actually occur.
The longer version adds a prediction exercise, a numeric target challenge and the standard error of the estimate. It is at Regression: Slope and Intercept in the main collection.