Research Methods
Two numbers define a line: where it starts and how steeply it climbs. Move them by hand, watch every residual it creates, and try to beat the line the arithmetic would have chosen.
Simulated — one generated dataset from a documented seed
By the end you should be able to say what least squares minimises, and why the intercept often describes nobody in the data.
About 25 minutes. Nothing you do here is saved or sent anywhere.
The model predicts. It does not explain and it does not cause. A slope of 0.6 says that two people differing by one unit of x differ, on average, by 0.6 units of y — not that changing one person's x by a unit would change their y. And "predicting" beyond the range of x actually observed is arithmetic rather than evidence, which is why the region outside the data is shaded on the picture.
You are about to be given thirty fictional observations and a line you can move by hand.
Thirty fictional people. The horizontal axis is minutes of daily practice, the vertical axis is a score out of 100, and the vertical line from each point to your line is its residual: what the person actually scored minus what your line predicted.
Every position of the two sliders produces thirty residuals. The sum of their squares is the one number the fitting procedure cares about.
| Quantity | Your line | Least squares |
|---|
| Quantity | x as measured | x centred on its mean |
|---|
Use the least-squares line you have just revealed. Two predictions and one decision.
Every point in the picture has a vertical distance to the line, and that distance is the part of the person the line failed to capture. Positive above the line, negative below it. The sum of those residuals is zero for any least-squares line, which is why minimising the plain sum would be useless: a line that missed wildly in both directions would score just as well as a perfect one. Squaring removes the cancellation and, incidentally, makes one large miss cost far more than several small ones.
For any dataset there is exactly one pair of values that minimises the squared total, and it can be written down directly: the slope is the covariance of x and y divided by the variance of x, and the intercept is whatever puts the line through the point (mean of x, mean of y). No amount of dragging beats it, because there is nothing to beat — the arithmetic has already been done to the smallest number available.
On this dataset nobody practised for less than about ten minutes, so the intercept is a prediction for a person who does not appear anywhere in the data. It is a mathematically necessary anchor for the line and a substantively empty number. Centring x on its mean fixes that at no cost: the slope, every prediction and every residual stay exactly where they were, and the intercept becomes the predicted score for someone with an average amount of practice — a number that describes a real region of the data.
It is the share of the variance in y that the line accounts for, and it is the same quantity as the squared correlation. A high R² means the points sit close to the line; it says nothing about whether the line's shape is right, whether the relationship is causal, or whether the model would predict anything about a new person. A perfect R² on a curved relationship fitted with a straight line is impossible, and a high one on a spurious association is entirely routine.