Research Methods · Simplified
What a Covariate Can and Cannot Buy
Adjusting for a covariate changes the number. Only the design decides what the number is allowed to mean.
The evaluation
A reading programme, and a score from September
Forty children take a reading programme and forty do not. Everyone was scored in September, before anything began, and again in June. The June difference is the headline. The September score is the covariate.
Key terms
- Covariate
- Something measured before allocation that is related to the outcome. Here, the September score.
- Unadjusted difference
- The gap between the two groups' June means, with nothing taken into account.
- Adjusted difference
- The gap between the groups once both are compared at the same September score. On the figure it is the vertical distance between the two fitted lines, which is the same wherever you measure it.
- Precision
- How tightly the estimate is pinned down, reported here as the standard error. Smaller is better, and it is not the same thing as the estimate being right.
One mark per child: filled circles are the programme group, hollow squares the comparison group. Each group gets a fitted line of the same slope, and the large ringed marks are the two group centres. The bracket is the vertical distance between the lines, which is the adjusted difference.
Switching between these changes no arithmetic whatsoever. Watch every figure above as you do it.
What this shows
The same arithmetic, two different warrants
Switch the design between random allocation and two intact classes. Not one number on the screen changes. What the adjusted difference is allowed to mean changes completely.
Key idea: Under random allocation the groups start level apart from chance, so adjustment is not correcting anything. What it does is remove the part of June that September already explains, which shrinks the leftover variation and buys precision: the same estimate, pinned down more tightly. With two intact classes the adjustment does exactly the same arithmetic, but it can only remove the part of the difference that September accounts for. Anything else that differs between two classes, and something always does, is still sitting inside the adjusted number. Adjustment is a way of asking a narrower question, not a way of turning a comparison into an experiment.
This simulation is generous to the intact-classes case: here September genuinely is the only thing that differs between the groups, because that is how the data were built. In a real pair of intact classes it will not be, and you have no way of checking. That is the whole difficulty, and no amount of adjusting reaches it. Two further points. The adjusted difference assumes one common slope for both groups; if the covariate works differently in the two groups there is no single adjusted figure to report, and the difference of slopes is itself the finding. And precision is not accuracy: turning the covariate strength up tightens the estimate whether or not the estimate is aimed at the right thing.
The longer version adds a variable-role assignment stage, a second experiment on multivariate tests and what they are not, a slope-difference violation to diagnose, and several described designs to classify. It is at ANCOVA / MANOVA Decision Laboratory in the main collection.