Research Methods
ANCOVA / MANOVA Decision Laboratory
Before either of these analyses is a calculation, it is a decision: what is this variable for? Assign the roles, watch a covariate adjustment happen — and watch it stop meaning what people think it means.
Simulated — fictional studies and generated data
Learning objective
By the end you should be able to say what a covariate buys you, and why it is precision rather than equivalence.
About 30 minutes. Nothing you do here is saved or sent anywhere.
Neither analysis rescues a design
A covariate cannot turn two intact groups into a randomised trial, and a multivariate test is not a way to run several analyses without paying for them. Both are useful tools with narrow jobs, and most of the harm they do comes from being reached for as repairs. Several of the scenarios in this laboratory have the answer "neither", and saying so is the skill being practised.
- Assign the roles in the opening scenario and choose an analysis.
- In Experiment 1, adjust for a covariate and watch the adjustment move.
- Break the equal-slopes assumption and see what the adjusted difference becomes.
- In Experiment 2, compare two outcomes one at a time with the two of them together.
First, a decision
Scenario — the September test
A fictional evaluation randomly assigns 80 students either to a new tutoring programme or to the usual timetable. Everyone sits a baseline test in September, before allocation, and an outcome test in June. The question is whether the programme raised June scores.
Experiment 1 — what adjustment actually does
Eighty fictional students, forty in each group. The horizontal axis is the September score, the vertical axis is the June score, and each group has its own regression line. The adjusted difference is the vertical gap between those lines, measured at the overall September mean.
Key terms
- Covariate
- A variable measured before the outcome and used to sharpen the comparison. Sharpening it is not the same as making the groups equivalent.
- Adjusted difference
- The group difference the model estimates at a common covariate value. It answers a conditional question.
- Equal slopes
- The assumption that the covariate relates to the outcome the same way in each group. Without it there is no single adjusted difference to report.
- MANOVA
- Testing two or more outcomes jointly rather than one at a time. What it adds depends on how the outcomes correlate.
The covariate adjustment
Adjustment asks a conditional question: how far apart would the groups be if they had started at the same place? Whether that question has an answer depends on the design, not on the software.
September against June
| Group | September mean | June mean | Own slope |
|---|
What this adjusted number means
The adjusted difference at three different starting points
| Evaluated at | September score | Gap between the lines |
|---|
Experiment 2 — two outcomes, taken together
The same two fictional groups, now measured on two outcomes at once. Each dot is one person, placed by both scores. A multivariate test asks whether the two clouds are separated in the plane, which is not the same question as asking about each axis in turn.
Why this is not several ANOVAs
Watch the joint separation while the two separate differences stay exactly where they are. The correlation between the outcomes is doing all the work.
The two groups in the plane
| Comparison | Separation, in SD units |
|---|
What the multivariate test has added
Challenge — four studies, four decisions
One choice per study. More than one of these has the answer "neither".
What this demonstrates
Adjusting for a covariate answers a conditional question: how far apart would the groups be if they had started level? With random allocation that question has an answer, and adjustment mostly buys precision. With intact groups it may have none, because the thing you are adjusting away could be the difference itself. The arithmetic is identical either way, which is what makes it dangerous.
For teaching elsewhere: take this activity as one self-contained block of HTML, on the clipboard or as a file. Either way it is styled so that it will not disturb the page you put it into.