Research Methods

ANCOVA / MANOVA Decision Laboratory

Before either of these analyses is a calculation, it is a decision: what is this variable for? Assign the roles, watch a covariate adjustment happen — and watch it stop meaning what people think it means.

Simulated — fictional studies and generated data

Learning objective

By the end you should be able to say what a covariate buys you, and why it is precision rather than equivalence.

About 30 minutes. Nothing you do here is saved or sent anywhere.

Neither analysis rescues a design

A covariate cannot turn two intact groups into a randomised trial, and a multivariate test is not a way to run several analyses without paying for them. Both are useful tools with narrow jobs, and most of the harm they do comes from being reached for as repairs. Several of the scenarios in this laboratory have the answer "neither", and saying so is the skill being practised.

Assign the roles in the opening scenario and choose an analysis.
In Experiment 1, adjust for a covariate and watch the adjustment move.
Break the equal-slopes assumption and see what the adjusted difference becomes.
In Experiment 2, compare two outcomes one at a time with the two of them together.

Scenario — the September test

A fictional evaluation randomly assigns 80 students either to a new tutoring programme or to the usual timetable. Everyone sits a baseline test in September, before allocation, and an outcome test in June. The question is whether the programme raised June scores.

This part needs JavaScript

The laboratory generates its data and recalculates its adjustments in your browser, so it cannot run with JavaScript switched off. The rest of this page is readable without it.

What role does each variable play?

June test score

Programme or usual timetable

September test score

Which analysis fits this design best? One-way ANOVA on June scores, ignoring September ANCOVA on June scores with September as a covariate MANOVA with September and June as two outcomes A t-test on the change from September to June

Key terms

Covariate: A variable measured before the outcome and used to sharpen the comparison. Sharpening it is not the same as making the groups equivalent.
Adjusted difference: The group difference the model estimates at a common covariate value. It answers a conditional question.
Equal slopes: The assumption that the covariate relates to the outcome the same way in each group. Without it there is no single adjusted difference to report.
MANOVA: Testing two or more outcomes jointly rather than one at a time. What it adds depends on how the outcomes correlate.

How the covariate behaves

How well September predicts June within a group. This is what decides whether a covariate is worth having at all.

How far apart the two groups already were in September. With random allocation this is near zero; with intact groups it is not.

The programme itself

How groups were formed

This changes nothing in the arithmetic. It changes what the adjusted difference is allowed to mean.

More of the design

ANCOVA assumes this is zero — the homogeneity-of-regression-slopes assumption.

September against June

Group means on the covariate and the outcome, and each group's own regression slope
Group	September mean	June mean	Own slope

What this adjusted number means

The adjusted difference at three different starting points

The vertical gap between the two group regression lines evaluated at a low, average and high September score
Evaluated at	September score	Gap between the lines

Group differences, one outcome at a time

In standard deviations.

How the two outcomes hang together

Not the correlation across the whole sample: the correlation among people in the same group, which is what the multivariate test uses.

Worked examples

The two groups in the plane

Separation between the groups on each outcome alone and on both together
Comparison	Separation, in SD units

What the multivariate test has added

Study 1 — two intact classes

The tutoring programme is given to one whole class and not to another. The classes were not randomly formed and differ by about nine points on the September test. The researchers want the causal effect of the programme on June scores.

Best analysis for Study 1

Study 2 — four reading measures

A randomised study of a reading intervention measures speed, accuracy, comprehension and enjoyment. The team argues that these four are facets of one construct and wants a single omnibus test of whether reading performance differs.

Best analysis for Study 2

Study 3 — three unrelated outcomes

A randomised study measures hours of sleep, an exam mark and self-reported height. The team plans a MANOVA "so that we only have to run one test and do not have to correct for multiple comparisons".

Best analysis for Study 3

Study 4 — the covariate measured afterwards

In a randomised trial of the tutoring programme, the team notices that students in the programme attended more sessions overall. They propose adding attendance as a covariate, "to control for engagement".

Best analysis for Study 4

Adjusting for a covariate answers a conditional question: how far apart would the groups be if they had started level? With random allocation that question has an answer, and adjustment mostly buys precision. With intact groups it may have none, because the thing you are adjusting away could be the difference itself. The arithmetic is identical either way, which is what makes it dangerous.

ANCOVA / MANOVA Decision Laboratory

Learning objective

Neither analysis rescues a design

First, a decision

Scenario — the September test

Experiment 1 — what adjustment actually does

The covariate adjustment

September against June

What this adjusted number means

Experiment 2 — two outcomes, taken together

Why this is not several ANOVAs