Research Methods · Simplified
Confound Detective
Something is wrong with this study. Work out what, then fix it and watch the number move.
Sort the variables
Using the revision app improves exam performance
Fictional study. Sixty students. The Tuesday 09:00 seminar group is given the app; the Thursday 14:00 group is not. Students chose their seminar group at enrolment. The Tuesday group is taught by the module leader, the Thursday group by a postgraduate tutor. Both sit the same exam, marked by the module leader.
Key terms
- Confound
- Something that differs systematically between the conditions and also touches the outcome. Both have to be true. It shifts the estimate away from the truth.
- Nuisance variable
- Something that varies between people but not with the condition. It adds scatter, which widens the interval and leaves the estimate where it was.
- Part of the treatment
- Something that is part of what the manipulation is. Removing it deletes the effect rather than cleaning up the comparison.
- Bias and precision
- Bias is being systematically off. Precision is how tightly repeated studies would cluster. More participants buys precision and buys no bias back at all.
The solid line is the true effect, which the study cannot see. The marker is what this design would report, with the interval it would report around it.
What moved and what did not
Bias and noise are different problems
Key idea: A variable biases a comparison only if it travels with the conditions and touches the outcome. Both have to be true. Things that vary between people but not between conditions add scatter, not bias: they widen the interval and leave the estimate where it was. And recruiting more participants narrows the interval around a wrong number, which is the most common student answer to a bias problem and the wrong tool for it.
The bias figures are invented and were chosen to make the argument legible, not to estimate confounding in any real study. Real confounds interact, point in opposite directions and sometimes cancel; keeping them additive here is a deliberate simplification so that the arithmetic can be followed. No real module, tutor, student or dataset is described, and the true effect is knowable here only because the activity invented it.
The longer version adds two further studies, a fuller repair set and a select-all challenge on what each repair buys. It is at Confound Detective in the main collection.