Research Methods

Multiple Comparisons, FWER and Forking Paths

Run enough tests on nothing and something will come out significant. The uncomfortable part is that you do not have to run them deliberately, or even know that you did.

Simulated — generated data containing no effect at all

Learning objective

By the end you should be able to say what family a family-wise error rate belongs to, and how analytic flexibility inflates false positives without anyone cheating.

About 30 minutes. Nothing you do here is saved or sent anywhere.

This is not a technique

The second experiment lets you search a fictional dataset until something comes out significant. It is there to show you what that search does to the meaning of the number you end up with, and every choice it offers is one a careful researcher might make in good faith. Nobody in this laboratory is cheating. That is exactly why the problem is hard, and why the answer is transparency about what was decided when, rather than better intentions.

Answer the prediction question to open Experiment 1.
Run a family of tests where every null is true, and count the hits.
Apply a Bonferroni correction and read both of its consequences.
In Experiment 2, analyse one empty dataset until something works.

This part needs JavaScript

The experiments are generated in your browser, so they cannot run with JavaScript switched off. The rest of this page is readable without it.

Key terms

Family: The set of tests a claim is being made across. The error rate here is a property of that set, not of any test inside it.
Family-wise error rate: The probability of at least one false positive somewhere in the family, when every null in it is true.
Bonferroni correction: Dividing α by the number of tests. It has a second consequence as well as the intended one.
Forking paths: The many analysis decisions available, each defensible on its own, which between them produce many possible results from one dataset.

The family

The decision rule

Correction

Read both consequences in the readout, not just the one the correction is advertised for.

Change the family and the rule

The rest have nothing to find. Any real effects are of a fixed size, d = 0.60 with 40 per group.

Per-test α

Random seed

Nothing run yet

What happened in the most recent experiment, split by whether each test had anything to find
In the most recent experiment	Count

Reading this correctly

Theory beside simulation

Quantities the formula predicts and the same quantities measured across the simulated experiments
Quantity	Predicted	Simulated

Experiment 2 — the garden of forking paths

One fictional dataset: ninety students, randomly assigned to two versions of a revision task, measured on four outcomes, with a baseline test, a response-speed measure and a cohort label. The data were generated with no difference between the two groups on anything. Below are the choices a careful analyst might reasonably make. Make them, and see what happens.

Your analysis

Outcome

Exclusions

Participants

One more thing you could have varied

Baseline score

Every combination of these four menus is a complete, reportable analysis. There are 72 of them.

No analyses run yet

The result of the analysis currently selected, and the search so far
Quantity	Value

The task Find an analysis of this empty dataset that reaches p < .05, and count how many you had to run. This is a demonstration of what the search costs, not a method to use.

Every path you could have taken

How the 72 available analyses of this dataset are distributed across ranges of p
Range of p	Analyses

Study A

A trial names one primary outcome and three secondary outcomes in its protocol before any data are collected, states that the three secondaries will be tested at a corrected threshold, and reports all four results whichever way they came out.

This is

Study B

A team measures twelve variables with no prior hypotheses, looks for patterns, reports everything it measured and everything it tried, and describes the findings as hypothesis-generating and in need of an independent replication.

This is

Study C

A team measures four outcomes and tries two exclusion rules. One combination reaches p = .04. The paper reports that outcome and that exclusion rule, describes them as the study's analysis, and does not mention the others.

This is

Study D

A preregistered study states one hypothesis, one outcome, one exclusion rule and one analysis, collects the data, runs that analysis and reports the result. It came out non-significant and was published anyway.

This is

Each test keeps its own 5% error rate; what climbs is the chance that somewhere in the family a null result is declared significant. Twenty independent tests at .05 give roughly a two-in-three chance of at least one false positive, with nothing wrong in the data. Analytic flexibility does the same damage without any test being repeated, because the path was chosen after seeing where the result fell.

Multiple Comparisons, FWER and Forking Paths

Learning objective

This is not a technique

First, a prediction

Experiment 1 — the family of tests

One experiment, k tests

Nothing run yet

Reading this correctly

Experiment 2 — the garden of forking paths

Seventy-two defensible analyses of nothing

No analyses run yet

Challenge — four studies, four practices

Study A

Study B

Study C

Study D

What this demonstrates