Research Methods
Run a study whose true effect is exactly zero, several thousand times, and watch what "nothing is going on" actually looks like. The p-value is the tail area of that picture — and nothing else.
Simulated — a generated null model, drawn from a documented seed
By the end you should be able to describe a sampling distribution, and say what a p-value is and is not the probability of.
About 25 minutes. Nothing you do here is saved or sent anywhere.
Everything on this page is computed assuming the null model is true, the samples are independent draws from it, the analysis was decided in advance, and this is the only test being run. A p-value is a statement about data under a model. It is not the probability that the null hypothesis is true, not the probability the result happened by chance, and not a measure of how large or important an effect is.
Two groups of 40 fictional participants are drawn from exactly the same population, so the true difference between them is zero. Scores have a standard deviation of 15.
Every study in the pile has a true effect of exactly zero. The spread you see is what sampling alone produces.
Two groups of n fictional participants, drawn from the same population. The statistic is the difference between their means.
| Region | Null studies | Share |
|---|
| Quantity | Predicted by the model | Observed in the simulation |
|---|
A study reports a difference of 5.0 points between two groups, with p = .14.
Every bar in the histogram is a study that was never run. The distribution says what results a stated model would produce if you repeated the whole procedure, and it is the reference against which one real result is judged. Nothing about it is a property of your data.
It answers: given the model, how often would data at least this extreme appear? It cannot answer the question people actually want — how likely the hypothesis is — because that would require starting from something other than the data, and the calculation never does. Reversing the conditional is the single most common error in statistical reporting, and it is the one the challenge above is built around.
Hold the observed difference at 5.0 and change the sample size. The number the study reported has not moved, and the p-value travels from about .5 to below .001. A p-value is therefore not a measure of how big anything is; it confounds size, variability and sample size into one figure, which is why effect sizes are reported alongside it rather than instead of it.
The tail area is exact only if the samples really are independent draws from the stated distribution, the analysis was fixed in advance, and this is the only comparison being made. Break any of those and the number on screen stops meaning what it says — which is the subject of the multiple-comparisons tool later in this module.