Research Methods
Sampling Distribution and p-Value Simulator
Run a study whose true effect is exactly zero, several thousand times, and watch what "nothing is going on" actually looks like. The p-value is the tail area of that picture — and nothing else.
Simulated — a generated null model, drawn from a documented seed
Learning objective
By the end you should be able to describe a sampling distribution, and say what a p-value is and is not the probability of.
About 25 minutes. Nothing you do here is saved or sent anywhere.
What a p-value is conditional on
Everything on this page is computed assuming the null model is true, the samples are independent draws from it, the analysis was decided in advance, and this is the only test being run. A p-value is a statement about data under a model. It is not the probability that the null hypothesis is true, not the probability the result happened by chance, and not a measure of how large or important an effect is.
- Answer the prediction question to unlock the simulator.
- Press Run 1,000 null studies and watch the pile build.
- Move the observed difference and watch the shaded tails change.
- Change n and σ and see the whole distribution rescale.
First, a prediction
Two groups of 40 fictional participants are drawn from exactly the same population, so the true difference between them is zero. Scores have a standard deviation of 15.
The simulator
Every study in the pile has a true effect of exactly zero. The spread you see is what sampling alone produces.
Key terms
- Sampling distribution
- The spread of a statistic across many repetitions of the same study. It is a picture of repetition, not of anybody's data.
- Null model
- A fully described world in which the effect is absent. The pile here is built by running studies inside one.
- p-value
- The probability of a result at least this extreme if the null model were true. Not the probability that the null model is true.
- Tail area
- The share of the null distribution lying beyond the observed value. It is what a p-value reads off.
A world in which nothing is going on
Two groups of n fictional participants, drawn from the same population. The statistic is the difference between their means.
Differences from 0 null studies
| Region | Null studies | Share |
|---|
Reading this correctly
Theory beside simulation
| Quantity | Predicted by the model | Observed in the simulation |
|---|
Challenge — say what p means
A study reports a difference of 5.0 points between two groups, with p = .14.
What this demonstrates
A sampling distribution is a picture of repetition
Every bar in the histogram is a study that was never run. The distribution says what results a stated model would produce if you repeated the whole procedure, and it is the reference against which one real result is judged. Nothing about it is a property of your data.
The p-value is a tail area, and it points backwards
It answers: given the model, how often would data at least this extreme appear? It cannot answer the question people actually want — how likely the hypothesis is — because that would require starting from something other than the data, and the calculation never does. Reversing the conditional is the single most common error in statistical reporting, and it is the one the challenge above is built around.
The same difference gives different p-values
Hold the observed difference at 5.0 and change the sample size. The number the study reported has not moved, and the p-value travels from about .5 to below .001. A p-value is therefore not a measure of how big anything is; it confounds size, variability and sample size into one figure, which is why effect sizes are reported alongside it rather than instead of it.
Everything here is conditional on a model nobody checked
The tail area is exact only if the samples really are independent draws from the stated distribution, the analysis was fixed in advance, and this is the only comparison being made. Break any of those and the number on screen stops meaning what it says — which is the subject of the multiple-comparisons tool later in this module.
For teaching elsewhere: take this activity as one self-contained block of HTML, on the clipboard or as a file. Either way it is styled so that it will not disturb the page you put it into.