Research Methods
Independent-Samples t-Test: The Null Distribution
A t-test is one number placed on one curve. The curve is what the statistic would do if the population means were equal, and everything the test says is read off where the number lands on it.
Simulated — illustrative summary statistics, no real study
Learning objective
By the end you should be able to say why p moves with sample size while the effect size does not.
About 25 minutes. Nothing you do here is saved or sent anywhere.
Two phrases to avoid
A large p means the data are compatible with equal population means. It does not mean the means are equal, so the decision is written as fail to reject H₀ rather than "accept the null". And p is never the probability that H₀ is true: the whole calculation begins by assuming H₀, so nothing about how likely it was can come out of it.
- Answer the three prediction questions to open the laboratory.
- Move the group means and watch t travel along the curve.
- Hold everything else still and grow the sample: d will not move.
- Change α and watch the critical value, not the evidence, move.
First, three predictions
A fictional study of 15 people per group finds a difference of 5 points with a common standard deviation of 10, so Cohen's d is 0.50. The team repeats the study with 100 people per group and finds exactly the same means and the same standard deviation, so d is 0.50 again.
The laboratory
The curve is the distribution of t if the two population means were equal. The dashed rules are the critical values at your chosen α, and the shaded tails are the two-tailed p — the area at least as far from zero as the observed t.
Key terms
- t
- A signal-to-noise ratio: the difference between the means divided by the standard error of that difference.
- Standard error of the difference
- The denominator of t. The sample size enters the calculation here and nowhere else.
- Null distribution
- The distribution t would follow if there were no difference in the population.
- α and critical value
- The error rate chosen in advance, and the value of t at which it puts the threshold.
One number on one curve
The sliders are summary statistics, not raw data: two means, two standard deviations and a sample size are everything a t-test needs.
The null distribution
| Quantity | Value |
|---|
The conclusion, written carefully
The same comparison as an interval, and t against z
| Quantity | Value |
|---|
Challenge — write the conclusion
A fictional study of 30 people per group reports a difference of 4.5 points, t(58) = 1.74, p = .087, d = 0.45, with a 95% confidence interval for the difference running from −0.7 to 9.7 points. α was set at .05 in advance.
What this demonstrates
t is a signal-to-noise ratio
The numerator is the difference between the two sample means. The denominator is the standard error of that difference — how much it would bounce around from study to study if you repeated the whole thing. t is one divided by the other, so it answers: how big is this difference compared with how big a difference this design produces by chance? A difference of five points is impressive or unremarkable depending entirely on the denominator.
The sample size lives in the denominator only
The standard error is the pooled standard deviation times the square root of two over n, so it shrinks as the study grows. Nothing in the numerator changes, and nothing in Cohen's d changes either, because d divides the difference by the spread of individual people rather than by the spread of sample means. That is the whole distinction between an effect size and a test statistic, and it is why the same finding can be reported with p = .18 and p = .0005.
The curve is fatter than the normal, and that is a price
A z-test would use the population standard deviation. A t-test has to estimate it from the same small sample that produced the means, and that extra uncertainty shows up as heavier tails. At three per group the critical value is above 4; by a hundred per group it has come down to 1.97, within a whisker of the normal's 1.96. Degrees of freedom are the accounting for how much information was left after estimating the spread.
α is a decision, and p is not evidence about H₀
Moving α moves the dashed rules and touches nothing else on screen: the data, the difference, the effect size and the tail area are all where they were. The threshold is a rule about what to do, chosen in advance, and choosing it after seeing p is the beginning of the problem the last tool in this module is about. And whatever α is, p is a statement about data under an assumed model, never the probability that the model is right.
For teaching elsewhere: take this activity as one self-contained block of HTML, on the clipboard or as a file. Either way it is styled so that it will not disturb the page you put it into.