Research Methods
A t-test is one number placed on one curve. The curve is what the statistic would do if the population means were equal, and everything the test says is read off where the number lands on it.
Simulated — illustrative summary statistics, no real study
By the end you should be able to say why p moves with sample size while the effect size does not.
About 25 minutes. Nothing you do here is saved or sent anywhere.
A large p means the data are compatible with equal population means. It does not mean the means are equal, so the decision is written as fail to reject H₀ rather than "accept the null". And p is never the probability that H₀ is true: the whole calculation begins by assuming H₀, so nothing about how likely it was can come out of it.
A fictional study of 15 people per group finds a difference of 5 points with a common standard deviation of 10, so Cohen's d is 0.50. The team repeats the study with 100 people per group and finds exactly the same means and the same standard deviation, so d is 0.50 again.
The curve is the distribution of t if the two population means were equal. The dashed rules are the critical values at your chosen α, and the shaded tails are the two-tailed p — the area at least as far from zero as the observed t.
The sliders are summary statistics, not raw data: two means, two standard deviations and a sample size are everything a t-test needs.
| Quantity | Value |
|---|
| Quantity | Value |
|---|
A fictional study of 30 people per group reports a difference of 4.5 points, t(58) = 1.74, p = .087, d = 0.45, with a 95% confidence interval for the difference running from −0.7 to 9.7 points. α was set at .05 in advance.
The numerator is the difference between the two sample means. The denominator is the standard error of that difference — how much it would bounce around from study to study if you repeated the whole thing. t is one divided by the other, so it answers: how big is this difference compared with how big a difference this design produces by chance? A difference of five points is impressive or unremarkable depending entirely on the denominator.
The standard error is the pooled standard deviation times the square root of two over n, so it shrinks as the study grows. Nothing in the numerator changes, and nothing in Cohen's d changes either, because d divides the difference by the spread of individual people rather than by the spread of sample means. That is the whole distinction between an effect size and a test statistic, and it is why the same finding can be reported with p = .18 and p = .0005.
A z-test would use the population standard deviation. A t-test has to estimate it from the same small sample that produced the means, and that extra uncertainty shows up as heavier tails. At three per group the critical value is above 4; by a hundred per group it has come down to 1.97, within a whisker of the normal's 1.96. Degrees of freedom are the accounting for how much information was left after estimating the spread.
Moving α moves the dashed rules and touches nothing else on screen: the data, the difference, the effect size and the tail area are all where they were. The threshold is a rule about what to do, chosen in advance, and choosing it after seeing p is the beginning of the problem the last tool in this module is about. And whatever α is, p is a statement about data under an assumed model, never the probability that the model is right.