Research Methods · Simplified
The Same Finding, Twice
A t-test asks one question: how surprising is this difference if the two populations are identical. The sample size changes the answer without changing the finding.
The test
One difference, placed on the null distribution
Two groups are compared. The curve below is what t would look like across endless repetitions if the two populations were exactly the same. The observed t is the marked line. Everything else follows from where it falls.
Key terms
- Standard error of the difference
- How much the difference between two sample means bounces around from one study to the next. It shrinks as the groups get larger, which is the only place the sample size enters.
- t
- The observed difference divided by that standard error. A difference measured in units of its own wobble.
- The null distribution
- The curve. Not a distribution of scores or of people, but of the statistic t across repeated studies in a world where the two populations are identical.
- p
- The shaded area: how often that world would produce a t at least this far from zero, in either direction.
The shaded tails are the two-tailed p: how often a world with no difference at all would throw up a t at least this extreme. The dashed lines are the 5 per cent critical values. The faint curve behind is the standard normal, which t approaches as the groups grow.
What this shows
The sample size lives in the denominator
Nothing about the two populations changed when you added people. The difference is the same, the spread is the same, and Cohen's d did not move. Only the standard error shrank, and t is the difference divided by it.
Key idea: t is a difference measured in units of how much that difference wobbles from study to study. Collecting more people does not make the effect bigger; it makes the wobble smaller, so the same effect sits further out on a narrower null distribution and the tail area collapses. This is why a p-value on its own tells you almost nothing about the size of a finding, why a small study can miss a real effect, and why an enormous study can return a significant result for a difference nobody would care about. Report the interval and the effect size beside the decision, not instead of it.
Read the conclusion carefully. A p above 0.05 does not show the populations are the same; it says this study could not distinguish them, which is why the wording is "fail to reject" rather than "accept". The p-value is the probability of data at least this extreme given no difference, and not the probability that there is no difference; those are different quantities and swapping them is the commonest error in the literature. The figures here also assume independent observations, roughly normal populations and equal spreads, and that this test was the one you meant to run before you saw the data, which is the assumption most often broken in practice.
The longer version adds a one-tailed and two-tailed comparison, the heavier tails of t worked through against the normal in detail, and a conclusion-writing exercise. It is at Independent-Samples t-Test: The Null Distribution in the main collection.