Cognitive Psychology
Human Attention versus AI Attention
The word "attention" appears in two literatures that have almost nothing in common. In one it names a set of experimental findings about selection, capacity and awareness. In the other it names an arithmetic operation: a normalised similarity between one position in a sequence and every other position. This page has you do both, on the same sentence, so the difference is something you have produced rather than been told.
The numbers here are chosen by hand to be legible — no model was run, and nothing was learned
Learning objective
By the end you should be able to say precisely what a transformer attention weight is, and why a weight map is not an explanation of a model's output.
About 15 minutes. Nothing is timed. Nothing you do here is saved or sent anywhere.
Before you start — the language problem
A weight is a number, not a noticing. Throughout this page, the computational side is described in arithmetic terms only. A high weight on a position does not mean the system "focused on" or "cared about" or "was drawn to" that word. It means a dot product came out large and the softmax normalised it upwards. The habit of saying otherwise is the single commonest error in writing about these models, and it is easier to avoid here than anywhere else because you can see the arithmetic.
Nothing on the machine side is aware of anything. There is no experience, no report, no effort and no cost of the kind the human literature measures. The absence is the finding.
- Answer the prediction below — it unlocks the first stage.
- Stage 1: read two sentences that differ by three words and say what "it" refers to in each.
- Stage 2: compute attention weights over the same sentence, by choosing a query position, a head, a temperature and a mask.
- Stage 3: compare the two things you have just done, in the sorting challenge.
- Read the debrief, which is where the one real rhyme and the divergences either side of it are set out.
First, a prediction
You are about to do two quite different things with the same sentence. Before you do either, commit to a view about what the relationship between them will turn out to be.
Stage 1 — what does "it" refer to?
Two sentences. They differ in their last three words and in nothing else. Read each one and say what "it" refers to. There is no time limit, and no trick beyond the one you can see.
Before you move on
What you just did
Stage 2 — compute the weights yourself
The same sentence, cut into ten tokens. Each token has a vector of three numbers attached to it — arbitrary numbers, written by hand for this page. Pick a position to compute from, pick a head, and the arithmetic below runs in front of you.
Key terms
- Attention weight
- A number saying how much one position contributes to the sum being computed for another position. The weights in a row add to one.
- Softmax
- The step that turns a row of raw scores into weights that are all positive and sum to one.
- Temperature
- A divisor applied to the scores before the softmax. A low temperature concentrates the weights on a few positions; a high one spreads them out.
- Head
- One of several weight patterns computed over the same sentence at the same time, each with its own scores.
- Mask
- A rule stopping a position from attending to certain others, such as everything later in the sentence.
One softmax over ten positions
Every number on this page was chosen by the author to make the pattern legible. Nothing was learned, no model was run, and the apparent meaningfulness of what you are about to see is manufactured. That is itself one of the points.
The computation
Every position, with its key vector and score
| Position | Token | Key vector | Score | Weight |
|---|
What this readout is
Where these numbers came from
Stage 3 — sort the statements
Eight statements. For each, decide whether it describes the human experimental literature on attention, the transformer operation you have just run, both, or neither. Two of them are true of both, and two are true of neither — those four are the ones worth arguing about.
What this demonstrates
A transformer attention weight is a softmax over query–key similarities: a set of positive numbers summing to one, deciding how much each position contributes to the next representation. It rhymes with selective attention in that both allocate limited influence, and diverges in almost every other way — there is no capacity limit, no serial bottleneck, no cost of switching, and nothing it feels like. The shared word is a useful metaphor and a poor argument.
For teaching elsewhere: take this activity as one self-contained block of HTML, on the clipboard or as a file. Either way it is styled so that it will not disturb the page you put it into.