Cognitive Psychology · Simplified
Two Things Called Attention
You will do it one way. Then you will do by hand what a language model does, and see how little the two have in common.
You do it
What does "it" refer to?
The trophy would not fit in the suitcase because it was too big.
One word in that sentence is doing a lot of work. Which thing was too big?
The model does it
Work the weights out yourself
A language model has no idea what a trophy is. When it reaches the word it, what it does is compare that word with every other word in the sentence and produce a score for each. Here are the scores for this sentence. Your job is to turn them into weights.
| Word | Score | Weight |
|---|
First, add the scores up. What do they come to?
Now every weight is that word's score divided by the total, so they add up to one. Here they are.
This is what "attention" means inside the model: a set of numbers, one per word, adding up to one.
Which is which
Five claims. Who does each one apply to?
You have now done both. For each claim, say whether it is true of what you did, of what the model does, of both, or of neither.
What this shows
One word, two mechanisms
Key idea: The thing you did and the thing you computed share a name and very little else. Yours used what a trophy is and what a suitcase is for, which is knowledge about objects in the world; the model's produced a set of numbers over word positions by comparing each word with the word it, and those numbers add up to one because they were divided by their total, not because anything was understood. Both can arrive at the right answer and both can be wrong. The reason this matters is that attention weights are often shown as if they explained a model's answer, and a distribution over words is a description of what was weighted rather than a reason for anything. Borrowing a word from psychology does not import the theory that came with it.
Three things to keep in view. The scores here are invented for a five-word illustration; a real model works over many more tokens, uses an exponential before normalising rather than a plain division, and runs many of these weightings in parallel at every layer, so what you computed is the shape of the operation rather than the operation itself. Whether attention weights explain a model's output is genuinely disputed in the research literature and this activity does not settle it; what it does show is that a weight distribution is not by itself an explanation. And nothing here says a model cannot resolve this sentence: current models usually do. The claim is about the mechanism, not about the score.
The longer version adds a second sentence where the resolution flips, more of the surrounding architecture, and a longer treatment of what interpretability claims can and cannot support. It is at Human Attention versus AI Attention in the main collection.