Personality & Individual Differences · Simplified
It Correlates. So What?
The most common claim made for a new measure is also the weakest one available.
What these measures are
What these measures are. All of them are fictional descriptions of method types: a questionnaire about how you usually handle feelings, a broad personality questionnaire, a reasoning test, and colleagues' ratings of how effective somebody is with other people. No published emotional-intelligence measure is reproduced, quoted, paraphrased or scored here, and no number on this page is an estimate of any real quantity. Nothing here takes a position on whether emotional intelligence exists. It is about what a correlation does and does not license somebody to advertise.
Step 1 of 2
First, a judgement
A research group reports that their new measure correlates 0.45 with an outcome that matters, in a large sample, and the finding replicates. What does that establish about the measure?
Step 2 of 2
One measure, one question about it
The outcome is colleagues' ratings of how effective somebody is with other people. Two measures are already available and cheap: a broad personality questionnaire and a reasoning test. The new measure is a questionnaire about how people usually handle feelings. The only thing the slider changes is how much of that questionnaire is ordinary personality content.
Key terms
- Incremental validity
- How much a measure improves prediction beyond what the measures you already had were achieving. It is the question a new measure has to answer.
- Variance explained
- The share of the differences between people in the outcome that a set of predictors accounts for. Adding a predictor can only raise it, so the size of the rise is what matters.
- Jingle
- Two measures sharing a name while measuring different things, or sharing content while wearing different names. The second is what the slider produces.
Two lines across every version of the new measure. The upper one is the correlation with the outcome, which is what gets advertised. The lower one is how much it adds beyond the two measures already available. The upright marker is the version you have built.
| Measure | Correlates with the outcome | Correlates with the personality questionnaire |
|---|
What this shows
The advertised number is the one that moves least
Key idea: At every setting of that slider the new measure correlates between 0.36 and 0.48 with the outcome. That is the number a press release quotes, it is true at every setting, and it distinguishes none of them. What it adds beyond a personality questionnaire and a reasoning test runs from about ten percentage points of explained variance down to almost nothing, and the difference between a measure worth adopting and a measure worth ignoring lies entirely in that second number. Notice also that the overlap does not have to be extreme before it starts to bite, and that a measure can overlap substantially and still add a good deal: a correlation of 0.5 with an existing questionnaire is not by itself an objection. The question is never whether a measure correlates with something important. It is what a decision-maker would do differently if they had it, given what they already have.
Every figure here is what a specified model implies for a population, with no sampling error, so nothing carries the confidence interval a real incremental validity estimate would; those intervals are wide, and incremental validity is estimated poorly in samples of the size most studies use. Three further cautions. Incremental validity is relative to whatever you compare against, so a measure that adds nothing beyond a long personality inventory may add a great deal over nothing at all, and choosing the comparison set is a substantive decision this page makes for you. A measure can also be worth having for reasons other than prediction, including being quicker, cheaper or more acceptable to the people completing it. And nothing here bears on whether emotional intelligence exists or on any published measure of it: the demonstration is about what a correlation licenses, and it would work identically for any construct at all.
The longer version includes six measures rather than four, an ability-based measure alongside the self-report one, controls for shared method variance, and a set of marketing claims to judge against the matrix you have produced. It is at Emotional-Intelligence Claims Laboratory in the main collection.