Unskilled, Unaware, or Both?

Joachim Krueger & Ross A. Mueller, Brown University. Journal of Personality and Social Psychology 82(2), 180–188, 2002. Subtitle: The Better-Than-Average Heuristic and Statistical Regression Predict Errors in Estimates of Own Performance. Read in full from the PDF with pypdf. T1 — a peer-reviewed replication with its own data, in the journal that published the paper it attacks.

The critique the corpus has been citing at second hand for a month (dunning-kruger-misread, dunning-kruger-effect), now read directly. It is the first half of an exchange: the reply is kruger-dunning-2002-reply, printed immediately after it in the same issue.

The argument

Two ordinary facts, the authors say, produce the whole asymmetry with no metacognitive deficit needed.

Regression. Actual and estimated percentiles correlate imperfectly, so the regression slope is below 1 and extreme scorers are predicted toward the middle. Their illustration is Galton on fathers and sons: in a generation getting taller, short fathers have taller sons and tall fathers have slightly shorter ones, and nobody looks for a mediator to explain it.

Better-than-average. Most people put themselves above the 50th percentile. That raises the whole regression line, which converts a symmetric regression effect into an asymmetric one — big overestimates at the bottom, small underestimates at the top.

What they ran

Sixty-two volunteers, items from a National Teacher Examination prep guide, split into a difficult test (M = 29% correct) and an easy one (M = 70%), 30 participants each. Confidence per item for themselves and for "a majority of students at their university," then a self-estimated percentile.

The pattern replicates. Mean estimated percentile 61.27 (difficult) and 68.77 (easy), both significantly above 50; actual-to-estimated correlations of .44 and .14; bottom quartiles overestimating hugely, top quartiles underestimating slightly.

The mediation does not. Five candidate mediators were tested. Raw percent correct mediated — and it confounds metacognition with performance, which is the authors' point. Of the eight tests using unbiased measures (adjusted percent correct, own-discrimination, other-discrimination, projection), seven changed the correlation by .01 or less, and the eighth (own discrimination on the difficult test, Δ = .05) was not significant.

Subtract the better-than-average effect and the asymmetry is gone. Per-quartile errors corrected for BTA sum to about 3% on their difficult test and roughly zero on average — and they run the same correction over Kruger and Dunning's own four studies, where the raw asymmetries of 33, 32, 32 and 29 percentile points become 1, 10, −4 and 7.

Reliability. Split-half on their own tests gives Spearman-Brown .17 (difficult) and .56 (easy). At .17 the asymmetry reverses when scores are corrected for unreliability; at .56 it shrinks substantially. They also report that a participant's chance of landing in the same extreme quartile on a re-test was .32 and .49 — so the "incompetent" group is partly a group of people having a bad twenty minutes.

One result that is not a null. Estimating your rank should combine a judgment of yourself with a judgment of everyone else. Their high performers, not their low ones, were the group more likely to neglect their own estimate of others when predicting their relative standing. The paper's own summary is that the top quartile, not the bottom, is where the reasoning error sits — the reverse of the finding it is arguing with.

What it does not do

It does not claim the effect fails to appear; the replication is explicitly successful on the pattern. It says the pattern needs no psychological explanation. That is the same shape of objection magnus-peresetsky-2022 makes twenty years later with a censored-tobit model, and this paper is where the line starts.

It also leaves the two experiments in the 1999 paper untouched — the training manipulation in Study 4 and the peer-work manipulation in Study 3 — which is exactly where the reply attacks.

Related

kruger-dunning-2002-reply · kruger-dunning-1999 · dunning-kruger-effect · magnus-peresetsky-2022 · dunning-kruger-misread · replication-crisis · heuristics-and-biases · synthesis