Unskilled and Unaware — But Why? A Reply to Krueger and Mueller (2002)

Kruger, J. (University of Illinois) & Dunning, D. (cornell-university). JPSP 82(2), 189–192, 2002. Four pages, printed directly after krueger-mueller-2002. Read in full from the PDF with pypdf. T1.

dunning-kruger-misread said the popular retelling leaves the impression that Kruger and Dunning never considered the statistical objection, and that they had answered it in print. This is that answer, now held rather than asserted.

They concede the mechanism and dispute its reach

The reply grants that measurement error contributes, and grants the general point that imperfectly correlated variables produce regressive estimates — quoting their own 1999 paper saying so. The argument is about how much is left over.

The unreliable test proves nothing. Krueger and Mueller's correction made the asymmetry vanish on the difficult test, whose split-half reliability was .17; at that reliability the result was guaranteed. On the easy test (reliability .56) the low scorers still overestimated themselves by about 30 percentile points after the same correction. And applying Krueger and Mueller's own procedure to 1999's Study 4, where reliability was .93, leaves the bottom quartile overestimating by nearly 40 points.

The mediator was measured with the wrong instrument. Krueger and Mueller scored metacognitive skill as a within-person correlation between item confidence and item accuracy. The reply's counter-example is two weather forecasters: Rob's predictions correlate perfectly with the actual temperature and are off by an average of 48 degrees; Laura's correlate worse and are off by 7. A correlation ignores both the level and the variance of a prediction, and the 1999 studies used deviational measures, which do not. Same objection to the social-projection measure: what mattered was whether top performers overestimated their peers' overall performance, not how their item-level confidence in themselves covaried with their item-level confidence in others.

A regression artifact cannot respond to a manipulation. This is the load-bearing half. In Study 4, 140 participants took a logic test; half then received a ten-minute crash course in logical reasoning before re-estimating. The trained bottom quartile went from overestimating its raw score by 5 points of 10 and its percentile by 36 points to within 1 point and 17 points. Untrained participants did not shift. In Study 3, top scorers shown a sample of their peers' completed tests became better calibrated and bottom scorers did not. Regression predicts the same regressive slope whatever you do to people between measurements.

They also correct a specific reading: Krueger and Mueller wrote that the increase among high performers after seeing peer work "was not significantly larger than the increase among the poor performers." The reply says it was — F(1, 34) = 4.54, p = .04, an interaction unreported in 1999 because it did not bear on the hypothesis — while adding that the size of the increase was never the prediction; the prediction was that only the top group would become more accurate.

How it ends

Two closing moves worth keeping. The first is the joke the exchange writes itself:

"The discerning reader may have noticed that both camps seem to be rather confident in their conclusions, although, given the contradictions, someone must be wrong. Whoever is wrong, they do not seem to know it."

The second is that they decline to claim victory — regression effects are "notoriously difficult to spot but easy to misunderstand," more work is needed, and which account is right "remains to be seen." Twenty-four years later magnus-peresetsky-2022 is still running the same argument, which is evidence for the modesty rather than against it.

Related

krueger-mueller-2002 · kruger-dunning-1999 · dunning-kruger-effect · dunning-kruger-misread · magnus-peresetsky-2022 · justin-kruger · david-dunning · synthesis