Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks
Alexander Braylan, Omar Alonso and Matthew Lease, arXiv 2212.09503 (December 2022), also published through the ACM. The first source here on how labelling quality is checked, where data-cascades says the checking fails and pervasive-label-errors says the labels are wrong.
The problem it names
Inter-annotator agreement is the standard instrument: two people label the same item, and you measure whether they agree more than chance would predict. Cohen’s kappa, Fleiss’ kappa and Krippendorff’s alpha all do this well for categorical and ordinal labels — cat or dog, 1 to 5.
Most real annotation is not that shape. The paper works through seven task types where it breaks down: image bounding boxes, image keypoints, text sequence tagging, ranked lists, free-text translations, numeric vectors, and syntax trees. Two annotators drawing boxes around the same object never produce identical output, so “agreement” needs a distance function before it means anything, and picking one is a modelling decision that the metric then inherits.
Krippendorff’s alpha admits distance-based formulations in theory. The authors’ complaint is that it has not been validated empirically across these task types, and that the resulting numbers are hard to interpret. They propose two alternative measures aimed at consistency across tasks and across the choice of distance function.
Where it sits in this wiki
The data layer now has three sources and they form a sequence rather than a pile. data-cascades establishes that the practice fails often — 92% of practitioners. pervasive-label-errors establishes that the output is wrong at a measured rate — at least 3.3% across ten standard test sets. This one is about the instrument you would use to catch either problem before it ships, and its finding is that the instrument is well-developed for the simplest label shapes and shaky for the ones most modern datasets actually use.
That connects back to training-data-quality‘s open complaint. It also lands near a result this corpus already holds in a different spoke: er-evaluation-framework found that the naive way of measuring entity-resolution quality is biased upward. Two different fields, the same shape of finding — the measurement of the measurement is where the error hides.
Limits
It supplies a metric, not a practice. Nothing here covers writing annotation guidelines, training annotators, sampling what to double-label, or what a labelling operation costs — which is most of what the growth edge asked for. A better agreement statistic tells you a labelling process is inconsistent; it does not tell you how to run one that is not.
Related
training-data-quality · data-cascades · pervasive-label-errors · machine-learning · synthesis