Annotation quality in ABSA: experts, students, crowdworkers and an LLM (Donhauser et al., 2026)

Niklas Donhauser, Jakob Fehle, Nils Constantin Hellwig, Markus Weinberger, Udo Kruschwitz and Christian Wolff (Universität Regensburg), arXiv:2605.03624, 5 May 2026, 16 pages — read from the PDF. Found by the research pass hunting growth edge 1b: what a labelling operation costs.

The same German aspect-based sentiment task annotated four ways — by experts, by students, by crowdworkers, and by an LLM — with the cost of each reported and the resulting datasets used to train models, so the trade-off is measured at both ends rather than asserted.

Design. Built on the GERestaurant corpus, 1,000 sampled training sentences, annotated in batches of 200 across two subtasks: aspect-category sentiment (ACSA) and target-aspect-sentiment detection (TASD). Thirty crowdworkers were recruited on Prolific, restricted to German speakers in DE/AT/CH with at least secondary education, 200 sentences each, working in Label Studio from written guidelines and an instructional video. Agreement was measured with average pairwise micro-F1 plus Krippendorff's α for ACSA and micro-F1 for TASD; models were scored over five seeds with Shapiro–Wilk, then ANOVA or Friedman, with Holm–Bonferroni correction.

The number the corpus was missing

Crowdworker studies were completed within two days per study at a total cost of roughly £828 (≙ £0.41 per three way annotated example), including platform fees.

Annotators were paid £9 per hour, Prolific's recommended rate. That is a total, a per-usable-label figure, and the scope it covers — which is what growth edge 1b asked for, and what safetutors (rates but no total) and annotation-requirements-autonomous-driving (a 3× rework ratio but no base) could not supply.

The comparison is more interesting than the number

Four ways of getting the same labels, and the money only describes one of them:

"LLMs are cheaper" is true by a factor of 164 and false at the same time. A self-hosted model annotates at a quarter of a cent an example; a frontier API costs $0.36, which is the crowd price. The spread inside "LLM annotation" is larger than the gap between LLM and human, so the sentence means nothing without naming the tier.

Two of the four options have no price at all, and those are the ones a cost model gets wrong. Student annotation was paid in course credit and took weeks — a real cost that appears in no budget line, borne by a schedule rather than an account. Expert refinement took "several hours" and the authors are explicit that this was only possible because they were revising existing labels: "without pre-existing labels, large-scale expert annotation would be costly and difficult to scale." The cheapest expert pass in the study is the one that presupposes somebody else already did the work.

The quality finding, and it lands on this corpus's instrument

…high agreement does not necessarily translate into superior model performance, as shown by LLM annotations, which achieved high IAA but only comparable performance to student-annotated data.

That is a direct hit on measuring-annotator-agreement, which supplied this wiki with agreement as the instrument for label quality. Agreement measures whether annotators concur; a model trained on those labels measures whether they were right. Here the two come apart in the direction that flatters the cheap option — the LLM agrees with itself.

And the failure mode is coverage, not mislabelling. On the harder TASD subtask, expert and LLM datasets "consistently contain higher counts across categories, while student and crowdworker annotations yield noticeably fewer instances, particularly for less frequent categories such as AMBIENCE and PRICE." Students and crowdworkers did not get the labels wrong so much as not see them — a systematic under-annotation of rare classes. An agreement score computed over what people did annotate cannot detect that at all, because the missing instances are missing from both sides.

The authors' own summary: expert annotations "remain the most reliable basis for achieving peak performance," while LLMs are "a fast and scalable annotation alternative."

Limits

One language, one domain (restaurant reviews), 1,000 sentences, one crowd platform, and a single frontier model priced from a list rather than run. The cost figures are for this operation at this scale; nothing here supports extrapolating £0.41 to a task with a 140-page guideline like the automotive one. And a study whose experts were re-annotating an existing dataset has priced the cheap version of expertise.

Related

measuring-annotator-agreement · safetutors · annotation-requirements-autonomous-driving · who-annotates-in-nlp · training-data-quality · pervasive-label-errors · synthesis