RE for AI in Practice: Managing Data Annotation Requirements for AI Autonomous Driving Systems

Hina Saeeda and Jennifer Horkoff (Chalmers University of Technology), Mazen Mohamad (RISE Research Institutes of Sweden), Eric Knauss (University of Gothenburg) and Ali Nouri (Volvo Cars). arXiv 2511.15859, 19 November 2025, cs.SE. Preprint; tiered T1 as a primary empirical study reporting its own method and data. Ingested via the research pass against growth edge 1a.

19 semi-structured interviews, 20 participants, six international companies — one OEM, three Tier-1 and two Tier-2 suppliers — plus four research organisations, across two European countries and the UK. Roughly 50 hours of transcript, collected September 2024 to February 2025, coded thematically by two coders at Cohen's κ = 0.8 and validated with four senior practitioners.

Why it is here

safetutors answers "what does a documented labelling operation look like" with an academic one. This answers the other half: what the industrial ones look like, from the people running them. who-annotates-in-nlp established that the practice is unpublished; this study went and asked.

The trade is honest to state up front. These are practitioner accounts, not measurements from a system of record. Every number below is one informant's recollection, quoted with an ID, and none is independently verified. That is weaker evidence than a log export and stronger than the nothing the corpus had.

What annotation requirements look like in the field

Guidelines run past 140 pages. They are collaboratively defined, formalised and version controlled where the process is mature, and referenced against ISO/IEC 5259, IEEE P2801, ISO 26262 and SAE J3016.
Ambiguity is treated as irreducible, not as a defect to be fixed. "Impossible to define unambiguous annotation requirements — images are inherently ambiguous" (ID16). On edge cases: "the more you try to pin them down, the more exceptions you uncover."
Requirements change mid-project, which sends already-annotated batches back for correction.

The workforce problem, stated plainly

"Annotation is low-paid work… we can't attract domain experts" (ID5). Onboarding is built per project and described as "time-consuming and not scalable" (ID1); turnover then forces "frequent retraining and wasted resources" (ID5, ID18). The people doing the labelling are largely non-specialists needing extensive training (ID3). This is the same pattern data-cascades found from the practitioner side, now with the recruiting mechanism named.

Four numbers worth holding

Manual review cut from 10% to 5% of output under budget pressure (ID3) — the quality control is the first thing traded away.
Rework costs about 3× proper initial labelling (ID5), so the trade above loses money as well as quality.
Only ~20% of collected data gets annotated at one organisation (ID18); the rest is stored and unused.
Version conflicts destroyed 15% of a 3D point-cloud annotation set (ID17), in an environment where "we couldn't trace which annotation version caused which issue — traceability was missing."

The finding underneath

The paper's own contribution is the chain: flaws in annotation requirements propagate into annotation practice, into data quality, and into the perception system's measured performance. Five recurring challenges — ambiguity, edge-case complexity, evolving requirements, inconsistency, resource constraints — and three categories of recommended practice: ethical-standards compliance, better guideline structure, and quality assurance embedded in the requirement rather than bolted on after.

Note the shape of the thing: it is a requirements study, so it observes how the rules for labelling are written and governed, not a single pipeline followed from recruitment to delivered dataset. Nobody here published the appendix that safetutors published. What this adds is why not — 140-page documents, unstable targets, and a workforce that turns over faster than the guidelines stabilise.

safetutors · who-annotates-in-nlp · data-cascades · training-data-quality · measuring-annotator-agreement · demo-to-production-gap · hina-saeeda · chalmers-university-of-technology