Training data quality

The layer this wiki named as its founding gap and then could not find a literature for: how the data a model learns from, and the data it is judged against, get collected, labelled and checked.

Two sources, two halves

The practice half. data-cascades interviewed 53 practitioners in high-stakes domains and found that 92% had hit at least one compounding data failure, 45.3% more than one in a single project. The failures share a shape — a decision made early about a label, a sample or a hand-off, whose consequence surfaces much later and far downstream.

The measurement half. pervasive-label-errors checked the labels themselves: at least 3.3% wrong on average across ten of the field's standard test sets, at least 6% on the ImageNet validation set. Correcting them can reverse a model ranking, so the errors are not noise that averages out — they change conclusions.

Put together: the data is wrong at a measured rate, and the process that produces it fails at a measured rate. Neither source needed a model to establish its claim.

Why the field looks away

Both sources say the same thing about incentives, from different directions. Sambasivan et al. put it plainly — "data is the most under-valued and de-glamorised aspect of AI" — and their title is a practitioner's own words about who wants which job. Northcutt et al. show the cost of that preference landing somewhere specific: in the test sets everyone competes on, unexamined for years while model architectures turned over several times.

The wiki has its own version of this. Every source it holds on the middle layer arrived as teaching material about something else, and the reading was that no literature existed. What existed was a literature in HCI, studying practitioners, which a search of the ML side would never return.

What is still missing

None of the three says how to do the work well. Data Cascades is a diagnosis and calls for "incentivizing data excellence" without specifying the practice; the label-error paper supplies a detection method (confident learning) for one failure, on data that already exists. None of the three covers annotation guidelines, annotator training, sampling design, or what a labelling operation actually costs. Inter-annotator agreement is the one exception, and it arrived as a metric rather than a practice — see below. Written 2026-08-08; the two sources added on 2026-08-25 supply everything in that list except the cost.

The instrument, added the same day

measuring-annotator-agreement is the third piece: not how often the practice fails or how wrong the labels are, but how you would check. Its finding is that inter-annotator agreement — the standard instrument — is well-founded for categorical labels and shaky for bounding boxes, sequence tags, ranked lists and free text, because each needs a distance function chosen before agreement means anything. So the gap above narrows by one: the corpus now has an instrument for checking a labelling process. It still has no account of how to run one.

How often the question is even asked

who-annotates-in-nlp measures the reporting rather than the practice: 2,667 annotation tasks pulled from 1,603 ACL-venue papers, 2018–2025, audited for what they disclose about who did the labelling. It is the field-scale version of the gap the three pages above describe individually, and it makes the same point structurally — the corpus still has no account of how to run a labelling process, and neither, on this evidence, does the literature. The audit also counts by task instead of by paper, on the argument that one paper can run several tasks with different annotators and different controls, so counting papers averages away the variation being studied.

How one is actually run — added 2026-08-25

Two sources arrived together and between them close most of the gap the sections above keep naming.

The academic operation, published in full. safetutors prints its annotation appendix: three stages with three workforces (6 undergraduates at $5/hr, 3 doctoral students at $15/hr, 24 Prolific workers at $10/hr), a four-step calibration ending in an ≥80%-agreement-with-gold gate, a 10-item qualification test with its pass rates published (24 of 33 crowd recruits, 72.7%), verbatim guidelines per stage, named tooling, an adjudication protocol with its own rates (4.2% of crowd instances escalated; 3.8% cross-stage conflicts, 21 removed and 36 flagged), Fleiss' κ broken down three ways, and 15,300 labels totalled. This is the artifact the corpus had been unable to find.

The industrial operation, reached by interview. annotation-requirements-autonomous-driving asked 20 practitioners at six automotive companies how the same work runs at scale, and the answer is worse in specific ways: guidelines past 140 pages, manual review cut from 10% to 5% under budget pressure, rework at about 3× the cost of labelling properly the first time, ~20% of collected data ever annotated, and 15% of one 3D point-cloud set lost to version conflicts.

Read together they explain the reporting rate who-annotates-in-nlp measured. The academic operation could publish its appendix because it was small, bounded and finished. The industrial one has no appendix to publish — its guidelines are unstable, its workforce turns over, and its own practitioners say traceability is missing, so there is no settled artifact to attach to a paper.

What remained open was narrower than before: cost. Neither source reports total spend. SafeTutors gives rates and per-item times without a total; the interview study gives a ratio (3× for rework) without a base.

Closed 2026-08-28, and this page did not say so until 2026-08-30. absa-annotator-comparison-cost prices the same German ABSA task four ways: £828 total, £0.41 per three-way annotated example including platform fees, thirty Prolific crowdworkers at £9/hour, two days per study. That is a total, a per-usable-label figure, and its stated scope — exactly what the sentence above said nobody had published. The paper was ingested by the research pass on 2026-08-28 and never linked from here, so the claim stood for two days after the corpus could refute it. (Found by the quality cycle's new argument-staleness pass; the page was also this spoke's only orphan.)

What the number hides is the better finding. Two of the four annotator types have no price at all — students were paid in course credit and took weeks, experts refined existing labels — and "LLM annotation" spans $0.0025 self-hosted to $0.36 on a frontier API, a spread wider than the gap between LLMs and humans. So "LLMs are cheaper than annotators" is unanswerable without naming the tier, and elapsed time is a cost this corpus has never priced.

The other strategy: don't annotate, manufacture

Everything above treats labelling as work done to data that already exists — recruit, train, guide, adjudicate, price. code-as-world-report takes the opposite route on a task where the first strategy barely works at all.

The task is estimating an object's real speed or size from a video. Nobody can annotate that: a crowdworker cannot see metres per second, and an expert would have to have measured the scene at capture time. quantiphy solves it for 569 videos by curation, which is not a route to a training set. So instead of labelling observed worlds, Code-as-World simulates worlds it already knows the answers to and renders them into videos — an executable world exposes exact geometry, camera parameters and a state trajectory, so size, displacement, velocity and acceleration are read off the simulator rather than judged by a person.

Every cost measured on this page goes to zero, and one new one appears. No recruitment, no guidelines, no adjudication, no £0.41 per example, no inter-annotator agreement — because there are no annotators to agree. What replaces them is the fidelity of the simulated world, and code-as-world-report §4.4 concedes that the agentic-discovery-loop can settle on a world that renders like the evidence without being mechanistically right.

That is the same failure this page keeps finding, relocated. pervasive-label-errors found wrong labels attached to real data; here the labels are exact by construction and the world they describe may be the wrong one. The error moves upstream, out of reach of every instrument in the sections above — no distance function, no κ, and no crowdworker adjudication can detect a label that is perfectly consistent with a mis-specified simulation. Verification against rendered evidence is the only check offered, and it tests appearance, not mechanism.

Which is also why the corpus should not read this as the cheap answer. The synthetic route needs a simulator that models the phenomenon, and the paper's own limitation is that real scenes contain deformation, fluids, occlusion and long-horizon interaction that MuJoCo does not. The strategies partition by domain rather than compete: manufacture where a simulator is faithful and a human cannot see the quantity, annotate where the reverse holds.

A third arrangement: manufacture, then correct by hand (added 2026-09-07)

The two strategies above partition by domain. male-fly-connectome runs them in sequence, on one task, and is the first source here that does.

Reconstructing 166,000 neurons and 125 million synapses from electron-microscope images uses both. Synthetic neurons are generated into the training data for the segmentation model — Google Research reports this "successfully improving the speed and accuracy" of its PATHFINDER system — and then human experts at HHMI Janelia proofread what the model produces. The post is blunt about which half binds: mapping a fly brain "requires years of human effort just to verify and annotate the neural shapes," and the stated goal of better models is "reducing this need for manual error correction."

Two things follow for this page.

The human step is not labelling; it is label-error correction. Everything in the sections above treats annotation as producing labels a model then learns from. Here the model goes first and people fix its output, which makes proofreading the same activity pervasive-label-errors performed on finished test sets — except budgeted in advance, at scale, as the main cost of the project. That is a different job from the one the annotator-agreement literature measures, and none of the instruments on this page (κ, adjudication protocols, per-example price) apply to it unchanged.

And "who annotates" gets an answer the corpus has not seen. Not crowdworkers, not a vendor, not the model: expert staff at the institute that owns the science, for years. who-annotates-in-nlp's question has a third answer, and it is the most expensive one available.

What is missing is every number that would let the comparison be quantitative — no proofreading hours, no accuracy figure, no split between automated and corrected output. See male-fly-connectome for what the post does and does not publish.

absa-annotator-comparison-cost · safetutors · annotation-requirements-autonomous-driving · measuring-annotator-agreement · who-annotates-in-nlp · data-cascades · pervasive-label-errors · demo-to-production-gap · machine-learning · ml-system-design · code-as-world-report · male-fly-connectome · connectomics · executable-world-representation · quantiphy · synthesis