The AI Incident Database
Sean McGregor, Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database, IAAI-21 (arXiv:2011.08512). Read in full. The paper describing the AIID, a project of the Partnership on AI, holding more than 1,000 incident reports at publication. It answers coverage edge 6 awkwardly, which is the useful part.
The argument
Mature sectors keep incident databases. Aviation distinguishes an accident (substantial damage or loss of life) from an incident (something that substantially raises the risk of an accident): a cockpit fire quickly put out is an incident, the same fire burning the crew is an accident. The NTSB indexes timelines, location, weather, severity, aircraft, operator and phase of flight as facets over a searchable narrative. McGregor cites an 81-fold fall in fatalities per passenger mile since 1970. The other model he names is CVE, then holding 141,076 disclosed vulnerabilities, whose contribution is not analysis but a consistent identifier that other systems can hang scoring, research and tooling from.
AI has neither. Practitioners were keeping failure lists in Google Docs and on GitHub — admirable, he says, and not comprehensive, not identified, not extensible, not full-text searchable.
How it is built
Incidents get a number; multiple reports attach to one incident (incident 3 is the Boeing 737 MAX 8 crashes, with 18 reports). Incidents are deliberately untitled, because descriptions change as more is learned and because getting corporate and non-profit participants to agree on a title would be its own fight. Metadata captured on entry is the objective part: title, source, author, submitter, publication date, incident date, incident number. Everything subjective is a taxonomy, and taxonomies are namespaced — partners define their own, and the paper says so explicitly, "avoid[ing] the challenge of developing a single shared universal ontology for AI incidents and instead allow[ing] for multiple viewpoints on the data to develop and compete for mindshare." The AIID itself declines to provide top-down analysis or narrative.
What that means for governance
eu-ai-act Article 73 creates serious-incident reporting duties, and this wiki has catalogued governance instruments with no evidence of harm to govern. The AIID is the largest collection there is, and reading the paper carefully shows it cannot serve as a denominator:
- Coverage is driven by press attention. The paper is candid that the number of reports on an incident is "a proxy for interest," and that more-reported incidents are typically more damaging, more sensational, or both. Sensational and damaging are not the same variable.
- There is no severity scale and no shared taxonomy by design, so incidents cannot be summed, weighted or compared across systems without picking somebody's namespace.
- Entry is by public submission of already-published reports, not by a reporting obligation, so absence of an incident is not evidence of absence.
Growth edge 1 asks whether certification correlates with anything. This is the closest thing to an
outcome dataset in the field, and the honest reading is that it supports case-finding — "what
can go wrong when someone deploys this system" is the question the AIID says it answers — and not
rate estimation. That is the same wall ../defensive-security-wiki hit with breach datasets that
carry no per-company compliance variable: the incidents exist, the population behind them does not.
The design bet worth watching
Refusing a universal ontology is a real decision with a governance cost. A regulator implementing an Article 73 duty needs a definition of "serious incident" that is the same in Dublin and Rome; the AIID's answer is that competing taxonomies should fight it out. Whether that produces the CVE outcome — a shared identifier that everyone builds on — or fragmentation is the thing to check when a second source arrives.
Not a volunteer-project problem (added 2026-08-25)
It would be easy to read the missing denominator here as the limit of a non-profit doing what it can with press reports. ai-ml-device-adverse-events closes that reading off. The FDA has a mandatory reporting duty, a published roster of 950 authorised AI/ML devices, and its own harm database; researchers linked 823 of those devices to 943 reports and still concluded that without a figure for how often the devices are used, relative safety cannot be discussed. Same hole, federal statute, fourteen years of reporting. What the AIID lacks, the best-funded instrument in the field also lacks.
A method for building the denominator, from the AIID's own team (added 2026-10-11)
ai-incident-public-health-lens is McGregor (this page's author) and seven co-authors proposing how to get the missing denominator, borrowing public-health surveillance's assumption that incident data is always incomplete and noisy. Worked case: autonomous vehicles, where California DMV mandatory reporting gives miles driven as an exposure denominator and incident rates are expressed per mile rather than as a raw count. This is a method, not a fix for AI generally — it works because AVs have a mandatory-reporting regime most AI systems lack. The residue: no published exposure denominator yet exists for a general-purpose model deployed without that kind of mandate.