The demo-to-production gap

The distance between an AI system that works in a demonstration and one that runs as a product. A demo has to succeed once, on chosen inputs, with its author present; a production system has to succeed continuously, on inputs nobody chose, while data drifts and the person who built it moves on.

What the corpus claims

One headline, and it is the whole of the evidence: the causes are infrastructure and people, not model quality ai-projects-fail-infrastructure-people. The article body has never been readable, so what this wiki holds is a framing, not an argument — that page states the limitation in full.

Against it sits the survivors’ record: 300+ production ML systems that did cross the gap, written up by the companies that built them ml-system-design-case-studies. Neither source addresses the other, and between them they leave the central number unknown.

The open question

How often do AI projects actually fail, and against what definition? The corpus has a success-story catalog with no denominator and a failure claim with no evidence attached. That is a gap in the sources, not a finding — and it is the first thing a new source here should close. See synthesis.

A third answer, with evidence behind it

data-cascades (CHI 2021, T1) reaches a layer neither source above does. Interviewing 53 practitioners in high-stakes domains, it found 92% had hit at least one compounding data failure and 45.3% more than one in a single project — failures whose cause sits in a decision about data taken before any model existed, and whose effect surfaces far downstream.

So the corpus now holds three claims about why systems fail to cross this gap, and they are not the same claim:

• infrastructure and people — ai-projects-fail-infrastructure-people, T4, body never readable, no evidence attached;
• the data, upstream of everything — data-cascades, T1, 53 interviews;
• not model quality, which both agree on.

Kept side by side rather than resolved. The T1 source is far better evidenced, but it studied practitioners recruited to talk about data practice in high-stakes domains, so 92% is a prevalence among people asked the question — not a failure rate for AI projects generally.

Which means the open question above survives intact. There is still no denominator. What changed is that the strongest claim in the corpus about this gap now points at the data layer, and pervasive-label-errors adds that even the yardstick used to judge a model is wrong at a measured rate. See training-data-quality.

One source crossing it in miniature (added 2026-08-03)

timesfm-2-5-forecasting-tutorial is a notebook, so it sits on the demo side by construction — but it ships the things the demo genre normally omits: a data contract (“regular spacing, one row per period, gaps as NaN (not dropped rows)”), an alignment constraint (max_context a multiple of 32), an API indexing convention that silently corrupts output if misread (quantile index 0 is the mean, not q10), plus NaN handling, short-context degradation, determinism checks and a warning that the forecast call mutates its input list.

That list is the texture this page’s two sources argue about in the abstract. It doesn’t supply the missing denominator, and one tutorial’s checklist is not evidence about failure rates. What it does give the corpus is a concrete sample of what “infrastructure and people” means at the smallest scale — a handful of contract and convention details that decide whether working code keeps working somewhere else.

Adjacency the hub kept separate

The parked _inbox record ai-test-automation-paradox (InfoQ, on AI generating brittle end-to-end tests at scale) shares this page’s shape — AI output that works in the small and fails in the large — but its subject is software testing methodology, not shipping ML systems. The hub declined to fold it into this spoke’s founding cluster for that reason, and the record stays parked as an adjacency. Worth revisiting if the boundary starts costing more than it buys.

Related

ai-projects-fail-infrastructure-people · ml-system-design · ml-system-design-case-studies · timesfm-2-5-forecasting-tutorial · machine-learning · synthesis