To Type or Not to Type: Quantifying Detectable Bugs in JavaScript (Gao, Bird & Barr, ICSE 2017)

Zheng Gao and Earl T. Barr (UCL) with Christian Bird (Microsoft Research), ICSE 2017, Buenos Aires. The first — and in this corpus the only — measurement of what a gradual type system buys a JavaScript codebase: Flow 0.30 and TypeScript 2.0 each detect 15% of public bugs.

Method

• Population and sample. 3,910,969 closed bug reports in GitHub JavaScript projects on 2015-08-19; a 95% / ±5% sample size of 384, rounded to 400 bugs, drawn uniformly from closed issues linked to fixing commits. They came from 398 projects, 32 to 1,144,440 LOC; the median fix touches 6 lines.
• Procedure. Check out the code just before the fix, add type annotations by hand for at most 10 minutes per bug, and ask whether Flow or TypeScript reports an error on the buggy version. A bug is ts-detectable if it does. Three authors calibrated on 20 bugs (Gwet's AC1 0.82–0.85; 0.90 once "unknown" was counted as detectable), then one author assessed all 400.
• Result. Flow flags 59 of 400, TypeScript 58; 18 could not be decided inside ten minutes and were worked to a decision (8 minutes to over an hour each). The two systems overlap almost entirely — 3 bugs only Flow catches, 3 only TypeScript — and all three Flow-only cases are TypeScript 2.0 staying silent when a possibly-null value is concatenated to a string.
• The annotation tax, under-approximated two ways: tokens (Flow needs 1.7 on average per detected bug, TypeScript 2.4 — Flow infers more and writes ?number where TypeScript needs a union) and time, which favours Flow more sharply.

What it settles and what it does not

It answers the spoke's gradual-typing edge in the one form the edge allowed — a defect-rate study rather than a survey — and the number is 15%, with the annotations added after the fact by people who knew where the bug was. That is the design's honest limit: it measures what the type checker could have caught given the right annotation, not what a team writing TypeScript from the start catches or misses. It says nothing about the other 85%, about bugs that never reached the public, or about whether annotated code takes longer to write. And the tools are a decade old: Flow 0.30 and TypeScript 2.0, before strict null checks became the default. A 2022 GitHub comparison of JavaScript and TypeScript project quality (Bogner & Merkel) exists and was not read.

Related

typescript · type-system · javascript · nodejs · synthesis