Breach forecasting
Predicting which organisations will be breached, from properties measured before the breach. The method is always the same shape: assemble a cohort of organisations that reported an incident and organisations that did not, measure something about each, and train a classifier to tell the two apart.
This spoke arrived at the subject sideways. Growth edge 1 — does compliance correlate with not getting breached? — kept failing because no dataset carries a per-company compliance variable next to an outcome. The edge was re-specified as a design: sample both populations before an incident and follow both. Breach forecasting is that design, already built, twice, with a different variable.
The two executions held here
| posture-based-incident-forecasting (2022) | strisk (2024) | |
|---|---|---|
| Cohort | 200 victims, 600 non-victims (785 unique) | 795 victims, 3,016 non-victims (3,811) |
| Window | incidents 2017–2019 | incidents 2016–2019 |
| Incident sources | PRC, VERIS, HHS | PRC, VERIS |
| Features | 1,386 from 26 protocols (Censys) | scans, blacklists, darknet, certificates + Twitter signals |
| Headline | accuracy .73 ± .05, FPR .25 ± .10 | AUC 98.43% |
| Measured before the incident? | no — a single 2019 snapshot | only in a separate victim-only test |
What the pair actually establishes
The design is buildable, and expensive in a specific way. Most of the work is not modelling; it is attributing Internet assets to named organisations. One study resolves 2.3 million subdomains to 714,244 hosts; the other matches company names to ARIN registrations with two string-similarity metrics and a manual pass, publishing its near-misses. Anyone proposing a compliance cohort inherits this cost before the interesting question starts.
The control arm is wrong about 11% of the time. strisk treats its non-victim labels as noisy and flips 347 of 3,016 organisations into the victim arm — companies breached and not reported. The Variance paper names the same problem without measuring it, and adds a mechanism the other misses: honeypots on the networks of security-conscious firms teach the model that deliberately vulnerable hosts belong to non-victims.
How you pick your controls changes the answer. The Variance study collects non-victims three separate ways and finds the discriminating features differ by subset. The control group is a design decision, and reporting one is not optional.
Nobody has run the forecast properly. (Re-checked 2026-08-30: a fresh search returns the same two papers this page already holds — the Variance posture study and strisk — plus trade-press breach statistics. No prospective evaluation with a false-positive control has appeared. The claim holds.) Both papers' strong numbers come from separating two known populations after the fact. The one genuinely prospective test — strisk scoring breaches from posture measured 3 or 6 months earlier — falls to 42.85% and 71.42% accuracy, on a victim-only sample with no false-positive control at all.
Why it still does not answer edge 1
Every feature in both studies is externally observable, because that is what makes the method cheap enough to run at cohort scale. Compliance is not externally observable. An attestation is a document held by an assessor, and — per compliant-but-not-secure — it describes a date rather than a state, with only 29% of PCI-validated companies still compliant a year later. So the variable the edge wants cannot be scraped, which is why the design exists and the study does not.
That reframes the ask. The cohort design is not the missing piece; access to compliance status is. The realistic holders are insurers (who price on questionnaire answers and then see the claims) and regulators with audited populations, not researchers with a scanner.
Related
strisk · posture-based-incident-forecasting · compliant-but-not-secure · us-breach-dataset-compliance · endpoint-detection-attack-coverage · security-benchmark · defensive-security · synthesis