Security Posture-Based Incident Forecasting (Variance, 2022)

Dagmawi Mulugeta (Netskope), Ben Goodman (CyRisk) and Steven Weber (drexel-university, Electrical and Computer Engineering). Variance 15(1), 2022 — the peer-reviewed journal of the Casualty Actuarial Society, which is why an insurance publication carries an Internet-scanning study. T1: refereed, primary, and unusually explicit about its own design decisions.

Ingested via the research pass alongside strisk, against growth edge 1. Neither carries a compliance variable; both run the cohort design the edge asks for, and the two disagree about how well it works.

The cohort, built three ways on purpose

Incident report window 1 January 2017 – 1 January 2019.

• 200 victims, randomly sampled from 263 qualifying incidents across three reporters — 176 from the Privacy Rights Clearinghouse, 84 from VERIS, 3 from HHS.
• 600 non-victims in three separate subsets of 200, each collected a different way: the Cybersecurity 500 list, digital certificates, and DNS reverse lookups.
• 800 organisations, 785 unique. Assets resolved to 2.3 million subdomains, 1.9 million IPv4 addresses, 714,244 hosts, each represented as a 1,386-feature vector drawn from 26 protocols in Censys.

The three-way non-victim collection is the paper's real contribution and worth stating on its own: how you choose your controls changes which configurations look guilty. The three subsets do not produce the same important features. That is a warning to any future compliance cohort — the control arm is a design decision, not a given.

Results, and the claim about them

The features that separate victims are ordinary configurations rather than vulnerabilities — the paper is explicit that it profiles posture, not exposure. A misconfigured HTTPS server correlates with the victim label; SSH, being rare and secure, correlates against it. The reading offered is that victim organisations run undermanaged networks.

One claim to hold at arm's length. The paper says its performance is "consistent with prior studies." The prior study it names, Liu et al. (2015), reported 90% true positives at 10% false positives; this one gets 0.73 accuracy at a 0.25 false-positive rate — two and a half times the false alarms. Consistent in direction, not in magnitude. strisk, running a larger cohort with the same idea, claims 98% AUC. Three studies of one design, spread across a range that would mean very different things to anyone acting on the output.

Where the "forecasting" is aspirational

The posture is a single snapshot of each network, not a time series, and it is drawn from a 2019 Censys dataset while the incident window runs 2017–2019. So for much of the cohort the measurement comes at or after the event rather than before it, and the authors list time-series analysis as future work. Combined with strisk — which does run a genuinely prior-measurement test, and drops to 42.85% accuracy at three months and 71.42% at six — the corpus now holds two executions of the cohort design and neither measured the whole cohort before the incident.

The limitation that matters most here

Design Decision 3 states the assumption plainly and then denies it: that organisations which have not reported an incident are exemplary of good posture, "As Y. Liu, Sarabi, et al. (2015) have shown, there are numerous cases where this does not hold." Three mechanisms are named — breaches too small to trigger a reporting duty or suppressed by negligence; breaches reported just outside the window; and honeypots on Cybersecurity 500 networks, hosts left deliberately vulnerable, which teach the model that a vulnerable configuration belongs to a non-victim.

strisk measures what this paper only names: 11% of its own control group turned out to belong in the victim arm. Read together, the two sources say the cohort design is buildable and that its control arm is unreliable by roughly one in nine — before anyone tries to add a compliance variable that is not externally observable at all.

Interest declared

Two of three authors work for vendors in the space, and CyRisk sells cyber-risk analytics to insurers, so a result showing external posture predicts incidents is commercially convenient. The paper argues against its own strength more than for it — the modest accuracy, the three-way control experiment, the honeypot admission — which is the opposite of the vendor pattern this spoke keeps flagging in security-benchmark.

Related

breach-forecasting · strisk · us-breach-dataset-compliance · compliant-but-not-secure · endpoint-detection-attack-coverage · variance-journal · drexel-university