STRisk: A Socio-Technical Approach to Assess Hacking Breaches Risk

hicham-hammouchi, Narjisse Nejjari and Houda Benbrahim (ENSIAS, Université Mohammed V, Rabat), with Ghita Mezzour and Mounir Ghogho (TICLab, Université Internationale de Rabat). arXiv 2411.12435, 19 November 2024, IEEE format. T1 — primary empirical study, its own dataset and method, results compared against the prior work it extends.

Ingested via the research pass against growth edge 1: does compliance correlate with not getting breached? It does not answer that. It does something the edge had been asking for from a different direction — it runs the cohort design, on both populations, with measurements taken before the incident — and in doing so it puts a number on the objection that has made this edge unanswerable.

The design the edge keeps asking for

3,811 US organisations, victims and non-victims together, over January 2016 to September 2019.

• Victims: 795 unique hacking incidents, taken from the Privacy Rights Clearinghouse and the VERIS Community Database and filtered to hacking and malware only. Two sources rather than one, explicitly to reduce single-reporter bias — the same move us-breach-dataset-compliance's authors did not make.
• Non-victims: a random sample from the ARIN registry, about 4× the victim set and disjoint from it.
• Features, measured externally: open ports and services from Internet-wide scans, blacklisted hosts, darknet traffic, expired web certificates — plus a social block built from the organisation's Twitter presence (the paper's own terms: spreadability, agreeability).

Organisations were matched to their IP ranges through ARIN by Jaccard then Jaro-Winkler string similarity, with a manual check — and the paper prints its near-misses (florida bar association vs florida bankers association, white house vs whitehouse hotel). Asset attribution is the quiet hard part of every study of this shape.

The number this spoke came for

The control group is contaminated, and STRisk measures by how much. Treating the non-victim labels as noisy and correcting them with confident learning moves 347 organisations out of the clean set:

That is 11% of the supposed non-victims — organisations the model judges were breached and did not report it. The paper states the conditional directly: Pr(observed label = 0 | true label = 1) = 0.11.

This is the denominator failure compliant-but-not-secure and endpoint-detection-attack-coverage exposed, now with a magnitude. Any cohort study that sorts organisations into breached and not-breached from public reports starts with roughly one in nine of its controls in the wrong arm. A compliance cohort would inherit exactly this, on top of its own problem of pinning down what "compliant" means on a given date.

The headline, and the number that matters more

The abstract reports AUC above 98% (XGBoost, 98.43%) against 86.33% for technical features alone, and calls the difference "12% higher" — it is about 12 AUC points, not 12%. Social and technical features contribute roughly equally to the model (42% and 46%), and open ports and expired certificates are the strongest technical predictors.

Read the forecasting experiment instead. The 98% is retrospective, on noise-corrected labels, where the correction was made by a model trained on the same features — flipping labels toward what the features predict, then scoring how well the features predict the labels. The authors do report a control (training on the uncorrected labels still reaches 97.71% AUC), so the circularity is not the whole story, but the honest test is the separate one they run on breaches from March 2020 to March 2021 using only posture measured before each incident:

• 42.85% accuracy on three months of prior data
• 71.42% on six months

And that is on a victim-only sample, so it has no false-positive control at all. The gap between 98% and 42.85% is the difference between separating two known populations after the fact and calling a breach before it happens.

What it does not carry

No compliance variable, no audit status, no attestation — nothing an assessor would recognise. Every feature is externally observable by design, which is the point of the method and also the reason it cannot answer edge 1: compliance is not visible from the outside. The paper is evidence that the design is feasible and what it costs, not evidence about benchmarks. See breach-forecasting for the practice and posture-based-incident-forecasting for the independent run of it.

Related

breach-forecasting · posture-based-incident-forecasting · us-breach-dataset-compliance · compliant-but-not-secure · endpoint-detection-attack-coverage · security-benchmark · hicham-hammouchi