Enhancing Vulnerability Prioritization: Data-Driven Exploit Predictions with Community-Driven Insights

Jay Jacobs, Sasha Romanosky, Octavian Suciu, Benjamin Edwards and Armin Sarabi, arXiv 2302.14172 (February 2023, revised June 2023). The paper describing version 3 of the Exploit Prediction Scoring System (EPSS), the FIRST special-interest-group model that gives every published CVE a probability of being exploited in the next 30 days. Read at full text (13 pages) via a local PDF extract.

Tier T1, with a flag. The page rests on the paper's primary data and a stated evaluation, which is the T1 branch of the ladder. The authors built EPSS, so this is the makers grading their own model; the CVSS baseline it beats is scored on the same test day by the same team. No outside group's re-run is held here.

What it measures

The question this spoke's coordinated-vulnerability-disclosure page stops short of: once a flaw is public, which ones should a defender fix first?

• Base rate. From July 2016 to December 2022 the authors collected 6.4 million exploitation observations covering 12,243 of 192,035 published CVEs (6.4%). Most vulnerabilities are never seen exploited, which is why prioritization matters at all.
• Remediation capacity. Citing Kenna Security and Cyentia's tracking, the median company remediates 15.5% of its open vulnerabilities per month, and a quarter of companies manage under 6.6%.
• Labels and features. Exploitation labels come from Fortinet, AlienVault OTX, Shadowserver and GreyNoise sensors. The model uses 1,477 features: exploit code (Exploit-DB, GitHub, Metasploit), list membership (CISA KEV, Project Zero, ZDI), Twitter mentions, scanner templates, CVSS base metrics, CWE, vendor labels and CVE age. A gradient-boosted model predicts exploitation in the next 30 days.

The comparison with CVSS

Scored on the state of every CVE on 1 December 2022 and checked against exploitation over the following 30 days. Efficiency is precision (of what you patched, how much was exploited); coverage is recall (of what was exploited, how much you patched).

The area under the precision-recall curve is 0.7795 for v3, 0.4288 for v2 and 0.051 for CVSS v3.x. The headline "82% improvement" is v3's curve area over v2's.

The paper also scores simple heuristics. Patching everything with no privileges required covers 88.1% of exploited CVEs at 5.1% efficiency and costs 70.4% of all CVEs. Patching only what is on CISA's KEV catalogue costs 0.5% of CVEs at 53.2% efficiency and catches 5.9% of exploitation. Metasploit modules give 60.5% efficiency at 14.9% coverage for 1% of CVEs.

So a severity score is close to useless as a predictor of exploitation, and a curated "known exploited" list is efficient but catches little. The authors make the same point about CVSS in their introduction: severity measures how bad a flaw would be, not whether anyone will use it.

Limits the authors state

• The sensors are mostly signature-based network IDS/IPS and honeypots on public-facing perimeter devices. Exploits nobody wrote a signature for are invisible, and host-based, IoT, ICS/OT and medical attacks are under-observed. The labels therefore describe mass network exploitation, not targeted intrusion.
• Researcher and commercial scanning cannot be told apart from malicious exploitation; the authors believe it is a small share and have no estimate.
• Inputs such as GitHub and Twitter could be poisoned to move scores. Out of scope for the paper.
• One evaluation day (1 December 2022) carries every number in the table above.

Where it sits in this spoke

This is the spoke's second measurement with a stated denominator after endpoint-detection-attack-coverage, and it has the same shape as that paper's finding: the metric practitioners use by default (CVSS here, ATT&CK coverage there) does not track the outcome. It also inherits breach-forecasting's observability problem in a different form. EPSS can only learn from exploitation that sensors see, and the forecasting cohorts could only learn from breaches that were reported.

Cross-spoke: the 30-day probability is a forecast product, and its calibration is the kind of claim ../quant-trading-wiki and ../machine-learning-wiki hold methods for. Nothing was written there.