How does Endpoint Detection use the MITRE ATT&CK Framework?

Virkud, Inam, Riddle, Liu, Wang and Bates (University of Illinois Urbana-Champaign), USENIX Security 2024. T1 — peer-reviewed, and by the authors’ account the first full analysis of how endpoint products actually use ATT&CK. Full text extracted locally from the authors’ PDF.

What they measured

Three commercial ATT&CK-annotated detection rulesets — Carbon Black, Splunk, Elastic — plus the crowdsourced Sigma ruleset, against ATT&CK v11 (14 tactics, 191 techniques). Three questions: how much of the matrix do products cover, why is the rest uncovered, and do two products that cover the same technique detect the same thing.

The findings

Coverage is about half, and half of that is padding. “Products do not attempt to cover all ATT&CK techniques, with coverage ranging from 48% to 55%.” Filter out low- and medium-risk rules — the ones a SOC will not prioritise anyway — and technique coverage drops to 25–26%, roughly half the headline. Sigma, being crowdsourced and unconstrained by quality control, reaches 79% with the same shape of distribution.

A quarter of the matrix is implemented by nobody, largely because it can’t be. 53 techniques (27.7%) appear in none of the three commercial rulesets. Three coders labelled all 53 and attributed a reason for 51 (96.2%): ineffective detection method 21 (39.6%) — many of which MITRE itself notes will throw high false positives — targeting non-host infrastructure 13 (24.5%), client-specific 9, vague detection method 9, targeting third parties 8, and a tail of provenance-based, low-level, removable-media and human-factor techniques. The authors’ conclusion is the one that matters: “covering all these ATT&CK techniques could mean sacrificing the quality of the detection rules.”

The same label does not mean the same detection. Of 191 malicious entities named in rule metadata, only 9 (4.7%) are named by all three products and 153 (80.1%) by just one. On the 37 entities named by at least two, “vendors are applying ATT&CK technique labels in equally-valid but inconsistent ways” — rules overlapping in the behaviour they detect but differing in the techniques and tactics annotated, “such that a security analyst may reach different conclusions about the same threats depending on which product they are using.”

What it does to this spoke

It kills a metric the spoke was leaning on implicitly. Coverage counts are not comparable across products, are inflated by rules nobody acts on, and cannot be driven to 100% by any competent vendor. The finding rhymes exactly with security-benchmark: turn security into a score and the score becomes the thing optimised, whether the underlying property moves or not. Also note the direction of travel against soc-fast-slow-thinking‘s alert-fatigue framing — some of the uncovered techniques are uncovered because covering them would generate the noise the triage tier exists to absorb.

One limit to record: the data is one snapshot of four rulesets against ATT&CK v11, and rulesets change. The structural findings — unrealizable techniques, inconsistent labelling — do not depend on the snapshot; the exact percentages do.