Compliant but not Secure: Why PCI-Certified Companies Are Being Breached

Christian Moldes, CSIAC Journal (Cyber Security and Information Systems Information Analysis Center, a DoD information analysis center), Spring 2018 issue, published 9 May 2018.

The spoke’s standing edge asked for incident data carrying a per-company compliance variable set against outcomes — us-breach-dataset-compliance could only supply regulatory coverage, with no company in it holding an audit status. This article is where the per-company variable actually appears, and reading it settles the edge in a way the edge did not anticipate.

The two figures

Both from Verizon’s PCI compliance research, synthesized here rather than collected by the author:

• Only 29% of companies are still compliant one year after validation. Compliance is granted at a point in time and decays through the year.
• Across a decade of breaches investigated by Verizon’s forensics team, not one breached company was found to have been fully PCI DSS compliant at the time of the breach.

Why the second figure does not close the edge

It is a statistic with no denominator, and this spoke has now seen that failure mode three times.

Verizon’s forensics team examines breached companies. Reporting that none of them was compliant describes the numerator only; nothing in the figure counts compliant companies that were not breached, and with roughly 71% of validated companies out of compliance a year later, “the breached ones were non-compliant” is close to what you would expect if compliance made no difference at all. The claim needs the base rate to mean anything, and the base rate is not in it.

That is the same defect as the coverage counts endpoint-detection-attack-coverage disqualified, arriving from the compliance side instead of the detection side. It is worth naming as a pattern in this domain: the measurements that get published are the ones taken on the population that already failed, because that is the population somebody was paid to investigate.

What it does establish

The mechanism, which is more useful than the correlation would have been. Moldes separates two failure classes behind the “certified and breached” outcome:

• Organizational — compliance treated as an annual event rather than a continuous process, monitoring neglected between assessments, and resources weighted toward prevention over detection, so controls decay in the gaps.
• Assessor (QSA) — flawed assessment methodology, insufficient expertise, and shortcuts in evaluation rigor. The certificate can be wrong at the moment it is issued.

Read with the 29% figure, that is a coherent account of how an audited control set stops describing a company shortly after it is audited, and it goes some way to explaining Target, Sally Beauty and Heartland — all certified, all breached, all cited here as cases.

Evidential standing. T2: a named analyst writing in a DoD-affiliated journal, synthesizing Verizon’s published research and documented breach cases. No original data collection, and the underlying figures come from a company that sells PCI assessment services. The article’s argument is about assessment practice, and it is stronger than the statistics it borrows.

The edge, re-specified

The wanted object was a compliance variable set against outcomes. What exists in this literature is compliance status recorded only for the breached, which cannot answer the question no matter how many years it accumulates. So the ask changes: what would settle it is a study that samples compliant and non-compliant organizations before any incident and follows both — a cohort design, or an insurer’s book, or a regulator’s audited population matched to subsequent incident reports. Absent that, the honest statement for this wiki is that nobody has measured whether compliance reduces breaches, and the most-cited figure claiming it does is denominator-free.