Stop Using Vulnerability Counts to Measure Software Security

cacm.acm.org

15 points·zdw··4 comments

Totally agree with this take. That said, you must account for the size of the codebase and associated attack surfaces and compare that with frequency of a particular bug class consistently popping up on every CVE.

In other words, you shouldn't use vulnerability counts, but you can discern patterns of vulnerability to intuit something about the nature of the codebase.

For example, RCE vulnerabilities on Chrome, especially under V8 while not very common they happen commonly enough to suspect that maybe there is some code quality issue. However, if you look at the sheer size of V8, and how much scrutiny and research it undergoes, it is surprising there aren't even more critical vulns being found all the time. JIT is inherently a risky endeavor.

There is no objective way to measure the security of an inherently insecure system.

Assume breach.

Okay, but my boss still demands I give him a metric. I'm not allowed to tell him, "Just trust me bro" when I'm asked how much our security has improved over the past sprint. I'm supposed to give hard numbers, and the OP at least offers an alternative for that.

Pick something that resembles a vuln→patch interval,

not just a context-less number that means they're popular, audited, or reviewing their OWN code all the time.

Instances where 0-days can't be used in isolation are a perfect example of where nontechnical people absolutely need to "just trust" someone to triage, and perform threat modeling for them.

Reply on news.ycombinator.com