Hacker News Reader

Comment by tptacek | Hacker News Reader

As is the case with SOC2, the "vulnerability scan" requirement here is likely to be meaningless; any automated process that can plausibly be described as instrumental in finding some kind of vulnerability is a "vulnerability scan", so all you have to do is run nmap.

View on HN

they have comment/request for information sessions for HIPAA rule proposals, which your input would be valued.

I don't think the rule would be better with more detailed vulnerability scanning requirements! All these things inexorably become races to the bottom.

Yes, exactly, the rules are intentionally broad and vague. You can wave paper at most of them and technically succeed. And then when you release accidentally PHI for the first time and your bullshit comes to light, your chickens will come home to roost. Doing a good job on compliance is less about security and more about staying out of jail.

The ideal flow here is:

1. Do good security and operations.

2. Overlap the minimum subset of your existing good security and operations as evidence for whatever compliance regimes help you get paid.

3. Get paid.

Nobody is suggesting that you bullshit the auditors. They’re suggesting not letting the auditors accidentally trick you into letting step 2 get in front of step 1.

If it is like SOC2 I would expect respected auditors to reject that

But there are no auditors required for HIPAA. Only the government (HHS OCR) itself can enforce the standards.

Thanks for the clarification, in that case the text is indeed really weak. Does that system work in practice, or are companies just claiming they are HIPAA compliant with close to no actual auditing mechanism?

You get that the technical controls in SOC2 are also extremely weak, right?

Sure, yes. The way I understand SOC2 relies on the auditors to set the effective standard. So it really depends who audited you

SOC2 auditors are accountants. A SOC2 auditor verifies only that you're doing what you say what you're doing.

Obviously, yes

A SOC auditor who tells you that you can’t use an nmap scan to meet SOC2 obligations is a bad SOC auditor, because they’re attempting to enforce a constraint on you that SOC2 does not.

But the far more likely thing is that a medium SOC auditor, upon being told “we do our vulnerability scanning with nmap”, would say “I haven’t heard of nmap. You should use Tenable,” and if you’re letting SOC auditor drive your engineering you’d make a mistake and accidentally think that meant you needed to change your answer for SOC2 and go buy Tenable licenses.

The whole thread drifted way too far from a very mild push back I had regarding the claim « any automated process that can plausibly be described as instrumental in finding some kind of vulnerability is a "vulnerability scan" ».

My experience is that no, SOC2 auditors won’t consider literally any automated process of that sort as compliant. Which in no way implies the auditors are forcing you to use a licensed tool or driving your engineering.

I will stop that thread here, I don’t think that exchange is productive

And the way they verify you are doing what you say you are doing is by asking you to provide evidence, which is usually pretty easy to demonstrate that a policy was followed once or twice, a lot harder for them to pick up consistency issues or exceptions.

I've had SOC2 auditors choose a random commit from our GitHub history, then ask to see the associated Jira ticket, logs from the build and deployment, etc. Hard to reliably pass an audit if you don't know which changes they'll drill down into.

They also asked for proof of system-enforced processes (e.g. GitHub branch protection rules and the setting for enforcing peer review for each change) which were basically proof of consistency.

They do that because in the DRL process you specified a change management process involving Github and Jira. If you specified a different process (for instance, Post-It notes applied to the bathroom wall), they would randomly ask for evidence about those Post-It notes.

That's what we're talking about when we say virtually any tool you can come up with will satisfy "vulnerability scans". For Cloudflare, it was nmap. I think they're right about this.

An example of what the parent commenter meant is more like:

1. You write a DRL that says “we do a disaster recovery test annually to ensure that we can restore a production backup”.

2. It takes you 20 tries in 2026 to do a successful restore because your find out the first 19 times that your backup is broken and incomplete.

3. You never have to mention the first 19 tries to the auditors when you prove you did do a successful DR restore.

If SOC2 relies on competent auditors (and you're right, it does), than it is an ineffective standard (and it mostly is).

It absolutely doesn't rely on competent auditors. The AICPA that fabricated SOC2, is the same AICPA that gives licenses to the auditors. At some point, they opened it up to getting it over the internet.

Indian companies open up shell businesses in Wyoming and elsewhere, get "certified", and offer rubber stamp auditing services. Few ever check if you actually have SOC2, or what auditor you used (since, by definition, they need to be "legit").

By the way, the AICPA website was recently throwing https expired cert errors. Their solution after weeks of me pointing it out on twitter, was to take down the entire website.

It's been a few years since I worked in this space, but HIPAA doesn't really work under the same kind of legal framework. Oversimplifying here, but basically HIPAA defines what constitutes personal health information, how such information may be used, and establishes monetary penalties for improper use and unauthorized disclosure. The law doesn't have any certification standard, no more than the prohibition on stealing cars does.

Maybe there's some kind of third party certification system to support signing information sharing agreements ("BAAs") with other health information systems. I worked at CMS on first-party stuff so I'm not really familiar with how it works in the private sector.

No? Like, wildly no? This is a big part of why you pay for the most respected auditors.

I guess we had different experiences. The ones I interacted with were ok and wouldn’t have accepted a simple nmap here

I'm not being snarky when I say that not getting your automated vulnerability scan, whatever it might have been, past your SOC2 auditors is a skills issue. SOC2 audits are not technical and the vulnerability scan control in SOC2 is categorically not meaningful. Cloudflare wrote a whole post about this.

FWIW I agree that SOC2 for automated vulnerability scans has a really low bar and isn’t too meaningful. At no point did I defend SOC2 here. The bar I’ve seen is above “just an nmap”, which is pretty bad standard IMHO. You seem to be reading way too much in my comments

I brought up nmap. You said you'd expect respected SOC2 auditors to reject it. I don't just think that's not true, I know it not to be true.

I know, that’s already established. I already acknowledged we had different experiences. I have no idea what you’re pushing for at that point

Just to clarify, this is a bugbear of mine. It's nothing personal with you, but I've spent the last 6 years or so evangelizing the idea that people should minimize their SOC2s and not get pushed around by auditors or evidence collection platforms like Vanta, because that drives a lot of terrible security engineering, and the hypercompetent best-staffed security orgs in the industry all push their SOC2 auditors around.

Compliance and security are entirely different practices in a well-run firm. Security can inform compliance. Compliance should not inform security engineering.

If you search my name and "SOC2" in the search bar below, I've expanded on this quite a bit.

As just one data point here, let me say thank you for all your writing on it; it was super useful to have things to point at to say “we don’t have to just blindly do a thing the auditor suggested!” for our SOC2.

tptacek just hates soc. its probably not personal.

We got some value from it! I just think it's important to remember what it actually is, rather than axiomatically deriving what you think it should be.

> so all you have to do is run nmap.

This is ignorance at best. No one who has ever actually had to do SOC2 compliance legitimately has just run nmap and been done with that.

>No one who has ever actually had to do SOC2 compliance

while i find tptacek's opinions very strong on the subject, you would be extremely mistaken to think those opinions were formed without experience

Yep, see that now. I'll try less next time.

https://blog.cloudflare.com/introducing-flan-scan/

You're right. I'm wrong.

Cloudflare continues to prove it's a shit service, too.

They're 100% right, substantively, about how they're satisfying their SOC2 control requirement here. I'd be alarmed and skeptical of any security practice that was led by compliance requirements.

I have direct experience telling a soc2 auditor that the approval control for installing applications on “endpoints” was a message to a slack channel that was assumed approved.

To satisfy the audit they looked at an app that was installed on a laptop that was not part of our base image from the previous 6 months and a screenshot of the message where the user “asked” to install it.

You can literally get a soc auditor to write up whatever you want as a control and if they don’t explain that and encourage it you should find a new auditor.

Yep, I see I was wrong.

I can see that my experience was in putting actual effort into things.

The problem with these audit regimes is that done poorly they cause you to put effort into the wrong thing.

We could have codified some more complicated approval process for apps on laptops. That could involve tickets and strenuous review and making everyone spend tremendous resources trying to keep an endpoint secured that way.

But we didn’t believe that was the right way to do it. Instead we assumed no matter what we did laptops were going to get owned up. So we focused on blast radius, detection, forensics and time to remediation. We’d need all those things even if we believed in a pre approval process.

But we didn’t document most of those things in audit scoped controls, because that would have ossified them. If we wanted to improve our detection regime we didn’t want audit ceremony to get in the way.

I’m comfortable with those decisions and would make them again. We worked very hard to make sure our endpoint security was as good as we could make it.

A specialist accountant certainly would not have improved the process.

So no, I don’t think the solution we came up with was not putting effort into the problem, it was thinking very hard about it and not letting a checkbox audit cause us to make bad decisions.

Reply on news.ycombinator.com