Hacker News Reader

Comment by jerf | Hacker News Reader

Those "assessments" amount to sending over a document that says

    1. Are you secure?
    2. Are you secure?
    3. Are you secure?
    4. Are you secure?
    ...
    37. Are you secure?

and the company sends back

    1. Yes.
    2. Yes.
    3. Yes, definitely.
    4. Oh yes.
    ...
    37. Yes.

There's a lot more words, but not necessarily a lot more value in those words then what I have here. Some, I admit, but not necessarily a lot.

Maybe at the high government end a real assessment is done where the experts of the client's choosing go into the provider's actual environment and makes a real assessment. But from what I've seen it's self-reporting the vast majority of the time, and the provider could honestly believe they're running a tight ship and not realize there's one setting in one AWS account that's just a bit too open and oh no my database. (Or perhaps rather "oh no your database".)

View on HN

Self-assessments are definitely an option some companies use, but there is a middle ground between a full assessment conducted by the customer and a self-assessment. ISO, SOC2, etc. all provide what I would consider to be better than a simple self-assessment as they require a third-party audit. They aren't anywhere close to perfect but they are significantly better than a self-assessment IMHO. There are no guarantees, of course.

I think you haven't seen enough. Self assessment is a great start. I've seen companies demanding their vendors to go through pentesting from several different pentesting companies, and then addressing all high-severity issues to the satisfaction of the pentesting company before officially allowed as a vendor.

Naturally you still have to trust the pentesting company to do a good job but it's better than just a self assessment.

Another way of looking at it is that if a company doesn't truth you to answer truthfully, why are they choosing you as a vendor?

Presumably you'd be lying about the thing they'd be buying too, or innumerable other things.

And 'subsequent lawsuit' is usually a powerful motivator to be honest.

The next layer of cynicism down is more complicated. I'm going to make this up to protect the guiltocent, but I've been on the receiving end of security assessments that ask the moral equivalent of "Do you salt the MD5 hashes you use to store password hashes?" and we end saying Yes in some large number of words because we use bcrypt/scrypt properly, or some industry SSO, or our systems communicate with an internal CA TLS authority where "passwords" aren't even in sight, or whatever other solution leaves salted MD5 in the dust.

It's literally a deal breaker to send back No, so there's a lot of incentive to do what it takes to send a Yes back.

And "subsequent lawsuit" is a very distant threat in this case.

Less cynically, there are some standards that do have some non-zero teeth in them. Some audits are challenging and at least rate "a good exercise". But that's what I mean by even if the vender truly believes they are compliant with ISO-Thirty-Three-Million-And-Two-Subrevision-A24, they're just one error away from ohnoyourdatabase anyhow.

The net-net of it all is that, as I sort of alluded to, are these assessments worthless? I mean, no, not quite literally zero. If you send one of these documents to a startup of two dudes and a cat and they claim ISO-Thirty-Three-Million-And-Two-Subrevision-A24 compliance, they're lying and the person examining their assessment at least has a chance to be suspicious about it. But in real terms, are they going to be the difference? Unlikely.

Or, let me put this another way. I would bet substantial money Okta has in their possession a response to their questionnaire from Rightway Healthcare in which Rightway Healthcare sings the praises of their immense, extraordinary, back-breaking, industry-leading security efforts, complete with citation of the relevant industry standards they comply with, and that it looks as good as anyone else's answers.

Yet, here we are.

Yeah. This is what always trips engineer-types up when dealing with legal. A contract isn’t code. There is usually, on both sides, a certain degree of BSing that goes on. Put another way, both parties are usually knowingly taking on risk associated with non-compliance. “We meet the security requirements in spirit only” is easily one of the more tame instances of this.

It's not that clear cut.

A contract is a probabilistic promise.

According to the writer's understanding of the law (and the risk tolerance they have for being caught) these are the terms they're putting to paper.

Generally, that means the writer's legal team feels confident that if everything goes sideways and they're standing in a courtroom, they have the best possible chance at winning the case with the language they used.

Now everyone around legal (e.g. sales, marketing, product, etc.) likely has different incentives. But the entire reason legal is somewhat firewalled is because they're the ones thinking about that future courtroom.

So it's less "there's a certain degree of BSing" and more that sometimes sales gets their preferred language and sometimes legal wins.

And of course, sometimes neither of them know relevant technical details and both their proposals are jibberish.

Reply on news.ycombinator.com