Hacker News Reader

Comment by traceroute66 | Hacker News Reader

> well-trained LLMs are more efficient in finding security holes than all but the best developers out there, even for OS kernel code?

No.

Like everything else an LLM touches, it is prone to slop and hallucinations.

You still need someone who knows what they are doing to review (and preferably manually validate) the findings.

What all this recent hype carefully glosses over is the volume of false-positives. I guarantee you it is > 0 and most likely a fairly large number.

And like most things LLM, the bigger the codebase the more likely the false-positives due to self-imposed context window constraints.

Its all very well these blog posts saying "LLM found this serious bug in Firefox", well yeah but that's only because the security analyst filtered out all the junk (and knew what to ask the LLM in the prompt in the first place).

View on HN

A 0% false-positive rate is not necessary for LLM-powered security review to be a big deal. It was worthless a few months ago, when the models were terrible at actually finding vulnerabilities and so basically all the reports were confabulated, with a false positive rate of >95%. Nowadays things are much better - see e.g. [1] by a kernel maintainer.

Another way to see this is that you mentioned "LLM found this serious bug in Firefox", but the actual number in that Mozilla report [2] was 14 high-severity bugs, and 90 minor ones. However you look at it, it's an impressive result for a security audit, and I dount that the Antropic team had to manually filter out hundreds-to-thousands of false-positives to produce it.

They did have to manually write minimal exploits for each bug, because Opus was bad at it[3]. This is a problem that Mythos doesn't have. With access to Mythos, to repeat the same audit, you'd likely just need to make the model itself write all the exploits, which incidentally would also filter out a lot of the false positives. I think the hype is mostly justified.

[1] https://lwn.net/Articles/1065620/

[2] https://blog.mozilla.org/en/firefox/hardening-firefox-anthro...

[3] https://www.anthropic.com/news/mozilla-firefox-security

> A 0% false-positive rate is not necessary

To be clear, I'm not saying 0% false-positive because that will always be impossible with any LLM.

However, to greatly over-simplify what I already said ...

The presence of >0 false-positives means you still need someone who knows what they are doing behind the keyboard.

The presence of an LLM, no matter how good, will never remove the need for a human with domain expertise in security analysis.

You cannot blindly fix stuff just because the LLM says it needs fixing.

You cannot report stuff just because the LLM says it needs reporting.

There may well be scope for LLM-assisted workflows, but WHO is being assisted is a critical part of the equation.

That is the fundamental point I am making.

If you can trigger address sanitizer from input outside the program, and the program may interact with untrusted input, isn't that always worth reporting and fixing?

> You cannot blindly fix stuff just because the LLM says it needs fixing.

> You cannot report stuff just because the LLM says it needs reporting.

Not today, maybe. Though with a good enough harness we're pretty close to seeing that already.

But in 6 months after another halving of the error rate? I wouldn't be so sure.

Reply on news.ycombinator.com