Hacker News Reader

Comment by nomilk | Hacker News Reader

> However, these agents still perform poorly on real-world vulnerabilities that are unknown to the agent ahead of time (zero-day vulnerabilities)

Only a matter of time until LLMs are brilliant enough to find sophisticated real-world vulnerabilities. Perhaps not every novelty, but certainly vulnerabilities that resemble any seen in the past.

The good news is in this arms race devs will be armed with similar tools, and the solution will be a few lines of CI code to run our own vulnerability-seeking LLM and prevent merging when (not if) vulnerabilities are detected.

View on HN

I've worked on zero days for decades and friends of mine with great academic pedigrees and deep experience in both AI and security are uniformly frustrated with the inability of even cutting-edge LLMs to reason in the way required for security bug detection.

> to reason

Should we expect current LLMs to be able to reason at all?

As the paper said GPT-4 maybe make a better scanner. The problem is too many people think all we need is better scanners, which is not new.

Yup. If you believe vulnerability research is a problem solved by throwing more bodies at it, I can see why LLMs are appealing.

I remain skeptical about their efficacy. You need a lot of nonlocal contextual information and very good reasoning skills to be an effective reverse engineer.

Give it time. GPT-4 is barely 14 months old. Things might look different in a year or two.

> Only a matter of time until LLMs are brilliant enough to find sophisticated real-world vulnerabilities.

There sure seems to be a lot of “hopes and prayers” around AI’s future. :)

My money is still on SMEs using specialized tooling backed by large funding vehicles. Aka, how things work right now.

The open source maintainer gonna pay this?

> pay

For a few API calls?

The proposed approach used multiple LLM chains. Then you don't know how much context you need. I imagine it can get expensive. Per Commit.

OpenAI's latest (GPT-4o) model is US$5.00 / 1M tokens [1]

For large repos it might get expensive, but many will have orgs footing the bill.

To satiate curiosity, I made a fresh rails app and it has 9141023 characters, let's /5 to estimate 'tokens' (wild guess), so to scan an entire rails app, that's about $10. Not nothing, but not back-breakingly expensive. Scanning could be reserved for non-trivial PRs and important applications where vulnerabilities are especially likely (e.g. perhaps not static sites, demos, or apps not in prod or a live environment).

Scanning only new or edited files, along with those they interact with (where sensible patterns could emerge over time) could decrease the total volume of files needing scans, thereby reducing costs.

[1] https://openai.com/api/pricing/

Or $2.50 / million tokens if you run it in batch mode (results in up to 24 hours, though in practice much faster than that): https://platform.openai.com/docs/guides/batch/getting-starte...

You don’t need this per commit unless you’re especially paranoid. Can easily do it just on a per release basis. The problem is that right now it’s still all wasted cost - this thing can’t really do the thing needed.

Most OSS maintainers don't even pay for GitHub, so you're competing with free. I for one certainly won't pay for a service for the OSS projects I maintain for free.

(Or another framing: I might pay for such a service, if the service could demonstrate that it would save me N hours of work fixing bugs instead of costing me N hours of work triaging false positives. Nobody has presented such a demonstration yet, which is also why turning non-LLM program analysis tooling into paid products is such a struggle!)

There's a breed of OSS maintainers that really do care about whether their project is used and satisfying to the users, vs. building stuff just because. In a situation where having your project used vs. deemed insecure is a matter of forking out $10 for an exhaustive pass with GPT-4+-based tool, such maintainers may actually care to pay.

Or Github will fund it for them, like they always do.

The implication that I'm building stuff "just because" isn't appreciated.

What I really care about in these instances is protecting my time: I have only so many hours in the day, including time that I rightfully reserve for not doing unpaid maintenance. If the X hours that I previously spent doing OSS maintenance is now partially occupied by triaging false positives, my projects are both worse off and I'm less inclined to work on them, because I equate that work with mindless churn rather than helping my users.

You've pointed out that GitHub will fund it, which is potentially true! But observe that this hasn't gone all that well with CodeQL, which they also fund for public repositories: I've had to disable it on a lot of my projects, because the FP/FN ratio simply wasn't worth it. I did, and do, better for my users dedicating my time to actual issues filed.

> The implication that I'm building stuff "just because" isn't appreciated.

It seems this came across not the way I intended; I apologize - I meant that very positively, as a set of possible motivations related to the code/project, problem domain, and/or author themselves, as opposed to product thinking. Like, e.g. making for fun, as part of learning, or because it's useful to author - where other users are at best a secondary concern.

Got it, apologies for the misunderstanding as well.

Reply on news.ycombinator.com