Hacker News Reader

Comment by 9cb14c1ec0 | Hacker News Reader

> We all know that LLMs were used to find these vulnerabilities

How do we know that? You seem quite certain.

They pitch their company as finding bugs "with AI". It's not hard to point one of the coding agents at a repo URL and have it find bugs even in code that's been in the wild for a long time, looking at their list that looks likely to be what they're doing.

The list is pretty short though for 8 months. ossfuzz has found a lot more even with the fuzzers often not covering a lot of the code base.

Manually paying people to write fuzzers by hand would yield a lot more and be less expensive than data centers and burning money, but who wants to pay people in 2026?

Bugs are not equivalently findable and different techniques surface different bugs. The direct comparison you're trying to draw here doesn't hold.

It does not matter what purported categories buffer overflows are in when manual fuzzing finds 100 and "AI" finds 5.

If Google gave open source projects $100,000 per year for a competent QA person, it would cost less than this "AI" money straw fire and produce better results. Maybe the QA person would also find the 5 "AI" detected bugs.

This would make sense if every memory corruption vulnerability was equivalently exploitable, which is of course not true. I think you'll find Google does in fact fuzz ffmpeg, though.

Google gives a pittance even for full ossfuzz integration. Which is why many projects just have the bare minimum fuzz tests. My original point was that even with these bare minimum tests ossfuzz has found way more than "AI" has.

Another weird assumption you've got here is that fuzzing outcomes scale linearly with funding, which, no. Further, the field of factory-scale fuzzing and triage is one Google security engineers basically invented, so it's especially odd to hold Google out as a bad actor here.

At any rate, Google didn't employ "AI" to find this vulnerability, and Google fuzzing probably wouldn't have outcompeted these researchers for this particular bug (totally different methods of bugfinding), so it's really hard to find a coherent point you'd be making about "fuzzers", "AI", and "Google" here.

My guess is the main "AI" contribution here is to automate some of the work around the actual fuzzing. Setting up the test environment and harness, reading the code + commit history + published vulns for similar projects, identifying likely trouble spots, gathering seed data, writing scripts to generate more seed data reaching the identified trouble spots, adding instrumentation to the target to detect conditions ASan etc don't, writing PoC code, writing draft patches... That's a lot of labor and the coding agents can do a mediocre job of all of it for the cost of compute.

If it's finding exploitable bugs prior factory-scale fuzzing of ffmpeg hasn't, seems like a pretty big win to me.

For sure, and I think it expands the scope of what factory scale efforts can find. The big question of course being how to handle remediation because more bugs without more maintainer capacity is a recipe for tears.

I can't speak to what exactly this team is doing but I haven't seen any evidence that with-robot finds less bugs than without-robot. I do have some experience in this area.

Reply on news.ycombinator.com