Hacker News Reader

Comment by thatsabadlook | Hacker News Reader

AI is actually a big reason why there are so many vulnerabilities over the past year and a half. People don't even know what their code bases look like anymore. It's really bad so sayeth my friends doing consulting.

A lot of companies cyber security posture is asking Claude to "find all security issues make no mistakes".

View on HN

That's certainly one reason, but frontier LLMs are also really good at finding existing vulnerabilities in completely human-authored codebases.

That’s not backed by data at all. We’re discovering more existing vulnerabilities.

It's both.

I would expect code written by the latest frontier models with a "find all security issues make no mistakes" pass to be more secure than average human code written in the pre-AI era (not a high bar).

A lot of people expect that. Apparently it's not working out so well in reality. My buddy who is big in this said something to the effect of, when a human codebase has human understanding it may have issues but it's strengthened by ai. When an ai codebase has little human understanding it's not strengthened by people very much. So his take was the most secure code bases he is seeing right now are human ones which use ai tools to help read and review rather then write. My guess is that's also cultural and industry specific. He is niched into businesses that actually care and usually doesn't take work from cowboy outfits.

I would expect it depends on the language. Less unchecked memcpy, more forgetting to check the user is an admin.

> I would expect code written by the latest frontier models with a "find all security issues make no mistakes" pass to be more secure than average human code written in the pre-AI era (not a high bar).

"If you write code as cleverly as you can, you are, by definition, too stupid to debug it."

Do you disagree with this? Because if you don't, then the rate of bugs/RCEs in a completely AI-written-software world will be exactly the same as the rate of bugs in a completely human-written-software world.

A combination of human-authorship + AI-review, with absolutely no AI-written code is going to have the lowest rate of bugs, because there is always one entity who understands the code (because they wrote it) and one entity who is smarter than the author who can find bugs.

The AI-generated code I see from SOTA models is absolute spaghetti, and they start running into problems spotting bugs after even a small number of refinements/changes to the software (say, 50).

The problem with all this spaghetti is that they are quick to write something that even they can't easily read. Even mediocre but experienced humans are not so eager to write something that they can't easily debug.

IOW, for any non-trivial problem, with AI-generated code you are guaranteed to get spaghetti that it cannot understand. With humans, it's only a probability that that will happen.

Well the AI machines don't have to rely on cleverness to find vulnerabilities, so maybe they're not limited by Kernighan's Law. Does throwing virtually unlimited sufficiently clever monkeys at spaghetti code find these vulnerabilities better than humans? Can you prove they can't?

At the point you starts asking to prove a negative, like "Can you prove that there's no God?", you should have the self-awareness to realise that no argument will convince you, because you are proceeding on the power of belief and belief alone.

If you don't have the self-awareness to realise that you are asking the same question as "Can you prove there's no God?", then I'm afraid no proof will ever be sufficient for you.

We're not talking about God, we're talking about a machine, here on Earth, we can poke and prod and explore and test. It's perfectly within the realm of logic to prove it can't do things. Can you prove the AI machine cannot solve the halting problem for arbitrary programs? Yes, that's a negative you can prove, so your objection about proving negatives needs qualification.

Asking you to justify a limitation you asserted doesn't mean I cannot be convinced. You're asserting Kernighan's law holds for AI, I'm saying there are other ways to debug than applications of cleverness. Does throwing more attempts at that spaghetti find vulnerabilities even when the thing that wrote it can't easily understand it? I'm not saying AI must be able to do it because you can't prove it can't. I'm asking what supports your claim that it can't.

Reply on news.ycombinator.com