Comment by verdverm | Hacker News Reader

Comment by verdverm | Hacker News Reader

Parent Full thread

"secure implementation" is vague, for example, here is an OpenCode issue (links to related PRs/follow ups) that I worked with their team through to improve security.

Our hunch is that the Ai told them this was ok to do, but it broke invariants users relied on. Perhaps it was not "hackable" in the sense I think you are getting at, but it did open a very big security hole through choices and design. The critical thinking was lacking (imo).

https://github.com/anomalyco/opencode/pull/51337

The bots make mistakes all the time, even through multiple rounds of reviews. Thus I have zero faith that they will produce secure code reliably 100% of the time. More Ai does not magically fix this, they go down the wrong path and double down, or work around incorrect assumptions.

This is a good talk by a member of the Linux security team: https://www.youtube.com/watch?v=NnV_cWeoo5Q

Also worth noting that OpenCode Console was hacked days after the configuration regression and two PRs to change it back. Notable because it shows a pattern.

https://news.ycombinator.com/item?id=50133401

A poor design decision in OpenCode configuration handling is interesting, but not directly relevant to the question.

Assuming the question is whether not reviewing generated code is likely to create vulnerabilities despite SOTA models having audited the implementation.

Here it seems to have been a design choice, rather than an implementation mistake. I do not know what models were used and whether a review or audit has been done.

We are basically back at a vague the AI failed.

And sure, this happens, but there are some strong claims I'm reading here that are imo not evident:

- not reviewing implementation is foolish

- doing so will result in a flood of insecure software

- more AI cannot fix this

--

My arguments on each point:

- depends on risk tolerance, attack surface, error rate

- depends on actual performance of the proper setup I described on typical tasks, compared to the average engineer

- more AI can fix this, provided the audits become accurate enough to identify vulnerabilities with eg. 99.9% reliability

--

Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?

> Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?

No one can be sure either way, but I have high confidence perfectly secure code will remain out of the reach for man and/or machine, especially while they remain based on the transformer architecture.

The benchmark is hardly perfectly secure code.

Was not your concern that the generated code would be significantly less secure than what an average engineer produces?

The goal posts here basically moved toward infinity.

> Was not your concern that the generated code would be significantly less secure than what an average engineer produces?

No, generally I think it possible to improve code quality with agents. The main problem I see to this is eyes free coding and compounding errors therein. With regards to major security issues, I do not think transformers can reliably find them, even when stacked, looped, or teamed. I've seen this setup still miss things or make mistakes.

Reply on news.ycombinator.com