If its more "bugs than humans can fix", its also likely "more bugs than humans can triage and verify as legitimate bugs".
Sorry, garbage in garbage out doesnt fly. I feel sad for open source maintainers having to deal with this massive slop-ageddon.
Sorry, garbage in garbage out doesnt fly. I feel sad for open source maintainers having to deal with this massive slop-ageddon.
There's a big difference between randos getting agents to hallucinate bug reports on GitHub, vs. when your own internal workflow of LLM-assisted code review is identifying real, legitimate problems in your new features.
This totally matches my experience. It's actually kind of insane how well LLM's like Claude are able to find edge cases, race conditions, overflows, states you didn't think could be reached. It really makes you think about code quality in a whole new way. Becoming aware of so many potential bugs when you write the code, vs. finding out from users over the coming months/years -- it forces you to really consider the costs of quality all at once in a way that simply wasn't possible before.
There’s plenty of ways in which you can guide LLMs, and sandbox them, to get a much higher hit rate.
And that’s before the increase of validity of bugs compared with a year ago — we’re past the peak of slopageddon, and maybe even at a point where a higher percentage of security bugs reported are valid than was the case pre-LLMs.
Opus continues to tell me that my API is missing a `/ready` route that a smoke test can check against. I assume it’s part of its memories, but it’s now 3 or 4 weeks that it won’t let go of that whenever tasked with test design.