Hacker News Reader

Comment by max_k | Hacker News Reader

Same here. I got access to Mythos two months ago, it confronted me with decades of nasty bugs I wrote. This has been stressing me out a lot, caused sleep deprivation (the bugs follow me in my sleep, yes, really). All other progress is halted, I've been squashing bugs day and night. I see a light at the end of the tunnel.

Bottom line: it's a good thing. My software will be better after that. Even if it's currently not good for my ego (and my health). But I'm glad for all the tools that help me improve quality.

And sure, I can just ask LLMs to fix the bugs, which works well half of the time, and produces bad slop in the other half. This is where human expertise and judgement is still necessary to avoid letting our code bases drift into LLM spaghetti hell.

View on HN

My question is, did it provide POCs? I have had some security issues flagged that I tried my best to exploit, but failed miserably. Since I do not have any kind of privileged access the LLMs I use have been unwilling to actually prove that it can be exploited.

While I did fix most of the things because while they might not have been exploitable they were bugs waiting to happen, in all but one case I haven't been very impressed by any of the LARGE IMPORTANT SECURITY HOLES that were found.

No, the Mythos reports do not come with POCs. It's a lot of redundant prose explaining the bug it found (much more than I'm willing to read), it explains how it can be triggered, but doesn't generate actual exploit code.

But that doesn't matter for me. It shows me real bugs and less than 1% false positives. Some bugs are truly nasty and urgent, most are trivial and not exploitable at all, but they're bugs and so I'm fixing them.

Mythos isn't mythically superb for being the best LLM model, but only for not having these annoying guard rails that make all others randomly refuse to cooperate when you need them the most. Whoever invented guard rails - please stop! - they just slow the good guys down in the race.

Ah, Askell's work: https://mcauldronism.substack.com/p/an-automated-takedown-of...

While it is certainly good to do estimation oneself and not just blindly follow whatever the LLM outputs, keep in mind the recent PHP-related events ... Just because at some point one is not able to make a PoC, it doesn't mean, that no one else ever will be.

> I've been squashing bugs day and night

> This has been stressing me out

> the bugs follow me in my sleep

Sounds like you need some vacation.

> Even if it's currently not good for my ego

If you don't mind answering, how come? Is it specifically that you dislike how abusable your code was, or something else?

That part was not 100% serious. Only 90% or so.

To err is human, I know that as much as anybody. Before the advent of LLMs, there were several (human) audits of my code for vulnerabilities, but they found little, which I was proud of. And now, Mythos had a quick look and found hundreds and hundreds of bugs (many of which were critical and even more were really emberassing).

Of course, I knew there were bugs that just weren't found because the auditors weren't looking closely enough, didn't have enough time/budget etc., but this unspecific feeling is different from having all the bugs accumulated over a quarter century slapped in your face at once.

Code auditors are a crapshoot. I worked around financial auditors and they were much more consistent but thats likely because standards are more well defined.

If you get a good auditor they can find a ton, but most are box checkers there to make the audit company as much money as possible.

Quite possible. Ours were good, they just didn't benefit from LLM superpowers back then.

After my Mythos experience, I was wondering if there is really still a market for human code audits. I guess yes, but the work will be different, just like coding (and marketing and so on...) is different now. You still need to have an expert to wield the new powerful tools.

Reply on news.ycombinator.com