Hacker News Reader

Comment by enthuseyourcurb | Hacker News Reader

As someone working at a bigtech co that has spent every single working day (and many weekends) since the launch of Mythos patching serious, company-ending type of security issues, these types of posts really irk me.

Public datasets of CVEs being underwhelming doesn't mean much if all of the issues are in private company codebases that will never get a CVE number assigned and will never get a fancy blog post. You're never going to hear about them, and the people working on them probably can't tell you about them, but that doesn't mean they don't exist.

We're all worried about OSS having security issues because of how widely used things like PHP or curl are... but the same can be said about email providers, document collaboration SaaSes, payment processors, etc that millions of companies depend. Those companies will never be in datasets like this, and they'll never publicly disclose how many issues they found.

View on HN

Yeah I am much more worried about some critical vulnerability in, say, the controller logic for pumps at sewage treatment facilities vs. a sandbox break in Chrome (which will be patched well before the blog post...). The broad issue is that you can turn DeepSeek-v6-unhinged loose on an arbitrary IP and exploit ~everything that is not developed and actively maintained by a billion-dollar company

One example they use in the blog post is curl. Not maintained or developed by a billion dollar company.

I suspect open source projects are better off than proprietary software - they already have many eyes on them, whereas commercial software that’s not a prominent and profitable target (e.g. windows, ios) hasn’t had much attention until LLMs made attacking them significantly easier

The average enterprise software is swiss cheese. It seems the hacker news type that likes to post just doesn't realize how much business/enterprise software exists out there. An F100 company has thousands, maybe tens of thousands of applications that you've never heard of and never will unless you work for them. Industry has applications ran by relatively few people, but it's keeping processes that make the modern world work alive.

Worse is everyone is like "Well, this stuff is protected or air gaped". Yea, it's behind firewall, or even private networks commonly, but the moment you get inside that network you have tons of lateral movement and could cause unbelievable damage in minutes.

    > Not maintained or developed by a billion dollar company.

Directly? No. Indirectly? Yes. They are mostly funded by billion dollar companies.

Mostly funded is different from “maintained” or “developed”.

Afaik everyone working on curl has a day job.

> I suspect open source projects are better off than proprietary software - they already have many eyes on them

This has been disproven by the sordid history of OpenSSL. It was open source, critical, and full of holes.

https://www.mend.io/blog/how-the-heartbleed-vulnerability-sh...

Single datapoint does not prove anything at all, but it is a data point.

> The broad issue is that you can turn DeepSeek-v6-unhinged loose on an arbitrary IP and exploit ~everything that is not developed and actively maintained by a billion-dollar company

Agreed. My fairly obscure company got hit by something like this last weekend. What stood out was the speed: attackers found something small, used it to uncover something else, and pivoted to clever exploits for the new thing within minutes. It hit a dozen of our apps in parallel within the span of a day, from a rotating pool of residential proxies, Tor, GCP and Cloudflare IPs.

I'm fairly confident it was LLM-driven. The payloads and throwaway accounts had names like "PT59553 TEST IGNORE" and "PocCorp1659r2", and it adapted to error messages the way a person reading them would, just never sleeping. A non-tech company with 100x the surface area and much more valuable data could be in big trouble.

Of course I used LLMs for the defense too, and they were amazing at log forensics and fixes. The annoying part was when I asked Opus 5.5 for clear repro cases of exactly what the attackers did, I hit the cyber safeguards. Opus was trying to keep the exploits for my own bespoke applications away from me. It was happy to fix things of course, but I worry if it really can't 'think' about actual exploits, its fixes will be less informed.

the broad issue is actually that those are connected to the internet at all and there is no real enforcement or penalties for it

billion dollar companies are not exempt from this, in fact they tend to be the worst offenders

Also work at a big tech company that has fixed many thousands of triaged/verified security issues found by Mythos which would never show up in CVE reports/blogs.

My understanding is that few have been critical vulnerabilities on their own, but LLMs are now able to piece together 10+ less critical vulnerabilities to create a critical exploit. So, some of the classic triage practices that rely on standalone CVSS scores should be coming into question.

Even if you have just have 1,000 low severity vulnerabilities today that don't add up to something critical, they might add up to something critical when the next model generation arrives, so you need to get ahead of those latent vulnerabilities. The safest approach is to fix everything, which is understandably difficult for OSS.

If that isn't possible, then we need new ways of prioritizing/triaging bugs... which would also make for a more interesting blog post than claiming Mythos is 'underwhelming'.

With code public many of the actively used and maintained open source projects have better code quality than proprietary or big MNC ones. Forget Mythos level models but even smaller models are able to find large number of issues in closed source tools in my example. One reason obviously is the peer review and quality checks by users/hackers/maintainers over the years.

Have you seen any patterns for the causes of the error you have seen? For example, more training for developers? Or maybe just systems were really very complex that led to bugs?

> these types of posts really irk me.

They shouldn't. AI is great at some things and underwhelming at others, and it's never predictable if you're going to get the great response or the silly one and when you'll get it and over what topic. And you and the author combined are illustrating one scenario where that jaggedness shows up.

Possible you might be believing the LLM too much. It seems to flag so many “critical, company ending” security issues, the ones surviving any semblance of scrutiny I’d estimate at 0.5%.

Make sure you’re still thinking.

[flagged]

I'm replying to a new, green account, immediately raising suspicion. Secondly, we're both random internet strangers. Why believe one person over another based on two easily falsified claims? In this discourse my experience should be as valid as any other.

I would agree with the sentiment of the commenter though. Software bugs in proprietary software are real, but LLMs tend to overstate their criticality, and/or inside knowledge is at times needed to exploit those.

We've sure seen that LLMs can make good assumptions about blackboxes and, unlike humans, are able to try many paths in a short time. However, one can't neglect that doomsday prayers have been present before the dawn of AI, and rarely companies got hacked.

From anecdata, the human is still most often the culprit for breaches, not the "We managed to penetrate that one byte often enough" exploit.

Not only that, their "fixes" tend to break things

Please don't attack others on HN, no matter how right you are or feel you are. https://news.ycombinator.com/newsguidelines.html

> Make sure you’re still thinking.

Please avoid patronizing barbs in comments on HN. I can believe you didn't intend to be mean, but it's easy for these things to be misinterpreted on a text-only message board. https://news.ycombinator.com/newsguidelines.html

You are right of course, for posterity I didn’t intend this as a ‘patronising barb’. It’s more intended as general advice, which readers can agree or disagree with.

Do you have any evidence for the 0.5%.

as a bystander not even working in the sector, your witnessing is exactly what I was worried for. if an agent is wily enough to defeat a security it should also be wily enough to cover its tracks, otherwise what's the point?

[flagged]

Reply on news.ycombinator.com