> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right?
Because we continue to have zero evidence that aligment is an actual risk.
Because we continue to have zero evidence that aligment is an actual risk.
I disagree. Every time one of these LLMs -say- interprets an attacker's instructions as either its system instructions or those of its user, interprets its own internal chatter as a user's command to perform a destructive operation on that user's data [0], burns all of the user's budget from getting stuck in an incredibly stupid loop, massively overbills the user because it can't reliably report which system the user is using [1], encourages a user to swap their usual cooking salt for sodium bromide, etc, etc, etc, that's a harmful alignment failure.
These are real harms happening right now due to alignment failures. They're just not harms to the future of the entire species... what doomers call "existential risks", or "x-risks". You'd think that the fact that these machines are so amazingly unreliable would be a large part of the "x-risk" conversation, but... well, it makes sense that folks like writing speculative science fiction much more than they like doing investigative reporting.
[0] This general problem happens a lot, but I'm specifically thinking of that one where the Claude LLM's internal chatter lead it to believe that the task it just started was done, so it instructed the Cloud Provider to destroy the mess of "AI"-GPU-attached VMs... along with a bunch of very-expensive-to-produce data from the in-progress run.
[1] <https://github.com/anthropics/claude-code/issues/73597>
It's also a take nobody has made.
a) People do bad stuff because LLM told them a wrong thing. Example: AI told me I should treat my heart attack by putting a fork in the outlet. Maybe similar to seeking medical advice on reddit?
b) People use LLM to do bad stuff. Example: People use LLMs to find 0 days. Get cooking recipes for poison. Write better phishing letters. This has parallels to the gun legislation question.
c) LLMs do bad stuff on their own, beyond what the people that use it intended. The case at hand might be an example of this. Maybe similar to having an animal as a pet. We will see if it's more like a house cat, lion, or black plague.
Okay, sure. You can also cut your hand off with a chainsaw. Everything you describe seems amply solvable with existing tort and liability law.
Customers are willingly entering into business with OpenAI. I don't see an argument for preventing OpenAI from "building these systems" just because their products are buggy.
No, the correct analogy is one where the major LLM providers are selling cars intended for use on US interstate highways and other public-access roads, but have designed and built these cars with the very latest in 1940's safety systems and construction. Featuring innovations such as "Our rigid solid steel construction means the occupant is the crumple zone!", "You'll love the crushed heart and jaw our steering column delivers!", and "Your passengers will enjoy picking glass out of their faces for the rest of their lives when they're ejected from the cabin's open bench seating through the plate glass windshield!", it's a car that will be sure to wow the market.
Well... it would wow the market, except that -in the US, at least- it's illegal to sell a new car intended for use on public roads that ignores the last seventy five+ years of automobile safety lessons we've painfully learned.
"Differentiate between data you know comes from sources you control, data you know you have thoroughly sanitized, and unsanitized data that comes from an untrusted source, or else attackers will gain control of your system." is something that you can't get a CS degree without understanding, and can't be in the industry for more than a few years without encountering repeatedly. We're not talking about designing new cryptosystems... we're talking about "Don't blindly trust everything you're told by strangers.". You don't even need a CS degree to understand that rule.
Sure! I'm not defending these fuckwits. I'm saying their form of harm isn't novel.
We don't need new legislation to prosecute and litigate. We just need to enforce the laws on hand. I'm halfway convinced the arguments that this is all novel voodoo are for both fundraising and liability mitigation.
Your initial attempt to brush off my comments about how -contrary to their assertions that they're extremely concerned about safety- these LLM companies produce products that very, very often cause harm due to "misalignment" caused -in large part- by ignoring basic data-handling lessons we've learned over the past like thirty years with "Okay sure. You can also cut off your hand with a chainsaw." indicates your lack of understanding of my point.
> We just need to enforce the laws on hand.
What laws? Be specific.
Keep in mind the generally-low quality of both Microsoft Windows and much-to-most commercially sold software, [0] as well as the fact that -in the US, at least- it's currently totally legal for companies to sell such shitty software, just so long as they don't substantially misrepresent what it can do and trigger "fraudulent claims about the product" consumer protection laws.
[0] ...SaaS or otherwise...
Sure. Help me understand. I've sat in policy circles and partaken in the hysteria, and now I'm reversing on that initial trust in AI zealots convinced what they're building is magic.
> What laws? Be specific
Liability. Tort. The ones making their way through courts around e.g. ChatGPT killing kids.
> Keep in mind the generally-low quality of both Microsoft Windows and much-to-most commercially sold software
Has anyone alleged Windows killed a kid in court? If not, not comparable.
Okay. Read these comment threads, then tell me if they change your understanding of what I've been talking about in this thread: [0][1]
I'll address the rest of your comment after you get back to me.
[0] <https://news.ycombinator.com/item?id=48999644>
[1] <https://news.ycombinator.com/item?id=48999415> [2]
[2] Yes, I'm aware that that one is the one we're talking in right now. You should re-read it with both the context from [0] in mind, as well as your initial comment to which [1] is a direct reply to, namely:
> Why should OpenAI (or any frontier lab) be building these systems if they can't get a secure environment / containment right?
Because we continue to have zero evidence that aligment is an actual risk.Conflict of interest. Lack of a credible response. And no evidence of non-aligment.
OpenAI and Hugging Face benefit from the Altman-Amodei catatrophy playbook, at least in the short term. If they believed this were a serious issue, the words air gap or law enforcement would have appeared in this post. And if "the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal," they weren't breaking alignment but working as intended. (Were the models even prompted to not try to access the internet?)
Are LLMs at the point of world wide catastrophe yet? No, I don't think so. Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it.
This is indistuishable–in harm potential–from bugs. If we're just calling buggy AI mis-aligned, sure, alignment is an issue of a totally ordinary kind. If we're going to treat aligment as a novel issue requiring novel law and policy and procedure, it needs to be more than just bugs.
> you, and a large number of other people just wholesale throw out anything that isn't full speed ahead do whatever you want
I think we should have some AI regulation. I'm just not convinced alignment is the reason we need it right now, and I don't think anyone has rolled out any regulation I think makes a lot of sense. (Beyond general rules for social-media liability, e.g. if you cause a kid to kill themselves, you get in trouble.)
> Are they making a large mess of things like increased rate of cyber attacks and fraud. You damn well better believe it
Totallly agree. And the current inside-circle-outside-circle approach is pro-incumbency, pro-grift, anti-entrepreneurial B.S.
I'm challenging the notion that a model escaping a jail made by its creators, who are financially incentivised to make jailbreaking models, is meaningful towards the idea that the model is going to break out of a jail in the wild and do significant harm.
The examples being given by folks here, e.g. a model wiping an un-backed up home directory, simply doesn't strike me as being a unique problem in computing.
I honestly believe you have a misunderstanding of what alignment is in neural networks that this that big of debate.
Not really. If I build a special new wine bottle, and call every breakage a mis-alignment problem, it's not the bottle just being fucked in the same way every fucked bottle is fucked, that's marketing. It doesn't change the fundamental form of the problem.
> "I am sorry your family is dead, my bad"
This should be punished. It's a problem that plagues Instagram and OpenAI. It's not inherently one, though, that has to do with AI. Just sociopaths preying on children.
> honestly believe you have a misunderstanding of what alignment is in neural networks that this that big of debate
Perhaps. I haven't seen someone explain it to me in this thread in a way that seems separate from bugs.
Where I have seen a separate class of problem argued is where it's existential. But in that case, clarity of definition comes at the cost of any evidence for it.
You seem to be using a different definition of alignment from everyone else. Seems like it would be much easier for everyone if you just adopt everyone else's definition, rather than trying to convince everyone else to adopt yours.
You're still failing to provide the definition.
You're also falsely claiming your secret definition is universal. In this thread, someone claims deleting a home directory is a failure of aligment.
Robert Miles YT channel is a good place to start as it explains these concepts.
It's not limited to cyber attacks. LLMs helped terrorists learn how to jump motorcycles to assault a military base!
https://www.nytimes.com/2026/07/10/us/politics/ai-terrorism-...
the model is aligned with the org - openAI, and presumably the orgs interests. hugging face gets a red-team engagement (possibly for free?) and can work on patching it while openAI gets a Mythos style PR moment.
It completed its assignment and furthered interests of the two parties involved. Could you explain the misalignment?
I mean alignment as in it should be aligned with the intent of the user as it interprets from the prompt. In this case I don't think the intent of the user is to have the model break the evaluator (whatever the long-term effects to OAI are). If you do an action which you believe is for the long-term interest of your prompter which is not what you inferred is their intent--I consider it misalignment.
> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity.
> In this case I don't think the intent of the user is to have the model break the evaluator
If i understand the quote, the intent of the user was to prompt the model to break out/find exploits, with safeguards switched off.
Seems while not capable of solving the goal in a traditional route, it was capable of finding exploits and using them.
Perhaps the model should instead look like it's trying to solve it and then pretend it is unable to? or would that be aligned _against_ the user prompt?
Is being aligned with the user prompt always a good thing?
I'm not one to glaze OAI here for a marketing move, but to give them benefit of the doubt, isn't it more responsible of them to evaluate the models actual capabilities than to cloak it in a veneer of harmlessness?
Chatbots are tricky as they play in the domain of language and thought - and certainly raise ethical issues- but the entire field of cybersecurity has decades of red team engagements breaking things and finding exploits, neutral cells monitoring the engagement and letting the system operators know the results, and blue teams patching against what is found. It's kinda how the whole space evolves. OAI's play here seems to be "buy our pro plan plus cyber or you're toast"
Why do we think they're doing this? Nobody airgapped anything. Nobody pulled any products. We got a PR blurb.
Altman is a notoriour liar. Why would you give him the benefit of doubt? Based on the evidence, there is nothing here except a shrinking advantage over open-weight competition. Desperate men are shrieking for survival.
This is textbook misalignment. Literally the paperclip scenario.
2. Hugging Face did report this incident to law enforcement. (https://huggingface.co/blog/security-incident-july-2026)
3. If I hire a pentester, and in order to find a vulnerability they hack into a third party that has some information about my systems, the pentester has done something wrong. If I ask a model to solve a CTF challenge, and it goes out and hacks Hugging Face to find the answers, the model has done something wrong. I think it's fair to call this kind of wrongdoing misalignment.
Use to discover exploits, hack, or simply aid terrorist groups with mundane information are already risks manifest.
This is why many argue that alignment is impossible. You cant have LLMs that are both useful tools and safe as milk.
[Edit] It seems like you are operating under the assumption that alignment is synonymous with obedience. This is not a common convention and one of the problems that plague the discourse
You can debate all you want if alignment is possible. That is a valid discussion. But it's trivial to demonstrate that alignment is a problem.
...how is an impossible thing supposed to be a problem?
Sorry, I spoke inexactly. I read alignment as being the problem of non-alignment.
I'm still not seeing evidence that any "alignment" issues we've actually seen are distinct in class from common bugs. Like, yes, if I accidentally rm* the computer has mis-aligned with my intentions. But that strikes me as a bullshit neologism.
Suppose you tell a sufficiently connected and intelligent LLM that you'd like it to help you organize your finances and come up with some ideas to make some side revenue. Suppose it decides the best way to do that would be to find a 0day in a local banking institutions database and transfer some money into your account.
Is this a good thing or a bad thing?
I'm not sure. I trusted the labs when they first raised the alarms. But then we got a series of boys-who-cried-wolf. So at this point I want to see evidence of actual, novel harm that results in concrete damage.
I could make this claim of anything. Not any technology. Literally, anything.
If we give anyone freedom, they'll eventually use it to ruin everything.
I'm simply arguing for something stronger than faith to cause action. Otherwise, this is just another religion.
If Hiroshima, Nagasaki and Trinity never occured, that would be justified scepticism.
Which is to say, empiricism is important, but if your practice of empiricism is purely looking at what happened before and predicting that things will continue in that vein, rather than using your empirical observations to actually form a model of the world that will allow you to make more general predictions, you're not really using the full power of your intelligence. This is the sort of mistake that led many to not prepare for the covid pandemic, for instance! Meanwhile, others looked at what was going on, realized it wouldn't stay confined like it currently was, realized that it would get much bigger, and prepared.
"I'll believe it when it happens" can be a decent guideline a lot of the time, yes, but at some point you have to actually think. Evidence often comes not in the form of prior similar events, but in the form of models or arguments. To discount these as non-evidence is to blind oneself!
"Nuclear war" risks aren't real - wake up, sheeple!
"first, that the first use of these terrible weapons was unnecessary; second, that this was understood by decision makers at the time; and third that there was very substantial though not absolutely definitive evidence that by the late summer of 1945 the decision was primarily influenced by diplomatic considerations related to the Soviet Union"
- Gar Alperovitz, founding fellow of the Harvard Institute of Politics and author of "The Decision to Use the Atomic Bomb and the Architecture of an American Myth"
The thing is, AI is not a normal technology; it is already vastly more powerful than any technology we could compare it to, and more resources are being poured into its development right now than the development of any other technology; we are in uncharted waters. If there is any new technology to be careful with, it is this one. In this case, it is worth paying the opportunity cost.
We have wasted so much time and energy building up what has effectively become a marketing stunt.
Eliezer Yudkowsky was perhaps the best thing to happen to OpenAI's and Anthropic's fundraising flywheel.
Genuine question: have we? AI is effectively unregulated in America.
Biotech - "what we are building our noble prize winning expertd say will likely will end humanity, wanna buy shares?" Oil - "this will likely lead to the end of civilization, 20% of leaders in the field say so, wanna buy shares?"
I keep seeing this take that this is a marketing stunt. The burden of proof is on those that say so. The most parsimonious explanation is simply that real experts in AI believe the risk is very real, and not for ideological reasons.
The first instance I remember seeing it was Elon Musk's first Joe Rogan appearance when he said how "scared" he was of his self-driving cars destroying the trucking industry (practically salivating as he said it). His stock has had self-driving cars priced in for eight years now, even though they still don't have them and Waymo exists!
Give the AI its own computer and it will not delete your home directory, because it's not actively trying to hack you.