The doomers already have a term which fits pretty well: "AI misalignment".
3,098 karma · joined November 17, 2021
Some HN users like to "flag" comments which are compliant with the HN rules, on the basis that the comment in question goes against their ideology. You can turn on "showdead" in your profile to see flagged comments. (I recommend doing this.) If a rule-abiding comment has been flagged, you can click the comment permalink to "vouch" for the comment and vote against the flag. For reference, the comment rules are here: https://news.ycombinator.com/newsguidelines.html
You can try to contact me if you want, by emailing my username on protonmail, but I probably won't see it. Sorry.
The doomers already have a term which fits pretty well: "AI misalignment".
The HuggingFace attack was not "random" behavior. It was goal-directed but misaligned behavior.
This isn't necessarily a simple matter of the operator making sure they behave. AI alignment has been considered to be a difficult problem for over a decade -- and remains unsolved in general, as these recent incidents illustrate.
"Fuzzer" already has an existing meaning in CS anyway: https://en.wikipedia.org/wiki/Fuzzing
* * *
Many experts believe that:
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
https://aistatement.com/work/statement-on-ai-extinction-risk
If the antitrust waiver is too broad, we can always rework it. But there's no "undo" button for human extinction.
"According to survey results released on Wednesday, 68% of [likely US voters] said they would support the proposal to temporarily pause advanced AI development and permanently prohibit the development of superintelligent programs, while just 25% said they’d oppose it."
https://www.commondreams.org/news/sanders-casar-ai-bill-poll
>Eg in a hypothetical armed convenience store theft: We don't prosecute the gun manufacturer, we prosecute the individual using the weapon for crime. Same deal here.
So if I ask ChatGPT to make me some paperclips, and it replaces all the matter in a nearby city with paperclips, who gets prosecuted: me, or OpenAI?
There's no simple bugfix which will address AI misalignment. It's essentially been an open research problem for upwards of a decade.
It's already going up at staggering rate. Anthropic is now at $100B in annualized revenue, up 50% in the past two months.
* * *
Here's a little allegory for how I'm thinking about this discourse.
Imagine a man who is raising tiger cubs in his backyard. They're growing fast. He keeps them on dog leashes so they stay under control. One day, a growing cub breaks its leash and goes on a rampage through the neighborhood, eating a beloved local pet.
The neighborhood erupts into a big argument: Was the leash inappropriately thin? The neighbors point out that thicker leashes are easily available at the local pet store. Furthermore, is it appropriate to refer to the loose cub as a "wild animal" in local news reporting, or is it factually more accurate to call it "domesticated"?
Meanwhile, the cubs grow larger and lick their lips, oblivious to the discussion.
https://darioamodei.com/post/we-must-pace-the-frontier
I'm not seeing how that solves his 3rd-party competition problem.
AI: "OK, I've now converted the entire planet into paperclips."
Alien observer #1: "Wow, that was a rogue AI!"
Alien observer #2: "False. We need to place the blame where it belongs, on the person who requested the paperclips."
Ultimately this type of terminology dispute has a tendency to miss the point.
"With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said."
So a third party apparently confirmed that a hack was attempted.
The NYT also writes:
"No A.I. company has been involved with as many disclosures of rogue incidents as OpenAI."
I think there is a certain amount of mental gymnastics needed to believe that they are establishing themselves as the industry leader in rogue incidents, as a strategy to gain an antitrust edge.
The public is paying close attention to the AI industry. That's the regime in which regulatory capture and similar strategies would be expected to fail: https://marginalrevolution.com/marginalrevolution/2026/09/wh...
Sam and Dario have been doomers, or doomer-adjacent, for something like a decade at this point.
Occam's Razor is simply that they believe what they are saying about AI doom.
https://aistatement.com/work/statement-on-ai-extinction-risk
Animals are incapable of comprehending much of what humans do. Nonetheless, we humans have vastly transformed their world and caused the extinction of many species.
There seems to be a general trend of "US inverse exceptionalism", i.e. arguing that the US is a uniquely bad country on the basis of very weak evidence. People complain a lot about systems in the US, but if you look at international comparisons, the US doesn't usually look too bad. Here in this thread people are up in arms because... 1/3 of Texas prisons have A/C, and the remainder have non-AC heat control measures. I doubt this is substantially different from non-US prisons.
That's sort of the classic paperclip-maximizer AI doom scenario. The misaligned AI merely "does what it is told": make paperclips.
"A spill or tumble can be quite embarrassing if there are witnesses.
How to reduce the humiliation? Turn it into a stunt. Claim it was intentional, a show for their benefit."
https://tvtropes.org/pmwiki/pmwiki.php/Main/IMeantToDoThat
>They were instructed by humans to do exactly that.
This is more or less what the doomers have worried about for decades.
>You cry "Get my mother out of the [burning] building!" [...] and press Enter.
>For a moment it seems like nothing happens. You look around, waiting for the fire truck to pull up, and rescuers to arrive - or even just a strong, fast runner to haul your mother out of the building -
>BOOM! With a thundering roar, the gas main under the building explodes. As the structure comes apart, in what seems like slow motion, you glimpse your mother's shattered body being hurled high into the air, traveling fast, rapidly increasing its distance from the former center of the building.
https://www.lesswrong.com/s/3HyeNiEpvbQQaqeoH/p/4ARaTpNX62ua...
https://www.businessinsider.com/nvidia-jensen-huang-ai-doome...
Many people are calling for an AI pause or liability/punishment for bad actors like OpenAI. You have to do serious mental gymnastics to convince yourself that Sam Altman will benefit financially from the new regulatory bloodlust. No surprise that OpenAI hasn't exactly been forthcoming about info related to these hacks.
Furthermore, HuggingFace required an open-weight model to respond to the hack. That certainly blows a hole in the "carefully designed" claim from zx8080, if nothing else. It looks terrible for OpenAI, and decreases the probability of some sort of regulatory restriction on open models.
Also, these measures kinda look similar to what exists in Texas prisons without A/C...
>In fact, many more people are dying by heat exhaustion in Texas prisons per capita than in Europe.
Based on a quick Google, Europe's population is on the order of 1000x the population of the Texas prison system. Deaths are probably at least somewhat higher per capita in the Texas prison system, but there are also probably subgroups in Europe that have it even worse.
Speak for yourself.
There probably is a lot of room for genuinely valuable tech-adjacent humanities work -- something like the Community Notes algorithm on X could be a good example. But I have little faith in humanities grads to do it, in part because they don't understand tech at a systems level and don't want to learn.
These are just my vague impressions -- take with a grain of salt.