1. They, and many other experts, have been saying the same thing since well before the current paradigm existed. 2. There is nothing about regulation that is required to give them an advantage; it’s a fallacy to suggest “regulation = advantage” 3. It would be preferable to argue the facts (is it or is it not dangerous? Why or why not?) than conspire as to why they’re stating what they and plenty of others have been saying for decades
1. All those who have been arguing this - many of them non-technical - are part of the same ideological movement.
2. Technically, there isn’t. In practice, majority of the time regulations benefitted those who wrote them.
3. Not sure if this is a criticism of the comment you are replying to or of the p-doom crowd. We would all appreciate facts stated instead of science fiction tropes repeated.
Look, “a lot of people say this” is a Trumpian fallacy. A lot of people say a lot of crazy things. It says nothing about the factuality about what they say. How many are this lot? Who are they? What do they know? Why are they incapable of expressing it without so much emotion? How come a big portion are non-technical?
In fact, their core ideology actually stemmed from Yudkowsky's arguments.
> 1. They, and many other experts, have been saying the same thing since well before the current paradigm existed.
Some experts sure. "Some experts" say lots of things.
However, this is actually much to their advantage. If you were a frontier AI company, a legitimate concern to use as justification for your regulatory capture is exactly what you want. That is, after all, generally how regulatory capture gains public approval.
AI safety was also why OpenAI had to charge per-token for GPT-3. It must've been such a struggle for them to make that difficult decision. I'm sure Sam Altman spent a lot of nights crying to sleep over that one.
It's good that we never got open weights for a GPT-3 class model, right?
> 2. There is nothing about regulation that is required to give them an advantage; it’s a fallacy to suggest “regulation = advantage”
It's difficult to outline the number of different ways that regulatory capture can and will benefit OpenAI and Anthropic.
If we limit things to "Fable"-class models, then nobody competing with Anthropic/OpenAI is ever allowed to outdo them. If you literally legally aren't allowed to make better models, you can never charge the "premium" of having the current top-tier model. You have to compete by being cheaper than Anthropic instead, which is a lot less enticing from a standpoint of trying to become profitable. Take a look at the per-token pricing of Fable real quick. Yeah. Being able to charge a premium for the model that does the cybersecurity thing is literally their most important asset.
If Anthropic really is making a profit now as rumored, then they need regulatory capture as fast as they can possibly get it. Once Anthropic and OpenAI prove that AI is a viable investment, entities with a lot of capital will suddenly become even more interested in re-allocating it towards taking a slice of this pie.
> 3. It would be preferable to argue the facts (is it or is it not dangerous? Why or why not?) than conspire as to why they’re stating what they and plenty of others have been saying for decades
OK. On the factual end of things, AI safety experts have been raising the alarms for over a decade and so far essentially none of their predictions have come true.
The rationalists, the main group pushing the AI safety rhetoric that dominates the Western AI sphere and is virtually absent in China, which appear to be going entirely off the deep end at this point in time, put together their best minds to produce AI 2027, which of course like most predictions started to fall apart a bit before hitting the one year mark. Predicting the future is hard.
I have made this point before but I will make it again: I think AI safety experts deserve at least as much skepticism as Ed Zitron in their claims. They've been making similar dubious claims about AI capabilities and an incoming singularity for a long time now, and so far the best we can say is that maybe it will eventually come true.
Maybe. Or maybe you've read too much goddamn science fiction.
They seriously underestimate how ready and willing Americans are to use Chinese providers. There’s a ton of spite
Last time China pulled this shit it shook portfolios for a week
I get this is Hacker News so we have to hate AI safety, but you should read more on this topic rather than blithely casting the x-risk viewpoint as fringe. The last hard data we have on the question is a late-2024 AI Impacts survey [1] of 1,580 authors at top AI conferences, who were offered $50 for answering a survey on the topic of the future of AI progress. The median respondent assigned a 5–10% chance (!) to "extremely bad (e.g. human extinction)" outcomes.
> On the factual end of things, AI safety experts have been raising the alarms for over a decade and so far essentially none of their predictions have come true.
Devastating. When I wrote my AI alignment PhD thesis [2] in 2022, and predicted a 70% chance existential risk by 2072 (which I later came down to about 1/3 on), you're right that that prediction hasn't come true yet.
While I agree many specific worries didn't materialize (like AI being like a "literal genie" it's hard to specify goals to), many key worries have (Hugging Face rogue AI swarm scans as cribbed from rationalist AI scenario writing).
[1] https://aiimpacts.org/wp-content/uploads/2026/09/ESPAI2024.p...
[2] https://arxiv.org/abs/2206.11831, top of page 4
You're saying regulatory capture gives you no benefit by just refuting an absolute with 0 context. There's nothing to even debate here. Ok... most people disagree?
Yeah I agree with point 3... Stop engaging in vageries and say something of substance with examples, so people can actually have a conversation with you without us doing all the legwork.
OpenAI and Anthropic aren’t even slightly worried about DeepSeek.
May this kind of worry find me…
It's quite hard to have the valuation of Anthropic if they are limited to US.
The model makers are bricking the models with their system prompts, and even the tools are tamper proof.
You can only use it their way. They think we’re stupid.
And their way is the shit way, the babyproofed way, while they talk to you about their unfucked models hacking out of their labs and hacking into other labs, and reminding you you have zero rights.
Fuck them twice, then once more… This is about compute. Who owns compute. China is saying fuck you all we can scale things bigger than you can imagine. And they can! They have so much more energy! Theyre closer to commercial fusion than us too. Meanwhile our guys are trying to… regulate? Lol
Training and serving LLM's is massively parallel. Your bottleneck is either in energy or chips. China will catch up on chips fully decades before we catch up on energy. Arguably, this is already happening: here in the USA, we have lots of chips, but we are seeing our electricity bills skyrocket as more datacenters come online. We are powering those datacenters with natural gas turbines, not solar, wind and nuclear. Right now, to purchase one of those, you are on a 5-year waitlist. There's your timeline!
China is invested, from a regulatory, cultural, workforce, strategic, and technological standpoint, in an energy abundant future. We are not invested -- and in fact, actively sabotaging ourselves -- on all of those fronts.
Once they acquire chips, we lose, basically permanently. Look at what Huawei is doing recently and what is on their roadmap to understand how fast this catching-up is taking place. I'm sure that David Sacks would say that our innovation motor is pulling their copycat caboose along, as our monopolist tech companies steam ahead towards the frontier. I am not so sure. They are already ahead of us in many technologies. Speaking of technologies, they're building natural gas turbines in house, starting in 2025.
> Literally zero, but that never stopped a good conspiracy theory
Are you living under a rock and never read any of Dario's blog posts?
Let me give you an example, here: https://darioamodei.com/post/policy-on-the-ai-exponential
Section "1. *Regulation* and public safety" (emphasis mine), then under it:
> AI companies that develop advanced AI models must have strong security standards that protect their model weights
You can't "protect model weights" if the weights are openly available, ergo this is an explicit call for regulation to make open weights illegal.
Presumably you'd want the AI companies to protect their weights to keep them from falling into the hands of bad actors. But the US can't make open weights illegal anywhere except the US, which does very little to keep them out of the hands of bad actors (most of whom are outside of the US).
He's completely transparently campaigning for regulation. It must be very frustrating to him that the Trump admin is completely non-responsive. He probably thought Anthropic would have the whole market stitched up and water tight by now.
1. He's made many posts about how he thinks "dangerous" models are bad (https://www.anthropic.com/news/position-open-weights-models)
2. He writes posts proving Chinese open weight models are "dangerous" (https://insightsintegration.com/anthropic-warns-of-advanced-...)
3. He talks about about how he wants stronger govt regulation of AI (https://www-cdn.abcnews.com/Business/exclusive-anthropic-ceo...), and explicitly mentions he does not trust China at all and doesn't want them to have Mythos.
Do the math: 1 + 2 + 3 = he wants to regulate/ban the Chinese open weights.
Of course, Anthropic and OpenAI will then write posts about how their own models are dangerous - but that's okay, they're the good guys.
There are no choices here which do not carry risk. I haven't seen a single convincing argument against these points.
We also put responsibility on humans to act responsibly. In theory, it's absolutely insane to allow human beings the ability to walk into a street with automobiles, because obviously they will be killed! But we don't ban them from walking there, and we don't ban cars, because it's pretty uncommon for people to walk directly in front of a moving car, despite there being obvious danger.
Ultimately the biggest risk of AI isn't the AI. It's what human beings do using the AI. Just like with hammers, knives, drills, chemicals, cars... we regulate the human's actions and create laws to direct their use, rather than regulate the items themselves.
When HuggingFace was in the midst of defending itself from OpenAI's automated hacking spree, they were blocked from using frontier models by Anthropic and OpenAI for "safety" reasons. HF had to resort to an open model to defend itself - and DeepSeek distinguished itself in that role: the second fact from that episode OpenAI would have you soon forget.