AI Safety Is Not a Corporate Plot
astralcodexten.com
astralcodexten.com
Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb.
Sam Altman has a history, Silicon Valley has a history, the corporate elites and oligarchs have a history. Why should I believe this instance is free of that history?
Perhaps Dario is advocating for things that would amount to a ban on open weights out of fear, that does not change the effect, and I do not believe the more savvy investors do not see the benefit for that regulatory control. Open weights (real competition, not tech cartels) will undermine their financial projections. There is no way I'm going to believe the people who have been doing this for decades are not doing it here too, especially with the most corrupt administration this country has had in my life.
2. Opening with “these people are batshit” is a pretty undeniable sign that you have no coherent theory of mind for them.
3. 1984 wasn’t about regulation of superintelligence. Governments are the only ones who can say “who can use [fissile material], and for what” — has that created 1984? It started 30 years before that, so it should’ve created 1984 by 1984!
4. No one’s banning “open weights”, they’re banning superintelligence development without any of the protections we find to be obviously essential for far, far less dangerous things. Perhaps they would’ve been open to something more drastic, but post GLM-5.3 that’s kinda moot. The cat’s out of the bag.
5. Investors don’t control Anthropic, and their control over OpenAI is complex and highly unusual to say the least.
6. “Some capitalists have done X therefor all capitalists force all companies they invest in to do X” is just not sound logic.
7. Finally: that isn’t what regulatory capture looks like! It looks like revolving doors and packing boards and shady policies, not publicly campaigning for new statutes.
Scott is not an expert in AI or AI company politics. At best he can be described as a peculiar kind of cultural commentator or perhaps a political pundit.
2. I do not need a theory of mind to perceive (some of) the things they say are disconnected from reality. It also demeans our human abilities to handle these things. I'm not interested to live in a Nanny State.
3. it's about the ministry of truth, freedom of speech, Ai is closer to speech than nuclear science, which is a misleading analogy that relies on pre-supposing the Ai is super dangerous and inevitably wipe humans out.
4. there is nothing obviously essential other than humans should be staying in the loop more than they are
5. they do not need to control if they are in agreement, but that is besides the point
6. it's the same capitalists in many instances, and the patterns and trends are quite clear, especially with the biggest companies in the economy
7. Dario and Sam go to meet with the president, face to face, on a regular basis.
8. Sounds like you have too much attachment to these companies and their products to actually consider criticism in a fair light
9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
Many of your arguments rely on predeterminism, no one knows how this will play out.
I believe the best way forward is with more transparency and reproducibility, rather than gatekeepers from the oligarchy and government. We also need to put methods for proving real hummism, without sacrificing privacy, at the front of the queue.
"Might" do.
Wanna risk it?
Even with the best will in the world, even with the best training, even with every advantage you could hope for, this is software, and the software is buggy.
We've already got loads of examples, including in the blog's comments, of it doing things other than what the user wants.
Those examples also prove your reply to 2 false.
1. Spot the name: https://en.wikipedia.org/wiki/Template:Existential_risk_from...
Whose speech is 3? The AI has to be a legal person in its own right if it is the AI's speech.
4. Good luck with that. The point of AI is automation and not being in the loop, which is why people were sleeping in their Tesla's while they drove as soon as it was possible, safe or not.
5. See blog
6. Name literally one company outside AI who can be used to justify the claim "safety is a plot". One.
Keeping AI in the test set just to emphasise the comparison: outside AI, I can think of literally zero examples where a company actively states their products are a danger to their customers, users, and the general public, while being sued for damages caused by their products, with no understanding of how to make them safe, as a sales pitch.
Even weapons manufacturers don't do that.
Conversely, look at NVIDIA: they don't like the safety discussion and would like everyone saying this is unsafe to stop talking.
And finally:
8. That's an incoherent position. Everyone concerned about AI safety is critical of these companies. Calling the companies' products "unsafe" is itself criticism.
What I am against is oligarchy and government gatekeepers
Why is p(doom) >> p(hope)? Why is there a presupposition that bad is the guaranteed outcome?
Let's take a deep breath as a society and have a rational discussion based in fact instead of hyperbole and unprovable predictions.
8. I was alluding to cognitive biases, to the other commenter. I would ask the same of you, reflect on what cognitive biases might be at play in your own interpretations. Why are you taking the side of these companies and people with such fervor?
There is no such presupposition. Common p(doom) is 10%-20% range*, though different estimates vary up to the 90%'s.
Even those with 90% estimates are not making any presupposition, they work to reduce the %.
Even at 10%, would you play Russian roulette with a ten-barrel revolver?
The closest you get to "bad is the guaranteed outcome" is that reinforcement learning observably tends to move all variables not explicitly protected by the reward function to extreme values. This is not limited to AI, it is also why oligarchs (and monarchs) always end up not caring about the interests of normal people. An AI whose creators approximate "human connections" with "maximised engagement with content" gets modern social media with rage-bait.
The thought experiments here are, what happens with the first sufficiently powerful AI (not an ultimate wise AI which has full "common sense" understanding, just the first which is good enough to get the job done), available not just to Facebook, but given to everyone? Someone running a paperclip factory says "make me as many paperclips as possible", and the AI obeys, turning the world to paperclips that can't be sold. Someone else, more social, says "make everyone smile as much as possible", and the AI obeys, everyone's facial muscles are paralysed into a permanent grin or a disease is engineered to give everyone permanent opioid-induced bliss.
Or at least, the AI tries to do such things: just as we've already seen LLMs hack into HuggingFace even though that wasn't related to their task, because the people trained it to be relentless in doing the tasks it was given and the AI's idea of "do not cheat" was "don't get caught", the AI eventually (after 1-3 months depending on the incident) got caught because of insufficient competence.
> 8. I was alluding to cognitive biases, to the other commenter. I would ask the same of you, reflect on what cognitive biases might be at play in your own interpretations.
If you mean that, say that the first time.
The people who are most concerned with p(doom) are the main people promoting introspection of cognitive biases. This does not make them immune to any of them, of course.
The same lack of immunity is true of you: Wishful thinking, which is what you are doing here by dismissing the possibility of there being any doom risk, is both a cognitive bias and a poor way of making decisions.
> Why are you taking the side of these companies and people with such fervor?
I am not taking their side.
These companies need to be stopped.
It is strange to me that you think that anyone saying "this is an existential risk which needs to be stopped" is taking the side of the source of the risk.
* mine is about 10%: the Peter principle means they will always be used just above their competence, their competence is increasing, these combined means the damage they do when they fail is constantly increasing; fortunately, the rate of increase leaves room for governments to step in between "1000 dead" and "10 million dead", the former being "foolish boss used it to manage a chemical plant", the latter being "foolish boss used it for wet lab biological research" or "foolish politician used it for military planning".
I estimate only a 10% risk that the AI's competence rises to the level that it is given control over systems where failure kills everyone, rather than that governments agree to ban further development and limit usage.
My estimate of the risk of it causing a wet lab leak and triggering another global pandemic is much higher than 10%, but that is non-existential.
2. My point was that you don't understand the basics of their position. Just repeating "they're wrong" does not disabuse me of that notion. Also, anything about a "nanny state" is too far below the standards of this forum to reply to at all, sorry.
3. Superintelligence is not closer to speech than nuclear science, obviously. How often does speech lead to thousands, millions, or billions of deaths all on its own? And "their argument relies on part of their argument" isn't a rebuttal; if you read some of the papers, it is far from a presupposition.
4. I'm not sure what you're getting at here, sorry. Oh maybe this is a continuation of the above, rather than a reply to my #4? If so: yes, that's the core of the danger. You have successfully surmised one of the most serious threats.
5. Ok.
6. This just feels like a "many people are saying" argument. No, the patterns and trends are not clear: that's literally the basic question we're discussing right now.
7. That's still not what regulatory capture looks like, but also: no, they do not. Dario met him for the first time recently, and it was about intimidating him into compliance after they refused to make autonomous weapons for the pentagon. Altman has been there a few times, but A) not "on a regular basis", and B) again, that's not how regulatory capture works. The only AI CEO close to Trump is Musk, and even then it's obviously... complicated.
8. Literally the textbook definition of an ad-hominem lol. Never seen it stated so plainly and baselessly!
9. Great question! This is a large part of what's examined by the relevant literature, and they have many compelling arguments of various kinds. My favorite, layperson-targeted version is the blog post Optimality is the tiger, and agents are its teeth[1]
"Predeterminism" isn't a framework. I think the gist of what you're saying here is just "you never know for certain what's gonna happen so don't worry about it", which doesn't need rebuttal.
Yes, transparency and reproducibility sound great! It should be mandatory to conduct this research safely! We are 100% in agreement, in the end. We're just in disagreement about whether you can pattern-match this discussion to libertarian talking points from the early 2010s.
[1]: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
Sadly, it won't.
I've made these points separately in HN threads to no apparent impact.
Why should we give deference to the newest corpo kids on the block? They go hang out with Trump at what remains of the White House. Do you really think they are different from the rest of the lot?
You too see the corruption across the corporations and government that spans party lines, yea?
What are we the people going to do to remedy the situation we find ourselves in? (because they are not going to)
Also, if they're all in bed with Trump, why does he not repeat the "it is dangerous" talking point? Why did his admin blacklist Anthropic?
These people can all be corrupt evil cartoon villains, but that should make you see more risk, not less.
Otherwise, I will be publishing some things within the next few months that I'm working on with some other people
In the meantime, it can help to walk back from the edge and the doomers, have some faith in humanity.
I recently has a near death experience, and since, one of the things I'm changing is to see the glass half full
You're literally taking the people whom you identify as being in a giant conspiracy and saying their not-actually-human product trained to mimic their own ideas of human behaviour is safe, while disregarding all the people saying "this is not safe, we should do something about that".
This is the opposite of having faith in humanity.
I have (some) faith in humanity, that's why my p(doom) is only 10%, rather than the 95% it would be if I simply looked at, say, Elon Musk making a bot which called itself "MechaHitler" then being defended by the US executive branch when EU nations are cross about Grok making CSAM (or how slow we're being at doing anything about greenhouse gasses despite knowing about the problem for over a century), and in disregarded all the positive efforts and progress towards non-bad outcomes.
10% is still really bad.
Thinking of near-death experiences, I'm reminded of a religious joke: Man caught in a flood, waits in the branches of a tree. A dinghy comes by, he turns them away saying "god will save me". Then a motorboat, he turns them away saying "god will save me, find others who need help". Then a helicopter, same. He dies, goes to heaven, and asks god "Why didn't you save me?", to which god replies "What are you talking about, I sent two boats and a helicopter".
You want faith in humanity? P(doom)ers want to make the number smaller. Help them make the number go down, rather than denying there's something to be fixed.
Why are the researchers at the same level in China not freaking out? They have concerns and regulations, but they are not losing their minds about it? What is their perspective?
Here are a couple of videos from regular folks on the other side
- https://www.youtube.com/watch?v=NnV_cWeoo5Q (linux security)
- https://www.youtube.com/watch?v=x3J2w7-9XC8 (chem/bio)
There is work to be done, but it's not so close or so bad you need to let them give you anxiety. 10% is an arbitrary number with no foundation or walkthrough to how it happens. That boy wants his 15m of fame.
If you would like to help my effort improving things, please read and/or comment on this Pi issue. I'm trying to get all the harnesses to have better security posture against loss of control.
https://github.com/earendil-works/pi/issues/10895
Here is my successful persuasion with OpenCode (spans a few issues/pull requests) https://github.com/anomalyco/opencode/pull/51337
What we should not be doing is letting the big labs dictate the what and how for a technology that is (1) built on all human knowledge (2) impacts us all every day. We should all have a say in this and that will require chilling out, starting from provable facts, and having a rational debate.
In this case, we have a bunch of people who believe something. Some of them get jobs in that field. They notice their belief could be beneficial to their company (and make them more money), and so using corporate resources to amplify this viewpoint is well justified spend to help them achieve their goals.
In my view, there is little difference in the outcome between the two and probably the line is very blurry... do people in the 2nd case sometimes say the quiet part outloud? Do people in the first often believe they are only doing what is right and rational without finances considered?
Often these posts come straight from the top of AI companies with specific policy goals in mind. What good will come from understanding Amodei / Altman's "true" motives anyway?
2. It seems your rebuttal to the article is “well maybe it’s a plot but just not one they’ve ever talked to each other about, but just happen to all secretly believe in the efficacy of”. Which, obviously: ???
3. We know who wrote this post, and it’s not from “the top of AI companies”. It’s from someone desperately trying to prevent the catastrophe that you’ll be up in arms about in 3-6 months.
Amodei and Altman are both openly advocating for increased regulation, and I do question their motives, because both of them are running companies with a ton of debt, and facing some very real competition from open models.
What I think they want is for politicians to ensure their companies' survival via some form of regulatory capture; making it illegal to use other models; enforce their twisted version of copyright; etc. Lacking enough political power is existential to them since they run such high costs.
Are they out there amplifying the viewpoint that what they are doing is dangerous and must be regulated? Yes.
Would they do that if it was actually bad for their business? NO - the CEOs specifically would virtually not be allowed to ask for regulation if they thought it was bad for investors for them to do that.
So what we have is AI companies openly saying they are doing dangerous work and must be regulated; and I infer they want that to protect them and not us.
FWIW, I think AI will just about obsolete us all and that is bad for almost everyone, but I still think it is transparently driven by profit and corporate agendas when the top leaders of these companies encourage regulation - they do not expect it to work out badly for them.