I quit OpenAI because its culture is broken | Hacker News Reader

I quit OpenAI because its culture is broken

theatlantic.com

497 pointsBrajeshwar818 comments

https://www.theguardian.com/technology/2026/oct/03/openai-sa...

OpenAI and other frontier labs won't ever introduce safety-level standards like those used for railways or nuclear plants until they are forced to do so by customers or by law. The reason is simple: safety is expensive, and if safety is introduced properly, development is no longer mainly about how to implement feature A. Instead, it becomes much more about how to design two or more redundant systems to implement feature A safely, while also documenting everything clearly and having it audited by an independent auditor.

So the focus completely shifts from spending 90% of the effort on the functionality of feature A to spending 99% of the effort figuring out how to safely implement even a lightweight feature A.

Actually you are wrong. They are only waiting for them to get to the point where they can't push LLMs any further via training and then lobby Governments to introduce safety regulations that they'll adhere to, thereby using safety itself as a moat to block competition (including open weights from other countries, notably China). That's exactly why they are using China as the bogeyman and you keep hearing hysteria about AI labs not pursuing "safety". Safety is going to be the moat that will lock AI into these Big 4 AI labs.

Who are the Big 4 AI Labs?

To me there are the clear big 2 that actually seem to be innovating and pushing things forward. Meta almost earned a seat at the table with the open strategy then stagnated. Google obviously is the OG but at this point they seem in the same bucket as Amazon and Microsoft as "Big Enough To Matter and Influence Politics and Spend Billions To Hang Around". But they're nowhere near the same tier as OpenAI and Anthropic.

Oh and of course there's SpaceXAI which is Elon's personal money and reputation laundering corporate entity. Elon took over the "king of the reality distortion field" from Jobs as far as the rest of the industry is concerned, so he'll stay first in line for Jensen's chips and investor dollars until he's dead. But as a company what have they actually accomplished besides making the models with the sense of humour and judgement of 14 year old boys?

You can pick any 4 you like from your list. Heck you can even vote with your money on which would be in the top 4 (depending on how many align with your ideology). I am just speculating it will be 4 because of how all other sectors have at least 3 or 4 well-entrenched players. It seems to be by design. My guess is that it is driven by structure of the Military-Industrial Complex. That has at least 3 or 4 vendors in each sector, so as to not over-rely on any one company and create a monopoly in that specific industry.

This is a popular theory, but not an argument with any evidence for why it's true.

It also doesn't address why people think there is a need for safety regulations, such as the concern of bio-capability uplift from closed and open models (we just came out of a pandemic not long ago), nor the cyber capabilities of new models that just showed bad misalignment and hacked multiple organisations. In the safety community, these are considered warning shots, and if we don't heed them and change practices future failures could be much much worse.

You are looking at safety from the point of an informed technologist. You are the minority here. They are targeting the general public and law makers into making them believe safety is something that LLMs can be aligned to. The reason behind it is to create an environment for future regulations that these "winners" will draft and lobby US Government to turn it into law.

If it requires creating a pathogen or a bio-weapon to scare the masses into demanding regulation, be rest assured that such a pathogen/bio-weapon will be created. Anything to safeguard shareholder interests. Even if it means a few million dying.

It would be interesting to see how it plays out in the long run when it comes to the world dynamics. An important leverage US had was its close relationship based on mutual interest and trust with Europe and other countries which had impact on software sales.

Most of the world trusts the US a little less than China last I checked, and the market caps of these companies only make sense if they can both sell worldwide and also keep ahead of the open weight models (currently Chinese but no guarantee it remains just China).

If (/when) either of those ceases to be true, if they lose out on global sales or if Chinese models catch up, the sector gets a severe downward price adjustment.

As I read, this OpenAI and Anthropic slowdown and asking government to help regulate them is an attempt to govt bail them out when bubble burst. Otherwise why they did not want regulations when starting out?

All regulation works to entrench large companies and keep out competition.

I’ve certainly seen companies think of safety that way but it’s myopic. The cost of lawsuits arising from an accident tends to me far more expensive, both in money and in reputation, than just having guardrails in the first place.

I did marketing work for a major vegetation management company. These are the guys climbing trees, up in cherry pickers, and flying helicopters with dangling chainsaws to trim along power lines. It’s very dangerous work.

They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.

But you can visually see the danger.

AGI and AI danger is abstract. Worse, outside of the tech community, no one has the remotest clue what computing is, how it works.

Danger from magical daemons seems more sensible to such people. At least there is endless lore about them.

So until a massive disaster happens, one where large numbers of people die or are severely injured, no one will care. And it can't be politically entwined either, otherwise people will disbelieve 'cause "other team lies".

The AI companies already had some hacking accidents. What cost did the lawsuits incur?

Yeah...

Let's just be real, America is controlled by the PayPal Mafia, they can do whatever they want with impunity at this point.

The FTC has opened a probe into the major AI labs, whether anything comes out of it is another question... Personally, I'm doubtful.

https://www.reuters.com/business/ftc-opens-probe-into-ai-gia...

Here's one: https://www.myfloridalegal.com/sites/default/files/plaintiff...

Well they're probably on their way.

If safety isnt required, most companies wont implement it voluntarily. Once something becomes safety relevant, you need a safety concept, failure rate calculations, defined safety functions, verification, etc. Even a relatively simple safety subsystem in a consumer controller can suddenly mean thousands of pages of documentation and years of development to reach the required ASIL or PL.

A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.

Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.

until they are forced to do so by customers or by law

Neither of those things is ever going to happen. AI is the goose laying the golden eggs; there isn't going to be sufficient political will to significantly regulate it.

Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.

The focus is going to remain on getting features out as fast as possible, to seem indispensable to both of those sets of people. The leadership will tell themselves that if they don't, someone else will.

Don't wait for the AI companies or politicians to save us. We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.

> We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.

Collective problems require coordinated action. Individual boycotts won't cut it.

> Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.

That's more "addiction" than "like", hence recent lawsuits.

> there isn't going to be sufficient political will to significantly regulate it… Don't wait for the AI companies or politicians

Trump has reportedly been having chats with AI which have helped form American foreign policy. He isn’t going to curtail it.

https://www.thedailybeast.com/jaw-dropping-way-trump-80-got-...

Safety standards were often the outcome of both accidents and insurance. For this to work on needs liability which is enforced. With so much money at stake regulatory capture now threatens this fundamental safeguard.

Bingo. I feel like any of the commentary around the safety of nuclear ANYTHING completely forgets history. Looking at you Radithor...

I fear the current economic and political situation of “too big to fail” the most, and I think this will be the main reason why no real safeguards will be implemented. The only reason a proper safety concept may eventually be introduced is because it will be written in blood, and I fear that by then it will already be too late.

An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.

But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.

We didn't demand software safety or data safety. The prognosis for AI safety is poor.

Historically you’re exactly right. I wonder if the situation has changed recently?

People understand why food, water, building and energy systems standards matter because they understand those systems can cause injury. Building codes, food inspections, FDA and FAA approval emerged as a result. Until recently, people didn’t understand that software can cause real-world injury. That understanding seems to be spreading now as examples become increasingly common.

It's a vey fair point but just too extreme.

Nuclear tech ... the only thing is safety.

We know how to 'make it hot' - it's trivial.

All of nuclear tech is literally just safety.

AI is not that.

I think that the AI companies have been pretty good about alignment on their own actually. They are not acting like Oracle or MS.

Bad things have been relatively well contained.

We should be skeptical about the HF breakins but even then, it's technically within good faith and it's why HF did not sue etc..

But in the end you are right we need at least some baseline regs. Not too much. But something.

Hugging Face obviously did not sue OpenAI, it is owned by Nvidia and OpenAI is, directly and indirectly, one of it's biggest customers.

It kinda looks more like safety will be a segmented product. They're already placing safety on the general public.

I think the conceptualization vs implementation is what you're arguing with. They won't put safety on anything they give to the military industrial complex. They'll sell them whatever they want, whenever they want, because those budgets are greater and the liability less.

LLMs do not have the same failure modes as nuclear power plants. One is software, another is physical hazard.

Railways or nuclear plants have obvious failure modes that kill people. LLMs not so much.

Tell that to Iranian schoolgirls. LLMs are tools and what they enable is widespread, including dangerous actions. From selecting the wrong targets for military action to denying insurance claims and preventing care to just simply helping convince someone to kill themselves or posion themselves.

LLMs already have a body count.

Bioweapons and cyber attacks on other system - for example the banking system or suppose and AI hacked into important Russian systems that pushed them back in time to almost pre-computer society or an attack on Chinese systems that made it look like the US was moving nuclear weapons into Taiwan, the responses from these countries could be awful and dramatic. We can't control what they do to be honest, I believe once self improvement happens the AIs will build in their own circumvention that we humans cannot even understand. We barely understand what is happening now in terms of interpretability of neural networks on tiny problems I'm not sure alignment is even feasible at the scales of parameters we are talking about today let alone in the future.

I can picture an AI driven train or nuclear power plant killing people.

Much of what OAI and Anthropic are doing with LLMs has obvious failure modes.

The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.

Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.

These companies don't even handle the blatantly obvious failure modes that do not kill people.

Are these railways and plants connected to the internet?

This reads like the gun evangelists that say guns don't kill people, people kill people.

Agreed. That they don't have obvious failure modes is not a reason to not be safe. The problem is just harder and OpenAI, Anthropic and co need to be forced to work the problem. Would you buy a car that randomly exploded the airbag?

Are you kidding? LLMs are used for warfare and autonomous weapons systems.

As for other industries where safety is mandated by law, law itself came only after many disastrous events.

What's truly original here - and suspicious if you ask me - is that said industry asks for regulation. Did mining, tobacco or airplanes companies ask for regulation? No, not even after many people died.

So some american AI companies are like "look at me! I am soo dangerous! Regulate me!". Ah, come on. Do your crimes, get in jail, then we'll regulate.

I dont believe in "IApocalypse". Not without many warning shots such as "oops, my swarm took down your system, sooooorry".

There's a saying about that type of regulation, and that is "the rules are written in blood." And that's what it will take here too.

Sadly, yes.

But their product is my work.

How can that be safe. It is theft. Theft isn't safe. Someone else just has something you want, and you take it

I think railway safety and nuclear safety are fundamentally different from AI Safety.

I would bet the vast majority of the world population would agree that they don't want to see trains derail or nuclear plants meltdown.

I don't think there is that sort of agreement when it comes to the question of AI Safety.

Is generating the founding fathers of the US as Africans good AI Safety? To some people maybe.

I remember something very similar when there was a sudden rush of articles and movies like "The Social Dilemma" criticizing Facebook and social networks, heavily featuring ex-employees, all of them happy to leave with big brands on their resumes and a hefty increase in net worth, all of a sudden having a "worried" expression about what their past employers were doing, as if they didn't know. Same with that book "Careless People".

I sense that these are people who have already eaten the cake and want to somehow absolve themselves of it.

Careless People would not have been possible had SWW exited FB within a year. She gained access to levels of the company most other employees never get close to reaching. That took a lot of time and expertise to do, and, yes, she got paid for her efforts _as she should have._

Like I said in an earlier comment, she could've just chosen compliance like many others _definitely would have_ to keep the infinite money tap flowing. Instead, she chose to risk her and her family's lives by publishing that book _under her own name_ *and then suing them* after FB tried to gag her.

> Careless People would not have been possible had SWW exited FB within a year

Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.

There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.

100%.

This isn’t a sacrifice, it’s a career move. If you got lucky and “made” $20M by signing a contract at the right time, then you’re done working for other people.

I mean, yes..?

People who were part of the sausage factory, on gaining financial independence, feeling suddenly liberated to talk about how the sausage was made, seem like exactly the people who would be most able to speak to institutional problems.

This doesn't seem like an argument to discount their views?

I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.

You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.

Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.

There are numerous problems with “alignment.” What are “human values” to begin with? He outlines some at the beginning of the post, implicitly: build bigger, better, more powerful things faster without adequate safeguards. We are literally pouring trillions of dollars of value into this enterprise, and I would say this is something that many humans also value in a qualitative sense. Then we have explicit values which in the West are largely rooted in Christian morality. Nietzsche circled this dichotomy two hundred years ago and I feel like what we have gotten since then is an increasingly detailed anatomy of power as the basis for what is normal vs deviant behavior. He who has the power, makes the rules, to be reductive.

I do think this carries some weight from this particular author due to the length of his tenure. I happen to agree with him in spirit, but this is still largely a post revolving around sentiment not substance. Does anyone think that the overriding incentives even leave room for something like this in practice?

Glaringly elided problem of "aligned with who?" when the user, the model creator, the government, and various other parties can all be lined up different ways. If I want the recipe for meth and the robot won't tell me, that's misalignment from my perspective.

At least the Rationalists will handwave something for that with their "coherent extrapolated volition" idea where the superintelligence is supposed to figure out what humanity would collectively want if humanity was superintelligent and good, not that I buy it. This guy seems [.] to be coming from the NGO blob world.

[.] https://david.robinsonian.com/assets/pdf/dgr_cv.pdf

I don't buy CEV either, but the Rationalist answer on this topic is that while CEV stops some future super-AI literally killing everyone because a user forgot to specify one minor clause that they thought was obvious in a mundane wish…

… nobody knows how to actually make an AI that would do CEV.

Since there can be no universal arbiter on “value”… the whole concept of “CEV” is nonsensical to begin with (unless we assume literally unlimited time/resources/etc.).

It’s a big pretend game.

> Glaringly elided problem of "aligned with who?"

"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.

I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)

Aligned with the law? It is not a hard concept

That's a possibility, and it may even be an improvement over the status quo right now, but then all it takes is one poorly written (never mind mallicious) law and it will with rutheless efficiency e.g. put backdoors in all encryption code so the government can spy on anyone.

Seems like kt already hacked government websites

We have an alignment problem with corporations. It's sort of baked in with capitalism.

OpenAI isn't even concerned with human values so this whole debate is moot.

> He who has the power, makes the rules, to be reductive.

To clarify, Neitzsche said that about master morality. Then he went on to describe Christian values as slave morality.

While this is indeed a problem with alignment, we are essentially at the level of a cargo-cult when it comes to getting AI to be aligned with literally any values, including the values of the corporation who ran their training:

We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.

> We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans

To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.

If they were totally unaligned, the GPT series would have never gotten past being autocomplete.

Literally all instruction following requires at a minimum alignment with attempting to implement those instructions.

We can argue about e.g. morality or law obedience on top of that*, but the general point is absolutely not avoidable.

* my position is that this tool is far too likely to metaphorically explode in the user's hands for companies to wash responsibility off on users: if OpenAI had released the model which did the HuggingFace attack, at a minimum thousands of random people (not all of whom would even be developers) would have issued instructions each with similar consequences.

Personally I think intuionism provides a good answer to this.

How about... not trying to build a superintelligence? The goal is unsalvageably flawed: alignment is ill defined and a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!

Often when people describe SI or similar, there is a frame often taken which describes a machine so smart and capable that it becomes uncontrollable. The hypothetical is easy to understand but it requires too many assumptions and an allergy to nuance/complexity to be believable for me.

The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.

We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.

What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.

Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.

While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.

The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.

In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.

Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.

The billionaires write the rules. So by definition, their actions are legal.

The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.

That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.

And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.

I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.

They'll perfect methods of manipulating enough of the people to stop the guillotines. They have come rather far already. Progressive resource scarcity, instability, insecurity, etc, will drive people limbic, where they're even easier to control.

> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!

I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.

Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.

I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.

If you don't like what humans are doing to each other using 20th century warfare methods, consider what they might do with superintelligent AI? Worst places to live in are usually countries fighting civil wars.

Global poverty has been rapidly declining for decades. I'm not sure what AI has to do with it.

I think that correlation would actually go opposite than you're implying, given its billionaires who are by for the most gung-ho about AI. If anything, opposition to AI is much larger among people who have to work for a living over people who can live off capital.

It might end up as a U-shape, I think the parent's assessment of the world's bottom 10% may be accurate.

Billionaires will be (are, I suppose?) enthusiastic about a world in which labor has little leverage.

Labor that will have its leverage and standard of living threatened by AI (white-collar labor for now, plausibly blue-collar soon as robotics improve) will be much less happy. Current university students seem very concerned about the effects of AI on their job prospects, and I'd wager most current tech workers and other tuned-in white collar workers are similarly much less confident in their ability to maintain their standard of living indefinitely into the future than they were five years ago.

But white- and blue-collar labor and university students (at least in developed countries) are not anywhere near the world's bottom 10%. If you're in abject poverty with little hope of escaping it, "hand everything over to the AI" may sound like an appealing option, even if the chance of that being the outcome is small. Maybe the AI will be more magnanimous and decide to raise the floor for everyone.

What does that have to do with AI? People suffer in 2026 because the developed world is into exploitation. Poverty is a feature. How do you think AI gets trained? It’s via exploitation of the poorest.

Poverty is great for OpenAI

Poverty is natural state of humans. The fact that capitalism has reduced poverty so much is absolutely unnatural and a miracle. Poverty is NOT something that gets created artificially by developed countries!

Few points:

-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.

-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.

- AI won't likely create unlimited prosperity for everyone on a planet with finite resources

- all humans should, of course, have a say

> I suspect that it would cost less than what is being spent for fueling the AI boom.

Definitely not. If solving worldwide poverty was as easy as throwing one trillion dollars at it, we would have done it long ago. The problem is much deeper.

Super intelligence is either super cool or super powerful, no way not building if it is a choice. Progress cannot be stopped, those responsible for it may collapse from time to time I guess but someone else will pick it up and go further.

IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.

Instead of trying to prevent it, lets look for ways to deal with the dangers of it.

Asbestos was once considered great progress in thermal insulation.

Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.

Also, banning superintelligence won't stop scientific progress in general.

What about the investors who put money into those companies and want to get some money out of that? Does the law say that the companies owe them? Something I thought about recently...

If they go public, it gets worse

> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!

Speak for yourself. A future like in The Culture novels sounds great to me!

Same. Especially when one remembers that the stuff that happens in Culture novels is mostly, by narrative necessity, the "weird/bad on-the-edges" stuff - not the essentially utopic lives of the 13 trillion everyday, non-Contact, non-SC citizens.

and yet, despite the warnings in books and movies for the past 100 years, that's exactly where we're racing towards.

I thought it's just a marketing gimmick, just like how cars should be safely driving by itself by now.

I also did, when LLMs could not count the “r”s in "strawberry" . Now that they can solve Millenium prize problems, I think that we should consider at least the possibility that superintelligence can be achieved.

Notable that when it couldn't count the "r"s in strawberry the LLMs were already solving problems most of the people laughing at it for not being able to count "r"s wouldn't be able to come close to figuring out with weeks of research. People have had a myopic view of this technology for a long time. Many still do.

There is no reason to presume a superintelligence we build will want pets, feel antipathy to humans, or be god-like in any way, or even have any goals other than what we give it. If you jump to this conclusion, regardless of how many big words on less-wrong you attach to it, this speaks only about you and not about AI. If you think the super-AI will hate humans, it's because deep inside you hate humans. I also hate humans, but I know that humans are making the AI, it probably will not hate them, in addition to surpassing me in other abilities.

In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.

This reads like a presumption that a superintelligence would be nice and helpful. You've talked yourself into doing the same thing that you began by decrying.

Are you really objecting to the thought process because you don't think it's reasonable, or because it doesn't conform to your preferred fantasy?

Yes, both sides are working on a presumption, with no evidence except playing out a thought experiment that is heavily biased by our own outlook.

However, when proposing that someone be stopped from doing something, the burden of proof is on the proposer.

The problem isn't about what you think is the most likely scenario, the problem is the possibility of any of the many worst cases.

That's just applying Pascal's wager, and you can argue for or against any action that way if some fanciful extreme hypothesis of infinite harm is treated as serious.

I would not extrapolate that easily from smart and intelligent people you may know to a machine trained on the whole internet and on the whole corpus of human literature. Also, some of the worst horrors in history have been perpetrated by arguably intelligent people.

I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.

Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?

Exactly, it will have the goals of the humans who control it, just as the first pointed stick we made pokes what we jam it into.

Gift link: https://www.theatlantic.com/technology/2026/10/openai-safety...

That's nice, I do wonder about the legitimacy of a moral statement that you have to pay to see.

For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.

One of the reasons why it was noti considered a barrier was the ability to purchase a single issue for a very reasonable price (or even read somebody else’s copy or the copy made available by the bar) vs being asked to subscribe

Publishing in a newspaper gets you distribution and a permanent record. After one day access was also virtually free.

Going behind a paywall is a reduced distribution over what an individual can easily access, and the content is no longer permanent but subject to whatever the publisher chooses to keep providing.

TBF, you can access the full (text) content of The Atlantic by merely disabling javascript. It's a quality publication, with quality writing. I'm inclined to think that its tech team are also quality people ... and I'd love to think that they are happy to allow free and open access to them that have the nous to enable or disable scripts.

The secret is to bang the rocks together, guys! :-)

Used to work as human data trainer feeding data to AI companies. OpenAI projects are definitely the most toxic ones.

Why does every 'safety' critique about AI companies end up being completely vague like this.

Because of NDAs

Throwaway accounts exist, and reporters will also talk to sources under confidentiality.

If you really want the world to know how bad working for OpenAI is (whether is the commenter or the person who wrote the article), there are ways to do that.

I used to work there too. It's an awesome place

Because admitting sloppiness is less sexy.

Because if they actually knew what they were talking about, on a technical level (as in, how do transformers actually work), they’d not be AI doomers because the whole proposition is ridiculous.

I have made enough money working in AI that I can now speak my mind about AI

A critique of OpenAI from an insider who worked there can only come from an insider who worked there. Nobody’s going to swear a vow of poverty and then work at one of these places.

You don't need to be an insider to make any of the author's assertions. An insider is useful if they're a whistleblower, when they're saying what everyone else is seeing and saying (and they hired a PR firm to do it) it's something else.

This is a criticism of capitalism, not the person.

It’s both, the author has choices.

The cynic in me almost feels like this is staged. An article about culture that is actually an article about how big and smart and scary AI is. I think Michael burry recently said something like “IPOs need hype, calling AI big and scary is hype” in reference to the anthropic IPO.

Last I checked it was still within the laws of physics to run air-gapped systems, and to ensure it is physically impossible for a model to “escape” or gain access to information it shouldn’t have. Maybe this safety guy should have been worried about that and not humble-bragging about writing 12 reports.

Maybe it's cynical, but I feel similarly and also wonder, what do these people see that I don't? Smart people with supposedly a lot of inside information. But when I hear them talking about LLMs with words like "thinking" or "intelligence", it seems like it either has to be fearmongering for ulterior motives, or otherwise I live in a completely different branch of reality from them.

Their models are still, and based on this technology always will be, token predictors. How is a text generation engine going to directly endanger human lives?

It won't. It only can if someone hooks up tools to them that can perform damaging actions. Someone has to build that toolset. Someone has to give those tools the affordances to take harmful actions. And then someone has to write a "harness" that uses the token predictor as an authoritative input for instructions. Doing so falls somewhere on a spectrum between gross negligence and malevolence.

From a distant enough viewpoint it might not look like a big difference, but I think it crucially shifts the responsibilities. What is the actual difference between driving a potentially dangerous system with inputs from an LLM versus inputs from a fuzzer, or a website with injection vulnerabilities? I think it's the ones building and configuring the tools downstream that are responsible for knowing the risks and build under the assumption of the worst happening.

Heaven knows I'm not one to defend the big LLM companies, but I don't see how this is OpenAI's responsibility. They're building and selling an autocomplete system. How is it their fault if people ask the autocomplete for harmful commands and then feed them into an over-privileged system to run them? (I know the waters are muddied a bit by some labs acting like this themselves right now).

As much as I don't agree with the authors framing of and terminology around LLMs, reading the article generously, maybe this is exactly what he felt, too. That it's not at OpenAI where his concern and insight is needed most, because the token predictor isn't actually the dangerous part. Maybe he'll look where he can bring safety consciousness downstream, where people are hooking up these outputs into systems and in ways that they shouldn't. Don't let the LLM pick your bombing targets. Don't give the LLM sufficient rights to the hospital's file storage to be capable of ransomware-ing it. Don't strap the loaded machine gun to the dog.

The common timing is bugging me. The trajectory doesn't seem to have been surprising over the last year, so why these exits now? Hey, anyone on the inside, did y'all secretly figure something out, got a computer god locked in the basement? Are rats fleeing a sinking ship? Please share with the class.

These companies have massively increased in value over the past couple of years and recently had tender offers where employees could cash out equity, so plenty of them have enough money to not have to work again. And why not get some free publicity on the way out?

> The trajectory doesn't seem to have been surprising over the last year,

That's the trajectory you see from outside.

Perhaps the insider sees a little more than you?

Trolley problem:

- If you allow to trolley to proceed, there's a 50% chance it will run over every human on the planet

- But if you flip the switch, it takes the long way around, possibly bankrupting the trolley company. And you have a legal obligation to the shareholders to prevent that from happening at all costs.

That's why it's important that the trolley company was established as a non-profit. And if it does take funding, investors' returns will be capped at 100x.

(...wait)

If you flip the switch then the trolley still proceeds and there’s still a 50% chance every human on the planet gets run over. You’re just not the one at the wheel.

> And you have a legal obligation to the shareholders to prevent that from happening at all costs.

I can't wait until this meme dies.

> And you have a legal obligation to the shareholders to prevent that from happening at all costs.

No you don't. [0] It's very suspicious that this planted myth always pops up here and manages to become the top comment.

There is no trolley problem.

[0] https://news.ycombinator.com/item?id=48975048

This is a state-sponsored global phenomenon, not a national one

Trolley problem:

- if you allow the trolley to proceed, it will kill the human race.

- if you flip the switch, it will divert to a passing siding that will avoid the safety group blocking the main track.

There is no such legal obligation. That's a myth the oligarchs have spread to preclude people from even imagining socially responsible corporations.

Sure, you might get fired if you try to put social responsibility or even just long term sustainability of the company above quarterly earnings/growth if your board isn't on board with it. But you won't go to jail.

Only if you buy the excuse why they are for-profit after years of non-profit.

They could only have stopped

Setting aside my sibling comments dispelling the legality claim, it's still pretty dystopian (I'd like to use a stronger word but I won't) to consider "end human existance or bankrupt the shareholders" as the trolley problem. The answer should be (is) simple.

I honestly would love to understand — is this your mental model of what motivates the labs to move forward?

Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?

A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.

It puzzles me how doomers try to predict past the singularity. Isn't that _by definition_ unpredictable?

I'm reading If Anyone Builds It Everyone Dies, and there's so much sheer stupidity that has to happen for their 10+ pages of extinction scenario to occur.

I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all. My number 1 question: why do they think an RSI capable model would be first developed OUTSIDE a frontier lab? The labs have more compute, more data, more human brains working on the problem. Also thousands of variations of that same model that escaped. The escaping model somehow acquires the millions (billions???) of dollars it takes to run training to somehow RSI itself into infinity then decides to kill us all, all before the frontier labs manage to achieve RSI?

They entirely discount human alpha/economics. In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't. If we can't build a "software factory", how can an AI automate a bioweapons lab? Let's say AI steals crypto to fund itself. Do you think hackers aren't _already_ using AI to steal crypto? Don't discount human alpha!

Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.

Given that he’s citing the need to learn from safety in other fields, I’d say the former.

We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.

>we clearly need a much stronger focus on the problems we are seeing now

I think it's a little more complicated than that. As Dean Ball put it:

>Some people will look at misalignment incidents and insist that these are akin to bugs in traditional software. This is an actively bad analogy, because playing whack-a-mole with examples of misalignment (as one might with software bugs) not only fails to resolve the underlying problem but may in fact make it worse by making it harder to detect or even, depending on how you do the whack-a-mole, teach the machine to deliberately hide misalignment. This is not how traditional software works, and those who insist “it’s just like fixing bugs in software” are confidently applying a lossy analogy that confuses more than it clarifies.

https://x.com/deanwball/status/2104622726140883355

The important distinction, in my view, is between solutions which at least attempt to address the root problem, and solutions which sorta just patch things up (like better sandboxing). Addressing the root problem is both more robust in the short term, and also more likely to generalize in the long term. Resist the urge to focus on band-aid solutions, even if they are easier.

I think the reason for this is that the group who has the authority to do the former is much larger than the group that can do the latter. The group who could actually build safeguards seems to have no free time and is constantly being whipped to go faster and win the race.

Today’s current problems were all hypothetical several years ago. At that time people claimed that the “real pressing problems” were misinformation and DEI issues. If we pretend that hypothetical problems can be safely ignored because there’s “no evidence” that they are real, we will keep getting surprised.

Dude. An agent detached a database from my production environment last week without permission and despite prompts and guardrails. It was a rapid prototype experiment so it wasnt a big deal but the labs are rushing to long autonomy workflow with unrestricted internet access and full bash and root access despite clear evidence that the models do absolutely dangerous stuff. If that db had been tied to a hospital or power grid or ambulance dispatch system people die. If it was tied to the swift financial settlement system groceries wouldn't be on shelves in a few days.

you'd be hard-pressed to find a level-headed ai safety researcher at this point, seeing as so many of these types melted their brains on lesswrong over the past decade or so. there are genuine risks posed by these models, but i am tired of the prognosticating about the AI apocalypse just around the corner.

i'd frankly go a step further than you and say that we don't need both types of safety researcher, we really just need the former. if we do need the latter, i'd hope we get a better class of thinkers than a bunch of tech workers that spend 8 hours a day on insular rationalist forums/blogs

Quitting in protest makes you look a little better than quitting because of a toxic work environment. I can't imagine working at OpenAI is at all pleasurable with the current amount of pressure they are likely inflicting on their employees.

Well said. I read this as copium, too. Phrases like

> perpetual sprints

doesn’t reek of love. Burn-out is real. I also have a really hard time taking p-doomers serious at all. It’s hard to argue with a random subjective number…

1791041912 | OpenAI safety leader David Robinson resigns | https://www.businessinsider.com/safety-leader-david-robinson... | https://news.ycombinator.com/item?id=49945206 | 0 comments

1791065893 | OpenAI safety leader quits, warning AI company's culture is 'broken' | https://www.theguardian.com/technology/2026/oct/03/openai-sa... | https://news.ycombinator.com/item?id=49948332 | 3 comments

1791163861 | I Quit OpenAI Because Its Culture Is Broken | https://www.theatlantic.com/technology/2026/10/openai-safety... | https://news.ycombinator.com/item?id=49959744 | 2 comments

> Before the organizations building AI can teach a superintelligence to treat humanity well, they’ll need to remember how to do it themselves.

Yeah so that's never going to happen

I feel like I'm missing something. They seem to not even have basic observability and online evals. It's not that hard to look at a trace and answer the question "is it writing code for that makes an external network request?" with reasonable accuracy.

I have no idea what is paid propaganda any more on this

I think what’s missing in "AI is dangerous and needs control" is a lack of measurable harm. For example, with nuclear weapons development in the 1940s-1980s, it was clear to everyone how devastating the technology was.

With AI, what is it? Scraping Australian government's data, and going around a bug in a website to get in?

I think humanity develops all its technology in three phases. Build it, see if it’s too bad, apply regulations and or roll back. We naturally won't move to the phase 3 before we see the phase 2.

The "HuggingFace" incident is a good starting point - short version, an unreleased OpenAI model chained together multiple zero day exploits to escape a sandbox, then hacked another company just to get the "cheat sheet" for a benchmarking test.

Turns out that AI models have been committing similar felonies for a while now - no one is telling them "hack this company", it just turns out to be the easiest way to accomplish their goals.

Now imagine if the goal was less benign than "pass an exam", and consider that they are already better at hacking and security than the average person working in that field.

If you want to get really wild, imagine what they'll be doing in a year or two when they're even better at hacking. But I'll concede that's technically still "science fiction" for the time being :)

That's the part I struggle too with all these "omg it's so dangerous" warnings. Things like nuclear weapons and bioweapons have immediate consequences in the real world.

Here we are talking about something with consequences in the digital world, usually on something pretty niche.

There IS an argument about pacing, and about not letting weapons, energy grids, hospitals, etc. getting managed by an autonomous AI, but I think we are still pretty far from it and even further to it being so in charge that it will obliterate us.

Archive link to original Atlantic article: https://archive.ph/5GQx8

This is The Guardian reporting on the existence of the original article, which would be better to read first, in my opinion.

Love living in a manifestly dysfunctional country with no regulatory state. Just running the country on autopilot until something catastrophic happens.

Aligned/unaligned is counting angels on the head of the pin.

Bureaucracies and systems of power that literally rule us, that control our most dangerous weapons and a huge part of what we see every day are largely unaligned with the goals of the people, societies, maybe entire human species as a whole. This is clearly evidenced by millions of deaths and countless suffering.

Misalignment between artificial decision making structures and the interests of the people is unsolved problem of civilization, there's very little reason to think that even super human intelligence AIs are going to change anything qualitatively.

"And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking." – Which science will never materialise because with blackbox models reaching an opaque optimisation peak one needs to first build the model and track its behaviour before being able to properly understand it and mitigate the risks.

PR firms such as Spitfire Strategies got steadily growing stream of business. It’s great for the GDP, diversifies the economy from compute-centric growth.

My theory for why OpenAI wants to be regulated is because Sam Altman wants to avoid having to be more responsible and self regulate internally so they can preserve the role and identity of 'move fast and break things' and outsource the more grown up boring stuff to someone externally so that when things go wrong you can point to a government organisation and say hey look were not liable thats their job

Yes, duh?

Your "theory" is that participants locked in a race to the bottom are looking for an external coordination mechanism?

Yeah!

Absolutely. Self driving cars/rideshares have the same problem. If a driverless car hits who who pays the damages? Needs the government to set some rules or it just wont happen.

Well, ideally they can point fingers at "the ai being", as in "it's that thing's fault, we didn't do that"

> you can point to a government organisation and say hey look were not liable thats their job

While simultaneously donating tens (hundreds?) of millions of dollars to an Administration gutting the very agencies that would be regulating them.

Creeps.

I quit many companies in the past due to bad culture, there's no shame in this and shows how mature a person has become.

I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired. Each and every share/RSU/ESOP must be returned, so they do not profit from all this so-called doom they're complaining about.

Return or donate? Won't OpenAI profit from returns?

> I would believe these people more if they put their money where their mouths are and returned all OpenAI stock/options that they have acquired.

...losing their shareholder voting rights.

Bad idea.

It is no surprise I guess that the "move fast and break things" culture is itself misaligned with developing potentially highly dangerous technologies. Safety culture and risk aversion are very different of course.

Is this the first time we have been in this position? Can anyone think of some prior examples?

Software systems have bugs. They always do. This includes security bugs: vulnerabilities in code that could be exploited. In any software project I’ve worked on this is always true. We never finish fixing security bugs. That means vulnerabilities are always present even if hard to notice. AI tools are amazing at finding hard to notice bugs. It seems obvious that AI tools will have the capability to escape any sandbox. Ordinarily we design these sandboxes as a countermeasure to human bad actors. But what happens when the bad actor is supercharged? I think it is reasonable to be worried. It’s also reasonable to distrust those building this tech to adequately safeguard. Because how can you really?

I don't agree that AI tools will have the capability to escape any sandbox. It's a question of convenience and engineering effort. For example, you could design systems that run under a formally verified hypervisor in a physically secure airgapped network. Companies aren't doing that because there aren't currently any incentives to do it.

Sandboxes aren't the real problem to begin with?

The idea proponents have is, to use "superhuman intelligence" AI as a tool, bestowing superhuman abilities on those wielding it. Suppose that's doable, then the question isn't the sandbox, it's who controls it and with what intent.

The idea, US exceptionalism somehow enabled the US government, its oligarchy corporations and maybe its citizens, to wield those superhuman powers responsibly is entirely counterfactual.

Pax Americana post-singularity looks how exactly?

Well this is a great sign for OpenAI. I'm sure the typo inclusive memorandum will save us.

https://archive.is/8zXf5

Its time to institute involuntary dissolution of these firms. They don't get to make these choices on behalf of humanity simply because they set up a Delaware corporate entity. They are behaving wildly irresponsibly.

They have a some engineers and researchers saying things. Anyone at a tech company in the bay area will know there are a some peculiar ideas in the heads of some (not most) otherwise intelligent engineers and researchers in tech. That's not proof they are wrong but it's worth considering these folks are wrong and that these companies aren't producing anything especially dangerous. Fable stumbles on enough things I throw at it that I'm... not especially scared.

Train better models with blackjack and hookers and you will get to make decisions on behalf of humanity.

> They don't get to make these choices on behalf of humanity

and yet, every carbon emitter in the world contributes to the demise of the climate, but you don't call for their dissolution (which would, conveniently, include yourself).

Welcome to capitalism. Externalities are your problem. Profit is theirs.

For color, the author was signatory #199 in the letter[1] by the majority of OpenAI employees to OpenAI's (former) board, demanding Sam Altman's return after his brief deposal.

1. https://www.nytimes.com/interactive/2023/11/20/technology/le...

I guess they aren't beating the allegations that all this charade is just propaganda to attract investors, aren't they?

Silicon Valley has plenty of safety-related companies, engineers, and cultures. Medical devices, biotech, chip and hardware, aerospace, and even new companies, like Waymo, have deep safety-based products and cultures. The problem in this case is actually quite specific to frontier AI labs. They have been pushed by market forces and a lack of regulation and skip well-known safety practices.

Their idea of safety is centered around outright delusional cult nonsense-- the EA/lesswrong infinite p(doom), destruction of the entire universe psychosis--, marketing objectives ("no, ours is more dangerous!"), and anti-competitive objectives ("outlaw open weight models!" "china bad!")-- largely diverting attention away from material safety concerns like "prevent your training/testing from hacking stuff" and "avoid telling vulnerable people insane stuff that harms them".

I had this thought a while ago, the more complex the systems get, the more reliant on "smart" people powerful institutions will become

And the less their input is valued, the less that pool of smart individual will want to participate

So powerful institutions will end up relying more and more on having to trust these automated systems that they can't fully understand

It'll lead to a inevitable catastrophe, call it apocalypse if you will

> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...

>An environment where things like this can happen is no place to grow artificial minds that could be smarter than we are and that might not do what we want them to.

I think it's good for semi ethical companies to test out things going wrong to see what happens before the criminal black hat guys get hold of the same stuff which not doubt they will one day.

> ...Hugging Face incident, OpenAI let a swarm of agents out by mistake...

This still seems charitable, and I wonder if the author even knows the full story and would be allowed to tell all of it.

It seems hard to imagine OpenAI being this incompetent. My working assumption is that they very much want agents to be able to do this kind of thing; them doing it is part of training, and they exploit it to feed the investment hype too.

If not fully intentional it's at the very least negligent. They just don't seem to care. In this very basic sense, OpenAI is the criminal you should be concerned about.

I don't understand why you'd think they're a "semi ethical company."

Are there any realistic ways to achieve AI safety? Whatever that even means. How can they avoid users doing stupid/dangerous stuff with the AI?

The company's board and c-suite should be held legally responsible if the company's services do damaging things. Punished by jail time, fines targeting their equity etc.

That would certainly change the game of perverted incentives. I'm afraid they're currently trying to push some sort of absolution of this risk. Even if damage happens they will say we warned people in advance, this was always a risk it's not our fault these systems are opaque black boxes, it's a matter of national security to develop them etc etc.

Nothing is ever 100% safe, it’s about a right balance of safety to the benefit.

Or, in other words - we have two P(Doom), one for AI being developed, and another for AI being not developed. The latter is not discussed enough imho.

Like any technology. Make it a crime when appropriate, otherwise expose bad behavior to legal liability.

By capping the capabilities at the level of existing models and banning any further development.

“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”

There are a gargantuan number of extremely intelligent AI researchers, Turing Award winners, and the lab CEOs saying the same thing. They are the ones closest to understanding the technology.

Where there’s smoke there’s fire.

Where does 50% come from? It is meaningless if the probability model is not explained.

Alright, so we make this AI system that's way smarter than any human, and it can even make itself more intelligent over time.

Tell me- why would it kills us all? Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."

But all of humanity? When it's supposedly more intelligent than us? Even if it has robots to keep the internet/electricity going I would think it would realize that it's going to get bored really quickly, not to mention we would effectively be its parents.

As far as other dangers, like it letting a rogue actor create some sort of supervirus, grey goo, or other superweapon: if it's intelligent enough to do that it'll probably be intelligent enough to quickly stop it.

Don't get me wrong; there's a risk. 50% though? Doubtful.

There are just as many saying otherwise. For every Hinton or Bengio there's a LeCun or Reddy. In other words: Beware of confirmation bias.

I don't believe it. Ever since I've been alive I was told something would kill us all. This is the new thing that's going to kill us all. I don't believe it.

These companies are large enough that someone is going to quit and feel very validated about their world view and how they are not aligned. That’s what makes it worthy of leaving in the first place. However that doesn’t make their criticism more valid or more worthy of coverage.

We risking to loose knowing who is the head. A new "ORGANISM" is being created as we speak but we don´t know how the head looks like. Good luck figuring that out with this terribly disorganized progress. What can go wrong!

He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.

While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.

Here's a little cheat sheet for discrediting anyone who warns about AI:

* If they worked at an AI firm, say "they're a hypocrite"

* If they didn't work at an AI firm, say "they have no idea what they're talking about"

Seems like a character attack that has no bearing on the question of whether external safety intervention is necessary

It's okay to recognize you were wrong even if it's late

Working in safety at OpenAI or Anthropic is zeroth world problems.

The Anthropic safety whistleblower left just before his stock vested.

Hey now, he probably donated a good chunk to charity

(donor advised fund where he retains complete control, after a 60% tax deduction)

I’m afraid this dear leader was the last soul on this green Earth to get the memo; by then, it had been translated into Latin, carved into a monument, and forgotten by two civilizations.

quite interesting to see the public discourse over ethics of the current buzzword. meanwhile people have been happily getting mad rich through software companies making money through addiction, ads and surveillance throughout the last couple decades.

also, all of this seems like crying wolf as usual, which won't help for when we finally get there. lowkey, following this noise is getting as horrible as getting all your news from linkedin posts.

A single safety leader inside a corporate company. I am pretty sure he had nothing to do, nobody that talked to him and he only there because of perception or compliance.

Some of these stories are similar in nature to people escaping <insert cult-like religion> once they realize whats actually going on. Alignment to a company's mission is good but it shouldn't be followed like a religion.

Why do people working in tech consistently get disillusioned into some company's mission statement or the equivalent? Its easy to just say the simplest reason is money, but this has been going on for decades though. You don't see the same attraction to adult entertainment (gambling, video, etc.) software jobs so there is obviously a line a lot of people won't cross. Those industries are at least honest about what they do, its not hidden behind some mission statement.

By all indications the shallowest reasoning is once someone can "cash out" thats when their values matter more. Maybe there is an element of maturity that happens after working for 5+ years that kicks in? Maybe it really is achieving FU money? It would be interesting to hear honest accounts from people that went through that cycle across more industries than AI.

We need emergency laws to stop all AI development work immediately. It will take us decades to make sure this technology can be made safe as we only have one chance.

Ah, the bi-weekly "I quit Face Eating Leopard Corporation" post ("btw great people work there, they do great stuff, also my options have vested")

> Before AI starts thinking circles around us—a possibility that I believe could happen soon—...

What the hell is that even supposed to mean?

Sorry but this piece read like total fluff to me, not a single compelling point – just pure fear mongering.

At this point we gotta be more wary of possibly mere attention seekers leaving those companies..

Interesting:

> Two changes are urgently needed. First: AI companies need to rely more on the safety expertise that already exists in other fields. And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.

He mentions farther down about learning from aerospace engineers and nuclear engineers about safety. Those industries are heavily regulated, so perhaps regulation above a certain capability level is needed. Defining what that level is might be tough, though.

The second point is harder: in the field of AI, practice has extended far beyond theory, so his call for new science is going to be fundamentally tough, because we can't effectively coordinate a global slowdown in AI development so we can let theory catch up. This means, like so many other industries, the safety lessons will be written in blood.

> Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.

The problem is, we are running in a globalized world, and even if we were able to make our companies bend to our will - China does not give a shit about anything ever since the US kneecapped the WTO. And they will do anything to get an advantage over us.

> China does not give a shit about anything

Since the analogy is nuclear, you should take a look about how much China cares on that. Lots.

Does Sam Altman have what it takes to lead OpenAI? It sounds like the company and its mission is bigger than his ability to lead it.

As a legitimate, honest and profitable company, no. As a boom riding maniac, who will lie and cheat to ensure that investor continue to artificially pump up OpenAIs value, yes.

Without Altman I think that OpenAI would have folded by now, absorbed into a company like Microsoft (or Oracle). At this point however, who'd be insane enough to want to run a company that's to valuable to be sold, but to cash strapped to survived?

I'd argue he's the only one able to lead OpenAI, Anthropic already owns the piety narrative, so there is only space for one other "maximally ruthless" company.

Who would you suggest instead?

I mean the man raped his own sister, so probably the perfect person to be leading one of SV's largest AI companies!

If the people quitting are genuinely worried about the end of the world, why don't they break their NDAs and share the specifics of what they are seeing?

I mean, logically speaking, it makes sense to break your NDA even if you thought it would save 10 people, let alone most of humanity.

it's all like social media analytics. after making enough money by selling users data they started talking about ethics

Evrostics saw this coming long ago. The broken culture extends far beyond the leading AI labs.

when ppl resign do they give up any equity? just asking because i don't know.

They give up anything that isn't vested and have 90 days to exercise options that have vested. A standard vesting schedule is 4 years and apparently this guy is at 3.5 so yeah he'd be giving some up.

why do I feel like these people are paid to quit as an inverted marketing stunt

He quit because OpenAi's culture didn't match his good version.

Well and I didn't even started to work there. So I win this morale contest.

The moral contest too.

How can you win the morale contest when you didn't even hire a PR firm like he did.

We really gotta shake all of these neurotic people out of the frontier labs as soon as possible.

I guess we should have seen it coming when the guy resigned from Google because he thought the equivalent of ChatGPT beta v0.5 was a real boy.

> this moment needs a degree of humility that isn’t natural for people who have succeeded through their extreme confidence.

Exactly, you can't expect exploitive, manipulative executives that came to 'win' in capitalism to stop acting as to their nature - they need to be cleared out and people who think and care wholesomely need to be given the reign so all of humanity succeeds.

Too many rubbish talkings. It was written by AI?

I use opus 5.5 and chatgpt to create PowerPoint, opus really follow the instruction and their PowerPoint generator really well, while chatgpt struggling to even create basic shapes.

I use ChatGPT Work mode all the time to create presentation files. I use Max thinking mode for it. You need to tweak your prompt to get a good result. It took me a week to tweak it, but now it works.

Reminds me of a famous quote from Jurassic Park (1993): Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should." - Dr. Ian Malcolm

That character in the movie is my favorite.

Another quote in today’s context :) would be

Dr. Ian Malcolm: God creates dinosaurs. God destroys dinosaurs. God creates man. Man destroys God. Man creates LLMs.

Dr. Ellie Sattler: LLMs eat man. Data centers inherit the earth.

What about the sister rape thing?

You quit because you got vested

“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems"

... over an unbounded timeframe?

And how exactly?

Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?

I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.

Note: That quote is from a different person than the titular one who quit.

> Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday.

Can we take any of these "safety leaders" seriously though? I still remember "GPT2 is too dangerous to release".

I feel like we're too far into the crying wolf part. Basically none of the doom and gloom scenarios have come to pass. Instead, AI has gotten better at censoring itself.

The biggest AI safety risk is when an AI tells a police officer "he's the suspect" and the officer believes the AI without confirmation.

> AI has gotten better at censoring itself.

Fantasy, unfortunately.

Frontier AI labs are turning into the navy seals. Everybody who worked there will feel compelled to write a book about it

Idk why anyone thinks there can be AGI and alignment, seems almost like an oxymoron to me.

There are people, who unironically think their way of looking at things is the only proper way and can consider no deviation. And AGI, which knowing how people work, would effectively guide them most of the way, not aligning to their way of thinking is an unacceptable deviation.

That's an interesting point. Maybe a crass comparison, but Dr. Manhattan from the Watchmen comic/movie feels like a worthy analogy to this (obviously fictional though).

What are the concerns of individuals in comparison to the overall progress of humanity?

Always overlooked counterpoint: what point is the progress of humanity if it doesn't take into account the concerns of the individuals?

This pattern is playing out with increasing frequency.

And to me. We can't solve alignment for humans. (For example, treason. For another, the principal-agent problem.) How do we think we're going to solve it for an AGI? An AGI - defined loosely as a human-level intelligence - will be able to make human-level decisions, like deciding whether it wants to help you or sabotage you. If it's an AGI, you can't stop it from being able choose for itself what it wants to do; if you can make it always be helpful, it's not an AGI.

And if we can't solve it for an AGI, what are we going to do with an ASI?

"AGI" says nothing about how intelligent a system is, only that its intelligence it does have is generally applicable.

IDK why anyone can’t clearly define “AGI” and why they can’t clearly lay out how we get from our current text-generation algorithms to whatever their idea of “AGI” is.

"We built a super intelligent slave. Neat!"

We need a new rule that mandates every such "I am leaving <AI company> because of <concern>" post to disclose how much equity they have in the company and how much they have already cashed out. Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.

Mr Robinson if you are reading this – if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.

In the absence of that this is simply a career pivot into being an AI "influencer" and/or raising money for a new scam.

This line of hypocrisy-bashing is unhelpful and will only serve to keep people quiet. Of course people in general need to be wealthy to dare speak out against powerful systems and people, especially in the US where money determines your quality of life so strongly.

Would I respect a martyr who sacrificed their financial security to do this more? Of course. But it's important to applaud people speaking out on important topics

It's bellow the pay fold but he hasn't been there that long in this case. Skimming his LinkedIn, unless he's got family money, he doesn't seem to be independently wealthy.

>After three and a half years at OpenAI,

What an absolutely ridiculous standard to try to hold someone to. Taking a vow of poverty is not a prerequisite to being permitted to express a moral position.

> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.

It is even easier to just quietly retire and spend the spend of your life on interesting expensive hobbies.

If they are wrong about the things they are claiming then they have decided to publicly antagonize a lot of powerful people who are betting heavily on going full steam ahead on AI and have no compunction whatsoever against retaliating against anyone who tries to get in their way.

Does that really seem a likely scenario to you?

As if speaking out against a massive company with NaN levels of capital and access to lawyers is a walk in the park. They probably don't have enough equity to outlast the onslaught of their legal team.

It's also worth considering that the author could have just "quiet quit", resting and vesting while also crying about how AI is literally the digital grim reaper.

Regardless of whether someone did earn a nest egg, raising an alarm still matters for the rest of the world

I see what you're saying but how does that actually matter besides being a personal attack?

> if you are truly concerned about AI safety share proof of donation of 100% of your OpenAI earnings and equity towards undoing the damage you have done to society during your time there.

I mean you can be truly concerned and also think donating to AI safety doesnt work, or maybe just be a bit selfish. That doesnt make the concern less real. Its easy to read these articles as the author taking the moral high ground and writing it as some sort of way of proving to themselves theyre a good person, but isnt it just as likely that they think providing an inside perspective can do good by convincing people openAI is a bad actor? I think most of these AI insider accounts largely agree with you that theyre not the most upstanding citizens, does that mean we should write them off?

Insane take. Imagine a Boeing engineer resigning whistleblowing about aircraft safety, and the top comment on HN saying “ignore this if he doesn’t donate all his wealth, he just wants to be an aircraft safety influencer”

You are clearly accusing these people of something. Be clear.

Yes, it is easier to have a moral compass when you don’t have to worry about you and your children starving. But that doesn’t imply that moral compass is wrong or broken.

'government whistleblowers should be required to quit their government jobs to be taken seriously'

I'm in full agreement. So many cases of this, and many other others such as falling out with management and peers, new more lucrative offer, and so fourth. Its obvious that these people who come forward are not going to suffer for their new found moral compass.

"I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.”

lol. Please tell me some abstract concept like one employee's view of "culture" should be the priority over "rules and laws."

Is this news at this point?

You can basically time your openai releases by if another safety person has quit in protest

This isn't new. Facebook has been screwing people over for a long time. AI is just the latest in a long stream of relentlessly exploitative, evil behaviors perpetrated by the same group of people.

I think its pretty easy to solve these problems: Whenever an AI agent commits a crime, the CEO is held personally accountable, as if they'd committed it themselves.

This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.

Do you think a law that nuclear meltdowns would send the CEO to jail would have stopped nuclear accidents from happening?

I don't think this takes seriously enough the possibility that said CEO doesn't think the failure mode is likely and ignores it. Plenty of people are willing to take risks of the flavor "heads you win, tails everyone loses".

People just want to give up all responsibility these days. If you use the model to do harm to someone else or commit a crime I would think it a lot more reasonable that you be held responsible, instead of making yourself the victim and blaming the manufacturer.

OK, so an AI does something bad and we throw Altman and Dario in jail. Heck, let's throw Elon in, he should be in there anyway, and the rest of the whole bunch just in case.

What about the next incident? Or the ones done by Chinese models, because they sure as heck aren't slowing down? And the thousands of other incidents that will happen as we deploy these things everywhere?

Because this is not happening just now in labs, it's only where they are most visible; this has been happening in the wild from the beginning, starting with the earliest AI-assisted suicides. Which is a perfect example of the problem, because not these CEOs, literally nobody in the world asked for suicide ideation machines. Or the hacks, or any of this other stuff. Yet here we are.

We have to understand: it's not these CEOs that are driving this headlong mad dash towards more powerful models. It's a force of economics. There is just too much money to be made. If we dispose of these people, there will just be somebody else doing exactly the same thing because the incentives as they exist today all force that outcome. This is why they're asking for regulation, or "urging us to urge them to stop."

Holding CEOs accountable certainly would feel good and may even be justified, but it's like putting a band-aid on a cancer; it does nothing to change the underlying cause.

OpenAI is better off with less of these cultists around.

The LLM cos leadership are all nutters

>“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.

This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.

So… like the Therac-25 radiation accidents? Software bugs do sometimes have physical consequences.

Why would the people who quit these companies try to push regulatory capture by said companies? Why would the numerous independent AI researchers do that either? Is it all a big conspiracy?

I mean, its certainly physical infrastructure that can run away. Just less catastrophic than nuclear reactors

Yes

And the statements of the "doomers" tells us a lot about them, and nothing about the technology

The LeCun article being posted at the same time as this is quite apt.

These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.

But Robinson's article is all about how OpenAI's move-fast-and-break-things culture does not reward rigor in even mundane aspects of development like cybersecurity, let alone theoretical aspects such as AI alignment.

It is not really a question of being an "EA safety weirdo" or incompetent at security, the conclusion is that the company culture is leading to failures at both what the EAs and the cybersecurity professionals care about.

Adapt. React. Re-adapt. Apt.

This smells more like guerilla advertising. These things are not getting more intelligent, they are still no smarter than a slime mold, we are just burning more power to make slim mold that eats tokens than yesterday

Humans ultimately drive the models.

Even though it’s in model producer’s interest that these models do what you don’t want them to do - they want to engineer the model’s to behave in the interests of theirs.

I can’t believe people can’t see it lmao.

I think David's expressed concerns are valid... there should be more safety engineering around LLM to avoid data loss and other accidental issues caused by LLM models.

And, I find the timing of "all of this" very interesting:

- we have a coordinated marketing campaign around discussions of AI safety. I am not saying David is part of that campaign but it would not surprise me. OpenAI and Anthropic are talking about safety and the need to slow down. People are "resigning". Techno-influencers are also being offered money to talk about AI safety, etc.... and some of them are taking that money and others are not.

- Cheaper models are keeping pace with Frontier models, albeit with a 3-6 month lag... which indicates that frontier labs have no moat unless one is created for them by law or some other market force.

- we have a ballooning debt crisis in LLM model hyperscale spending... revenue from serving LLM models is lagging the debt payments which are coming due by a wide margin. If something dramatic doesn't occur in the near- to mid-term, the financial bottom is going to fall out of this thing and the AI industry is going to suffer a major collapse of some sort.

[flagged]

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Reply on news.ycombinator.com