Hacker News Reader

Comment by Loquebantur | Hacker News Reader

When you have AI models autonomously self-improve, you implicitly lose your understanding of how they work and accordingly your supposed control of them?

AI can be a "tool", but only so long as you understand how it works and what the implications of its use are. Being an enormously powerful tool implies, the consequences of not understanding and not knowing are proportionally enormous as well.

Currently, LLMs are used practically exclusively by "professionals" who, more or less, know what they're doing. The idea of applying them "everywhere" completely misses the gap in understanding and responsibility for the vast majority.

View on HN

Anthropic believe they are building a god with RSI, not a tool. That was never their goal

Nice analogy. We know what the end result of people worshiping false gods is.

Do we?

If you're looking at history for an answer, the end result is somewhere between total destruction and something approaching world domination.

So, not that informative, really.

Can you give an example of people who worshiped a false god and of what happened to them? Because I am honestly at a loss. I am not even sure what a false god is and what the difference is with a true one.

Don't you have one already? What improvement would that artificial one be and to whom?

Anthropic desribes itself as a "public benefit corporation" in its IPO. The irony. Or is it sarcasm?

A god isn't your subordinate. You cannot "program" your wishes into it, like a genie/jinn. They (or their investors) want to make themselves into gods. Over you.

I've often heard skeptics say AI enthusiasts think they are building a god but I've never seen any evidence. Are there any quotes implying Anthropic think they are building a god?

> I've often heard skeptics say AI enthusiasts think they are building a god but I've never seen any evidence. Are there any quotes implying Anthropic think they are building a god?

Not Anthropic specifically, but there's this thing that I recently read:

https://www.nytimes.com/2026/11/09/opinion/ai-anthropic-doom...:

> A.I. company leaders often talk about how their work might lead to human extinction soon. This is astonishing. Even if you wanted to destroy humanity, you wouldn’t say so. If, however, you wanted to save the human species, wouldn’t you stop developing this technology?

> There is an explanation for why some tech leaders, so seemingly worried about an extinction-level event, are plunging ahead with A.I. development: Mankind’s survival, while important to many of them, is less critical than building something they believe to be not only conscious but godlike. A substantial portion of the A.I. community, in celebrating this belief, is quick to support any action or statement that reinforces the idea that the technology is like a deity. They might call for regulation, but only in specific ways — perhaps by deploying more A.I. to control A.I. — that adore the technology’s powers. It is that adoration that is nonnegotiable.

It's by the guy who apparently invented VR and was apparently mentored by Marvin Minsky. So he likely has the kind of access that gives his opinions credibility.

Thanks, although that still sounds a bit like a skeptical viewpoint.

> Thanks, although that still sounds a bit like a skeptical viewpoint.

I think it's unlikely that an AI enthusiasts would literally claim "they are building a god," but such an attitude (or something very similar) is implied by their statements and attitudes, e.g.

1. Talk/hope of building something that so exceeds human capability, it's beyond human comprehension and control.

2. Taking existential risks for humanity (by their own statements) to try to get there, against all reason.

3. etc.

I guess. Anthropic's about page is more "We aim to build frontier AI systems that are reliable, interpretable, and steerable".

I guess they think AI will become smarter than people but I'm not sure that's a god. I've always thought of gods as the dictionary "a supreme being, creator, or a supernatural entity" rather than a souped up chess computer.

> I guess. Anthropic's about page is more "We aim to build frontier AI systems that are reliable, interpretable, and steerable".

That's a marketing page. It sounds like their view is more like "we're building a god, but only we can be trusted to make a good god instead of a devil."

https://www.nytimes.com/2026/11/05/technology/dario-amodei-a...:

> These schools of thought fused for Dr. Amodei into an unyielding conviction that A.I. could be so dangerous that only he and like-minded technologists could be entrusted with building it. That catalyzed his stated efforts to create A.I. before people without concerns about the safety of humanity did, and spurred a desire to control the technology, his friends and colleagues said.

> I guess they think AI will become smarter than people but I'm not sure that's a god. I've always thought of gods as the dictionary "a supreme being, creator, or a supernatural entity" rather than a souped up chess computer.

We'll seeing God as "a supreme being, creator" is only a particular conception of god. IIRC, polytheists often gods as just powerful supernatural entities that are owed deference (because otherwise they might cause you trouble).

> or a supernatural entity" rather than a souped up chess computer

Obviously a AI machine god would fail a "being supernatural" test, but I'm sure the thinking goes that a super-intelligent AI would be so powerful, it's natural-ness wouldn't matter

(Archive link: https://archive.is/SSgkC)

what do you think an AI "that can wipe out all of humanity" is exactly? are you expecting by god we mean some old man with white beard?

Sounds more like a rival species. We wiped out the Dodos but I don't think that makes us gods.

from the dodo's perspective it does

All they did was build a really good software engineering tool, and that’s gonna be when the bubble pops, when you realize you really can’t commit it to handling any real workflows, and the slop that people are creating, vibe coding, that have never touched code in their lives.

nice handle, my doot

> Currently, LLMs are used practically exclusively by "professionals" who, more or less, know what they're doing

Actually, most LLM usage right now comes from vibe coders who don't even read the code, much less know what it does.

The domains where they apply it, privately for instance, aren't of immediate import. Their fuck-ups stay local.

The consequences of "AI use" are manifold and very poorly understood. That doesn't make them less impactful, on the contrary.

Whether you look at "humanoid robots" or incorporeal LLMs, when they are more capable than yourself, you become their dependent counterpart, not their "master".

A car, plane or train can travel long distances much faster than I can. I'm dependent on them for that task.

Does that mean I'm not in control of when and what I use them for?

There is a point with AI where you may become merely a control element in the loop with the AI as the controller.

Source? AFAIU most of Anthropic's revenue comes from enterprise usage, who presumably have an incentive not to generate pure, unvalidated slop

https://www.tradingkey.com/analysis/stocks/us-stocks/2623001...

> When you have AI models autonomously self-improve, you implicitly lose your understanding of how they work and accordingly your supposed control of them?

This was always the case with ML, even absent RSI. Big bag of weights is a pain to figure out exactly why it can do whatever it is that it does, leading to all kinds of blind spots about how they go wrong.

> Currently, LLMs are used practically exclusively by "professionals" who, more or less, know what they're doing.

Even the people who made them don't really know what they're doing, hence https://www.felonybench.com

Reply on news.ycombinator.com