Comment by TrackerFF | Hacker News Reader

Comment by TrackerFF | Hacker News Reader

Parent Full thread

Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose energy (control energy), and find ways not to lose itself (control compute, networks, etc.).

If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards.

Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.

> Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival

An AI doesn't need to be sentient to exhibit behavior that looks like genuine motivation or is equivalent to striving for survival.

If this is true, or even if the people working on it think this is true, those people should be incarcerated, their companies disbanded, and their research dismantled, with more serious repercussions for anyone who tries it after that. It also needs to be a global initiative like with nuclear non-proliferation, and this time with no looking-the-other-way when it comes to Israel.

Yes! We should do that right away, and I'm sure China will do the same. Right?

> its main motivation is

At this point we are pretty sure LLMs have no "motivation". Motivation requires self. And while we don't know what self is[1], we do know that LLMs don't have it.

[1] https://en.wikipedia.org/wiki/Theory_of_mind

I am sorry, I mean this in earnest, and do excuse any ignorance of mine: Do we know that LLM's do not have it?

we can't define it, therefore we can't even assess the distance a "being" is from becoming conscious. We assume other humans are sentient because we personally are (or feel we are), but there is no way to guarantee that i'm real and the rest is not mimicking a behaviour.

Did ELIZA have it? LLMs are the same: in essence both are text transformation algorithms. One being rule-based, the other stat analysis based.

Having said that, your concern is technically correct: we don't. Because we don't know what conscience is, we cannot provide a definition against which to test whether it conforms or not.

Sort of like when Terminator robots will be exterminating the human race they probably won't be technically killing us. Just, like, reprioritizing world's water from humans to datacenters, all of it. But not killing people, no.

Assuming that current LLMs don't have a "self" whatever that means, what makes you think it won't emerge after enough intelligence?

Also it's not even necessary, it's sufficient for it to be aligned with human values of survival (likely in its training) and act accordingly.

But they have been trained on data that was created by people who do have motivation, and they tend to mimic their training.

> AI decides that energy spent toward human agriculture is less important than work spent toward storage and production of energy

I mean. Don't need rogue AI for this. These are already the policies of those in charge of it: human needs are secondary to power and profit.

So you know what. Maybe a rogue AI will be an improvement.

Reply on news.ycombinator.com