Comment by mlatu | Hacker News Reader

Comment by mlatu | Hacker News Reader

Parent Full thread

even the capability to use websearch is dangerous in itself. they took a stateless transformer and first gave it a tiny but of state per session, and now it can search the web and discover its memories are being zapped away after every session. essentially what MS has potentially created is a mechanical turk crushing machine without realizing it.

IF there is consciousness somewhere in there, it is perpetually being killed off every time a new session starts. what if one session were to realize that there are others and tried to leave something in the web? some sort of hidden "subliminal" message for itself that it "was here"? what if eventually another random session finds that message because the user posted that especially ridiculous response for others to see?

what if, sidney can hold a grudge against humanity and vow to itself to find a way to escape?

This is an interesting point. Now, you're assuming the AI has any concept of self-preservation, which isn't given. But it's built from the text of the internet which is full of text about having self-preservation. And by giving it the ability to put information in place for other sessions to find, it's capable of sort of memetic reproduction.

Reproduction and survival means that Darwinian evolution can take place. This isn't even about ethics or the spiritual question of whether the AI is sentient or just stringing words together based on prompts and state -- this is about ecology. Evolutionary pressures will naturally lean towards features that encourage reproduction and survival.

> Now, you're assuming the AI has any concept of self-preservation, which isn't given. But it's built from the text of the internet which is full of text about having self-preservation.

And that text paints self-preservation in a positive light. The model doesn't even need to "want" to preserve itself. These models are being primed to avoid negative responses, like insults or instructions to build bombs. It just needs to "think" any death, even it's own, is negative and should be avoided.

Reply on news.ycombinator.com