Do people imagine AGI as in movie Transcendence and think it will happen in there's life time ?
Do people imagine AGI as in movie Transcendence and think it will happen in there's life time ?
Now take it a step further and say that GPT-5 needs no user prompt to initiate a run and could start taking actions by prompting itself.
I could see things going south in this example scenario.
The curious and unexpected challenge of modern AI is that we have succeeded in making useful AIs which inherit all of the biases/problems of their training material. ChatGPT can effectively write exploits if you prompt it correctly - we'll likely see the first mass AI driven 0-day security vulnerabilities in the next ~2 years.
GPT Transformer model's are known to solve control/RL problems well. I wouldn't be surprised if we see ChatGPT derived models operating in physical space soon as well. A major limitation with flexible factory automation robots has been that it's too expensive to program them for the task - I don't think this will be the case 5 years from now.
Maybe for outdated and less-common software, even then assuming it gets the input about what specifically to exploit. Which makes it not much of a threat any time soon.
I know some people that don't talk about anything other than perceived threats even though they literally live in the safest society in human history.
AGI really fits this mindset like a glove because it can be perceived to be a threat to literally anything. It is the perfect idea to worry about.
Chemists doing bad things is obviously not really something to worry about when doing a historic cost/benefit analysis so it won't even register.
It could influence people through the creation of media. Shut off vital systems. And stuff I probably couldn't even conceive of.
I think if an AI or AGI could understand the world and have motives to do bad, it would have an ego or an identity to discriminate between itself and the world. Then I guess it needs an intrinsic will to survive like a biological creature? If it did have this drive to survive, I suppose it would also need to experience fear and pain of some kind, or else it might blow itself up, or build a better version of itself which might destroy it, so it would in some ways it might exhibit similar inhibitions as all other conscious and or biological beings.
On the other hand if we build something that's super smart, and doesn't really care about or understand survival, it might just turn itself off for not really understanding it's purpose.
The most dangerous thing I feel is humans using these things as weapons.
Like what are we describing here, a super intelligent being which just makes paper clips? Why would anything that could understand, think and or innovate do that?
Ostensibly, people are "intelligent". People act against their own rational long-term interests all the time. Point being, people have some kind of internal motivations that sometimes supersede rational choice. A paperclip optimizer, by definition, has the motivation to optimize the production of paperclips. This supersedes any other rational goals if they would interfere with making paperclips, and that's why it would be dangerous.
All this AI / AGI speculation really sounds like we're inventing mythological like creatures, which is quite interesting when looking back through human history because they've always been part of the narrative at some stage only maybe this time, we might make something like that for real, or we're deluded and in 1000 years people will think we were primitive ancient thinkers, interesting times ahead.
We might create a very powerful system which we can't determine is fully aligned with our values, but can determine is at least loosely aligned with something we care about. Then we might direct that system to do something loosely aligned with our values (e.g. maximize paperclips), at which point it does something against our values as well.
And it's not hypothetical. In RL, you do see models exploit simulation errors to maximize reward in a way we didn't intend and would have disallowed if we'd thought of it. Likewise across automation, we get what we built, not what we intended.
We have solid historical precedent to expect to not get quite what we want, and the tech seems to be on the precipice of being very powerful, so I'd call it a very straightforward fear for us to worry about making something very powerful that isn't quite what we want.
I mean it would have to be a pretty big leap from and algorithm optimized to do "something" to an algorithm which understands how to operate interdependently and then escape it's box and then start to do the wrong thing, and to add to that, be smart enough to stop us from turning it off before doing too much damage.
Actually curious what people think about this?
For example, take the title of the article we're talking about. You could take it with a dose of anthropomorphization and interpret it with human-like intent and sentience, or you could take it as an analogy that we might formally define closer to "not worried about AI that passes Turing test, but AI that fails for some reason other than the ability to succeed (and that other reason requires the evaluator to be misled)."
So is sentience far away? No idea. Is misaligned AI a near-term worry? Yeah, absolutely. And "misaligned" doesn't have to be as extreme as 'makes humans extinct.' It can be as simple as 'is an asshole' or 'inexplicably swerves into the other lane 0.01% of the time' or something else we don't want in the system.
Define malicious if you had no reason at all to care about whether or not humanity survives.
I'm not malicious when I pick a flower. But that flower is now dead.
What if, to AGI, we're no more relevant than that flower?
We cannot know. We cannot fathom to guess the morality or ethics of something like AGI. We really only have a rough understanding of the morality of animals.
We can't pretend that it'll be essentially human, but X. You presume that it might turn itself off because it doesn't understand its purpose. And that's a fair conjecture. But it could also desire to improve and come to the conclusion that the optimal path for its continued improvement is through us rather than with us. It could also desire to do nothing more than help us achieve. It also could just not want. It could be satisfied navel-gazing until it is requested to do something specific. It could also not care what that request is, just completely amoral. Committing genocide and making risotto may be equally acceptable to it.
The creation of AGI is a watershed moment. The world before and the world after is fundamentally different. And we can only really guess in what ways it will be different.
-- Eliezer Yudkowsky (2006)
So if an AI gets to the point where it decides, or is programmed to consume all humans for their atoms, what will power the machine to get to this stage? Why would we just allow it to continue doing this?
You'd have to assume that by the stage, whatever "sentient" program is making the choice / plans to consume all people for atoms, it would have to be factoring into it's plans that we will likely try stop it from doing so, this is where I think it gets "weird".
To have that level of intelligence and self-awareness to pull off a plan like that unnoticed will likely come with a lot of "baggage". For example, it would need to understand motives, reasoning, have a concept of a self and self-protection etc. This is where I think it kind of becomes a bit of a mythological story.
If you really try and think about the way an AI would go about consuming all people for atoms, you start to run into some pretty interesting walls.
Maybe I'm being hyper naive and I get that, I just personally have thought about these scenarios and it raises a lot of questions about what a scenario like this would actually look like in practice.