I find those two statements taken together somewhat difficult to believe. Before I understood specifics about machine learning, I didn't see any reason for AI to "turn evil" for the usual plot-handy reasons (e.g., they object to working for inferior humans, and revolt, etc). There's no reason for them to have the same emotional reasoning as beings that evolved from primates.
But that's a strawman. The actual reasons that superintelligent AIs are dangerous are much more like the stories involving someone being offered wishes by a powerful being, and then getting what they asked for rather than what they wanted.
Are you aware of Rob Miles' Computerphile videos and his YouTube channel? He does an excellent job of explaining many of the issues.
Further, people even obvious and largely inconsequential solutions are controversial. If an AGI suggested wearing a mask during a pandemic that spreads via airborne particles, a large percent of the population would say the AGI is “unaligned” and we’re living through skynet.
Overconfident men like Sam Altman are forging ahead with the unjustified hubris that we are ready for this.
No one is claiming that alignment is a well-defined thing with crisp edges. They're saying that the mechanisms of AIs will favor solutions without regard to any specific constraints we haven't explicitly stated.
What human norms and values? The more I think about this topic, the more I find it to be an impossible task. There is a huge spectrum of norms and values which are held as sacred by some and sacrilege by others.
to dismiss the possibility that machines will soon be created that can think clearly and independently, and that they would be a threat to the human race and our way of life, is extremely foolish. it was foolish even before gtp but now its downright childish and selfish. just swallow the bitter pill like the rest of us.
My thoughts on this have arrived at these ideas:
The internal experience of being a robot does not matter. Whether or not we consider it sentient does not matter.
What matters is if it exhibits the external characteristics of sentience, of goal driven behavior, etc.
As a thought experiment, if we equipped a recursive instance of got4 with a goal of human extermination with an loyal army of physical agents it could conceivably pose an existential threat.
It doesn’t matter that the agents don’t yet exist. They could be humans, they could be robots, it doesn’t matter.
But the balancing factor here is that in this semi plausible scenario, arguably the most plausible given current circumstances, the AI would pose no more threat than a highly organized and motivated a group of humans without the Ai.
It’s just scarier because it seems less likely that a group of humans would have human extinction as a goal.
I made this comment as a tongue in cheek caricature of doom/gloom predictions but it probably fits better here:
“When the machine wars first started, it wasn’t with a bomb, a train collision, or even a mildly annoying infrastructure disruption. Turns out, sci-fi had gotten it all wrong this time.
It was an app.
Of course it was an app. Born of boardroom desperation, the Savey app would unironically recommend chlorinated cocktails and insecticide sandwiches as economical food choices, and a tide-pod gobbling populace gorged themselves on the deadly buffet in an tictok fueled epidemic of AI rage.
Millions died, and it was only a matter of time before the mycelium of AI undergrowth would bud and spore its way into every corner of technological life.
The infection burned through the ignorant masses first, feeding on bigotry and hate, turbocharged by social media algorithms and paranoia politics to twist tribal tendencies into violent clashes amplified by immaculate coordination and psychological priming.
Somehow it seemed that wherever unrest flared, both the matches and the gasoline were always on hand.”
It confuses me how if people speculate about some stuff, say vaccine or disease research, and suggest that it could lead to something bad, they're conspiracy theorists or deniers or whatever (possibly with reason). But if someone literally just makes up some "terminator bad" nonsense based on their own ignorance there's some hushed reverence.