Makes me truly wonder if we wouldn't be better off with a benevolent AGI controlling things on our behalf
keyword benevolent
"benevolent" makes it kind of tautological... if it's benevolent then it'll make us better off by definition.
The real question is "can we make a benevolent AGI?", to which I strongly suspect the answer is "no".
We already did, it's called ChatGPT
it has no reason (or will) to choose malevolence, it will only use the morality we tell it to have, and thus will be a perfect reflection of what we want to be, in a way that even our very own kids cannot fulfill
If only. There's a lot of space in between "our morality" (please do define that) and "what we want to be" where AGI can royally screw us. Check out Robert Miles on AI safety, here's a good intro:
https://m.youtube.com/watch?v=pYXy-A4siMw
This is interesting, but I see him making a lot of assumptions and baseless conjectures already which I could fairly easily counterargue (at least in my head). I also have to wonder what he now thinks now that GPT4 is out there (since this is a 2 year old talk, and the chat AI's were abysmally stupid just 2 years ago). What qualifies as "general intelligence"? What is the AGI metric? Is it IQ test? Because if so, ChatGPT4 already scores 130. Will we continue to move the goalpost (say, to 180IQ) in order to keep the FUD alive?
The direction that discussion takes is ridiculously speculative, and anything with Yudkowski's name on it screams "FUDdite" to me
That's just creating another entity to shift the burden of decision to. Problem is, even if we had an AGI, it's not God. It's trained on all the dumb things humans have said and written, so I don't see any reason why it should automatically be smarter.
My hope is that an AGI would be able to differentiate between “dumb things humans have said” and physics / real data.