I'm enthralled by the philosophical aspects of AI/AGI development at the moment. Particularly interesting is the reaction from laypeople, such as those on Reddit, to technology leaders expressing their worry over the future of AI.
Without having read into the matter at all, the mass seems to be comfortable with writing off Musk, Gates, Hawking, etc. as wringing their hands unnecessarily, and besides they're not AI experts so why should we care anyway?
There seems to be a gross misconception with what AGI would be and what it would mean for us, and an apathetic view to the cautious approach many are advocating regarding its development on grounds such as:
i) They'll never make a machine as intelligent as a human. Even if they do, we can still program it to do what we like.
ii) Anything developed to have the intelligence of a human will come to love our species or can be taught to love our species.
iii) Any artificial intelligence developed will be single task orientated and will not go outside of those boundaries.
iv) Artificial intelligence will be developed by eccentric enthusiasts and will therefore not be developed to do wrong.
Bostrom, in Superintelligence, has said that AGI is will be the last great challenge of our species. If we get it right, we will win, if we get it wrong we are doomed.
The problem there being we will not know if we have got it right until it is too late to do anything about it regardless.
With that being said, if we are to develop something that is:
i) Objective
ii) Purely logical
iii) More intelligent than our species combined
iv) Has free access to the complete knowledge we have produced
Then I simply cannot see a way that it will not go against us. If we were to observe a species acting in the manner we as a species do on this planet - to our own species, to other species and to the planet/environment/ecosystem as a whole - I cannot see how we would not categorize that species as a disease, a cancer, something to be eradicated.
Combined with the fact that the actors that will be involved in AGI - capitalists, the government and the military - I cannot see how it works out well for us.