Not scared about global warming, not scared about war and countries bravely stocking up on most deadly weapons, not scared about capitalism eating itself to the bone. Future potential AGI is what they choose to be scared about because suddenly producing words got a bit cheaper.
The rest of your post aside, people concerned about AI safety tend to also be concerned about nuclear weapons.
Eg:
- https://asteriskmag.com/issues/1/the-illogic-of-nuclear-esca...
- https://www.lesswrong.com/posts/FGTgeweYNxmMBx4fz/existentia...
* Global warming is going to be painful, but seems like we're on a track that doesn't involve earth becoming completely uninhabitable. I'm still concerned about it.
* Companies have a much stronger incentive to make AGI than deadly weapons. It will be incredibly profitable.
* We have zero historical examples of a species developing a technology that is smarter than it is to look back at and establish some sort of probability for how this will go.
* All the issues with capitalism, where we have this system that isn't completely aligned with the well-being of humans, though it has kinda worked in the past -- now imagine that it's turned up to 1000 times the power. If you're worried about capitalism eating itself to the bone, that's the same thing to be worried about with AI.
To say that "producing words got a bit cheaper" really trivializes what a massive impact AGI will have, and how soon it's coming. I personally think that the probability of humanity being ruined or destroyed is probably either close to 0%, or close to 100%, and we don't know which, or if it's bad, how long we have. Even though 100% end seems more likely to me, I still think it's worth trying really hard to get to a good future.
If we had knowledge of this happening to several hundred different alien species, and could see the distribution of outcomes, it would be a lot easier to plan for this. In the absence of such evidence, I think it's a mistake to try to over-generalize from challenges humanity has successfully faced in the past, to "creating a new form of superior intelligence".
Are you sure? Various countries are throwing hundreds of billions of dollars on the table of weapons manufacturers. I think this market grew recently way faster than AI.
> We have zero historical examples of a species developing a technology that is smarter than it is to look back at and establish some sort of probability for how this will go.
We already have so many superhuman technologies under our belt that allow organizations, which I could argue are a form of soulless AIs to compete with interests of whole societies. I don't think neural AI will come to beat our organizations at their game just because if you pour into it significant fraction of our total computing resources might be slightly smarter than a single human. Our bureaucracies surpassed a single human with their cunning, and self-preservation and crushing power centuries ago. We already live in the world made by AIs for AIs. They are just made of humans not silicon.
> To say that "producing words got a bit cheaper" really trivializes what a massive impact AGI will have, and how soon it's coming.
I am not convinced it will be more unique than internet or printing press. I believe AIs will pressure us into changing some things just as any relevant technology have done before. Internet came with its own set of SF scares of hackers taking over entire world or brains fried by man machine interfaces that were supposed to be just around the corner. We lived through so many technological revolutions that impacted our way of life and work. I don't think AIs or even AGIs developed from what we have now will be much different. I think the example of self-driving cars shows us that pace of progress going forward will be more disappointing than anything else.
Another thing is that people vastly overvalue intelligence. Tiger might be less intelligent than human but it will still tear a single unaided human to shreds. People concerned about AIs should focus less on how intelligent it is but instead of how large physical claws are we willing to equip it with.
This. But also, if it were smarter than us (or even if people believed that!), then it could program people to be its claws. I don't want to give it physical control of weapons, or even the means of production, but I also don't want it to manipulate peoples' minds.