>So it is with any technological innovation. Should computers not have been invented because they eliminated jobs? Should steel? What about agriculture? The future is sure to be different, but that doesn't mean we should fight to deny progress. That way lies the Luddite and Conservative. It's only possible to use new tools for good, not try to erase them to prevent evil.

It's not even necessarily evil, right? It's more likely just one big "whoops, majority of us didn't see that coming, and/or didn't agree with those who said it was". But that whoops lands you in a position that you may not be able to recover from.

Ben Geortzal makes your exact argument here: https://youtu.be/MVWzwIg4Adw?t=2795 and in the same interview he is talking about how his company already spent time and money on building decentralized infrastructure for it to run on "so that no one person can turn it off and stop the singularity" and gives his take that the takeoff is likely to be on the order of years and that it's going to be very rough.

So, he at least seems to acknowledge that it's very risky, and that we have no idea what we're actually walking into, but he also just doesn't seem to care if that happens to be a one-way door we walk through and find a big "oopsie" on the other side of.

This is the guy credited with popularizing the term AGI in the first place. I dunno man... am I wrong to be worried?