Imagine something that thinks 1000x faster than you, is thousands of times smarter than you, never sleeps, never eats, can self-improve, can self-replicate and it didn't like you, or someone deployed it against you and your ethnicity, family, whatever?
It would be devastating. You'd be fucked. We'd all be fucked.
Personally I don't think such a creation will ever be practical or work the way we think it will, but if we'd like to keep enjoying the world and value our autonomy, I think we should be very cautious when implementing it.
Yeah that would be bad. We're also way way off from that point. A real threat that we actually face now is someone like Sam Altman getting congress to make it illegal for individuals to run models locally because then he can't charge us subscriptions for them or harvest information about our lives, thoughts, and feelings. Imagine an AI therapist (I've already done this with llama 65B a bit) with telemetry that lets commercial interests know all your specific deepest desires insecurities and fears so they can sell you stuff.
Skynet isn't the pressing issue, a Corpo-Techno-Stasi is.
We will need AGI to ensure continued existence of the human race. Without it, our destruction is assured.