The latter is a more defensible concern. The former is a scifi premise that had more credibility before the current era than since, IMO.
> Especially if the AI itself is being used to harden their defences.
Your use of "is" rather than "isn't" only makes sense to me if the former interpretation is intended. To the extent that you mean to say the latter is also part of it, it's undercut by the defense being AI-augmented. (Unless you mean the bad actors will be the ones in control of all AI, i.e. OpenAI and Anthropic and friends?)
> A digital adversary could do huge damage just by taking AWS offline. That's just one possible attack vector among many others, most of which aren't well-defended.
I think you picked AWS because it's one of the few high-consequence digital systems a rogue ASI could plausibly take down on its own. Anything that could cause actual existential risk would involve more than disruption, but long-term orchestration of systems that are air-gapped, time-gapped, and meat-gapped.
Note: This is a very good argument to never remove the air, time, and meat gaps from those systems. If they do get removed, it will not be the genius of AI, but the stupidity of mankind that dooms us. Even Qwen could kill us at that point :)