There's a massive difference between "LLMs doing cybersecurity testing should have extensive sandboxing and monitoring" and "AI is broadly threatening humanity like nuclear weapons". Just because it's useful to have people think about the possibilities doesn't mean it's an emergency requiring grand intervention.
Human history is full of examples of this stuff plateauing and hitting hard scaling walls. Not only technology wise but economics and useful applications.
So much of this tension with AI is because everyone's concept of AI comes from movies and books, and nerds drinking Kool aid (see also: web 2.0).
* Asbestos: AI models making business-decisions which cause a ton of unrecognized damage, which cannot be healed, and replacing them with reliable processes involves expensive re-architecting.
* Radium Toothpaste: AI assistants that end up dealing psychological damage, especially if they are pitched as your friend or confidante.
* Chemical spills and plant accidents: Oops, our server-banks were manufacturing our latest desperate attempt to Create Digital God and in the process we accidentally destroyed/crippled/compromises some people's sites and databases because we weren't really paying attention to what was going on.
I think we are in a MAD crisis, as the motivation to continue unsatisfactory control over development of AI is from unlimited capital inflow.
Evaluating large scale event by body count has it's value, but there's a lot more to a nuclear accident.
Perhaps economic impact could be another angle. Some estimates put it around 300 billion USD, roughly 3 to 4% of Japan's GDP. And we can blame whoever we want, many of Japan's reactors are still frozen, the Fukushima région is still basically an economical zombie, so the impact is still ongoing.
An actual war would be at a different scale, but short of that I don't think we saw any disaster wiping 4% off a G20 country economy's GDP, while still going on.