If you discard all empirical claims categorically due to "inability to observe the counterfactual world", your own reasoning falls apart. Why would you assume people mean what they say?
> Diving their intention also still requires many assumptions about their intermediate beliefs that I find dubious
You're assuming CEOs of multi-billion dollar companies are honest. I find that dubious.
I don't.
There are a lot of counterfactuals we can observe empirically.
I'm just saying there is no empirical evidence to support the marketing strategy of "tell people our product is going to kill their families". I believe this is the first time it's been tried out.
OpenAI got $10 billion from Microsoft after Sam Altman claimed AI could lead to extinction of humanity.
Anthropic got similar multi-billion dollar funding from Amazon and Google after warning AI could develop biological weapons.
xAI got similarly massive funding after Elon Musk predicted a terminator-style future.
If you don't accept this as "empirical evidence", then there's nothing to discuss.