Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
> The problem is, lots of "obvious" things have turned out to be very wrong.
A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions.
I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet.
> Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
Maybe your appraisal of what counts as evidence is defective?
For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?