The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast.
> there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans
That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't.
> the claim that it seems far too obvious that we should
They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.