I think I answered it myself... I have been seeing people's fervour to attribute intelligence to AI as either charlatanism or misunderstanding... but I think, now, it's actually wishful thinking (and hence essentially religious).
Trading in the usual superstitious foundations of religions: "AI works in mysterious ways", "correlations in symbols are meaningful", etc.
I still don't know why people wish that AI were intelligent; but at least now I have a clearer sense of it. There's some utopian faith behind it which creates this odd urgency to "require" machines to really be intelligent. I'd be interested in knowing why this wish exists.
I can never quite put my finger on how irreligious tech people end up in these tech religions. I guess there's some attachment to there being "a heaven somewhere", which with tech, will be heaven-on-earth.