I think you are basically correct to say that the scalability of AGI is a speculative assumption. The argument is that if it were feasible, it might pose an existential threat, so we should consider it.

If one accepts the plausibility of AGI, then AGI that is bigger and faster than humans does not seem to be much of a stretch, but I certainly cannot imagine what capabilities a qualitatively more powerful intelligence would have, let alone how much effort would be required to get there.

Chollet claims that more powerful intelligence is, in fact, impossible, but as Scott Aaronson pointed out [1], his argument from the No Free Lunch theorem does not have any bearing on the question. As Yudkowsky points out in the article, Chollet's other arguments could just as well be used to claim that nothing beyond the level of intelligence of chimpanzees is possible.

But maybe just a bigger, faster, human-like intelligence might present a risk - people have been outsmarting one another for millennia.

[1] https://www.scottaaronson.com/blog/?p=3553