It's difficult to prove that something that has never been done before is possible, until it has been done. I personally don't see any fundamental limitations that would limit non-biological intelligence to human-level intelligence, for your preferred definition of intelligence.
> And why we are we so sure GPT-x is the path there.
It may or may not be. Regardless, the capabilities of AIs (not just GPTs) are improving exponentially currently.
> To human level intelligence sure, but it's not obvious to me that it will enable superhuman AI
If you think GPTs can get to human-level intelligence, why would the improvement stop at that arbitrary point?
GPT4 is not an exponential improvement over GPT3. It's is better, but not exponentially so (as in f(x) = e^x).
> If you think GPTs can get to human-level intelligence, why would the improvement stop at that arbitrary point?
Because GPTs are trained on human generated data. They might have some ability to generalize, but to a limited extent. GPT4 is not a super human chess player. It's can play chess, but far from Magnus Carlssen. But we do have super human chess engines, but those are made in a completely different way from GPT.
Suppose GPT-X just dropped and it could both generate text AND play chess better than Magnus Carlssen.
How would that change you position on it being unable to pass the human-level intelligence barrier?