And there could be a teapot in an orbit around the Sun. Do we have any evidence for that being the case though?

So far, there's a distinct lack of "wall" to be seen - and a lot of the proposed "fundamental" limitations of LLMs were discovered to be bogus with interpretability techniques, or surpassed with better scaffolding and better training.