The datasets behemoth LLMs are trained on include a lot of noise that derail progress. They also just contain a lot of irrelevant knowledge that the LLM has to learn or memorize so an obscene amount of parameters is required.
When you're not trying to teach a language model the sum total of human knowledge and you provide a high quality curated dataset, the scale barrier is much lower.
But LLMs do gain useful "emergent" properties from training on massive lower quality datasets.
It's just that given what we know about neural networks, it's often easier and simpler and more effective to increase the amount of training data than to change anything else.
If someone knew all the math and science in Wikipedia, for example, I think they'd probably be forgiven for not knowing every regionalism.
And if you want it to be superhuman then you're by definition not capable of knowing what's important, I guess.
But that looks like a 'smeared' out probability distribution on the next token. Not like the text produced by an unsure human.
Nevertheless, human feats can act as an existence proof of what is possible. Including of what might be possible for a neural network.
(I'm not sure whether a large language model necessarily needs to be a neural network in the sense of a bunch of linear transformations interleaved with some simple non-linear activation functions. But for the sake of strengthening your argument, let's assume that we are assuming this restrictive definition of LLM.)