If you have an extremely simple theory that debunks the status quo, it is safer to assume there is something wrong with your theory, than to assume you are on to something that no one else figured out.

You are implicitly assuming that no statistical model acting on next-token prediction can, conditional on context, replicate all of the outputs that a human would give. This is a provably false claim, mathematically speaking, as human output under these assumptions would satisfy the conditions of Kolmogorov existence.