I suspect you’re making assumptions that don’t hold up to scrutiny.
You appear to be defaulting to the assumption that LLMs and humans have comparable thought processes. I don't think it's on me to provide evidence to the contrary but rather on you to provide evidence for such a seemingly extraordinary position.
For an example of a difference, consider that inserting arbitrary placeholder tokens into the output stream improves the quality of the final result. I don't know about you but if I simply repeat "banana banana banana" to myself my output quality doesn't magically increase.
You're the one who raised it. Perhaps you should clarify what you mean by "isn't real" - do you believe a human narrating their thought process is saying something that's more real?
Someone else replied to your comment asking essentially the same question, perhaps better phrased:
> What would be different if it was "real"? What makes you think that when humans "narrate" "their" "internal thought process", it's any more "real"?
What do I mean by isn't real? Exactly what I said originally. It's a roleplay of something that sounds plausible as opposed to what actually happened. There is obviously some process that is producing the output. The thinking trace is not a representation of that underlying process. Rather the thinking trace is an adjacent output of that same process.
LLMs don't need language to do mental tasks, either. Their input and output is language - like humans - but in between, the high-dimensional vector representations (often loosely called latent space) are not language in any meaningful sense.
LLMs can benefit from "thinking out loud" much as humans can. The issue is not whether the supposed "thoughts" are actually representative on any "internal" thoughts, but rather that explicating the problem in more detail can help reach better conclusions.
One point I was making is that the idea that humans are doing something "special" (or in the OP comment's terms, "real") in this area isn't well-supported, in fact there's plenty of evidence against it.
The two processes aren't equivalent. An LLM that fills the thinking trace with a meaningless placeholder token will still exhibit improved performance. There are also regularly things in the thinking trace that don't match the final output if you look closely but on the surface they appear convincing.
It's largely a trained performance. If you go in with the erroneous expectation that it accurately reflects the underlying thought process then you're likely to come away with faulty conclusions.
That's a loose analogy but it fails to fully illustrate the degree of decoupling here. For example the weirdness of LLM performance being increased via the output of empty sequences.
Is this meant to be an analogy for a human or an LLM? Where would it be different in the other case?