If language communicates thoughts, thoughts have a relationship with reality, and that relationship might be true or false or something else.
Then what thought is LLM language communicating, to what reality does it bear a relationship, and what is the truth or falseness of that language?
To me, LLM generated sentences have no truth or false value, they are strings, literally, not thoughts.
Take the simple "user:how much is two plus two? assistant: two plus two is four". It may seem trivial, but how do ascertain that that statement maps to 2+2=4? Do you make a leap of faith or argue that the word plus maps to the adding function? What about is, does it map to equality? Even if they are the same tokens as water is wet (where wet is not water?). Or are we arguing that the truthfulness lies on the embedding interpretation? Where now tokens and strings merely communicate the multidim embedding space, which could be said to be a thought, now we are mapping some of the vectors in that space as true, and some as false?