I didn’t say it was random behavior. You did when you said it was a happy coincidence.

I know it is just a language model. I know that if you took the same model and trained it on some other corpus that it would produce different results.

But it wasn’t so it doesn’t have enough data to say that bananas are larger than the Empire State Building, not that it would really matter anyways.

One important part of this story that you’re missing is that even if there were no texts about bananas and skyscrapers that the model could infer a relationship between those based on the massive amounts of other size comparisons. It is comparing everything to everything else.

See the Norvig-Chomsky debate for a concrete example of how a language model can creat sentences that have never existed.