I don't think the elephant can be solved by a tweak to LLMs. Producing a statistically-likely continuation of a pattern is what they do; there is no encoding of the world, just an encoding of language and image data.

A general crossing of that gap is, dare I say it, a problem requiring real intelligence.