Why don't you consider its training set (usually the entire internet, basically) worldly facts? It's true that the training set can contain contradictory facts, but usually an LLM can recognize these contradictions and provide analysis of the different viewpoints. I don't see how this is much different from what humans can do with documents.
The difference is that humans can do their own experiments and observations in the real world to verify or dismiss things they read. Providing an LLM with tools can, in a limited way, allow an LLM to do the same.
Ultimately its knowledge is limited by its training set and the 'external' observations it can make, but this is true of all agents, no?