Would you consider it possible that the way non-intentionally placed items break the game immersion for you is because they appear in such a way that you think you can interact with them in a certain way, but you can't?
Like if there's an extra door in the house you're trying to get into, but that door doesn't really open, then in your mind that breaks the integrity of the game's systems. If so, I think the LLM response is that there are no more doors that don't open and that the world can be generated as needed.
No computer can handle the complexity of even a small town. But it would be possible, at least in the future, to generate the part of the world you interact with, which would heighten the emersion.