This causes a huge break in immersion, where I've been walking around a town/building in a game for 30 minutes and have no idea where I am or how I really got there.
I think that VR is going to be the nail in the coffin for this problem. Awhile ago I tried a VR demo on one of those omni-directional platforms where you literally walk in the direction you want to go, and I noticed that my mind was internally mapping out the scene. As a result, the simulation was -much- more immersive, and I was quite content with just walking around the game world that was set up.
If this tech becomes widespread, I can see it opening up video game niches that have previously been untouched. Simulators for stuff ranging from exploring complex ruins to talking a walk in a forest, to showing home buyers what houses are like without having to drive out to them.