And as an infant our world was incredibly smaller than our world is now. Surely the 'principal components' of that day were reshaped or subsumed by other pointers later, thereby shifting the landmarks used to retrieve memories.

I would hazard a guess that the original pointers aren't revised to point nowhere, but are, if possible, moved to point into the revised reference model. (Presumably we do this now when we learn a new domain like music, and as the model for harmony becomes more concrete, our original pointers to elements within it are adjusted rather than lost.)