The Map Is Not the Territory. Your memory of the mental model of our local neighbors that you'd worked out is overriding the actual memories.
Something similar happens when we remember childhood houses. Spaces where there was a lot of activity seem bigger, because we've mapped those areas extensively.