One thing that has kept me from ever returning to MIT (where I did my PhD) is that I don't think I could bear the feeling of loss that I am sure would overwhelm me when confronted with the sights and smells and memories of that period of my life that I can never go back to live again. I have lived in quite a few different places during my life, and have pleasant memories from all of them, but my time at MIT feels completely different - it's not just memories, but it feels much more personal, as if it's not just a place but represents who I really am. I have often wondered whether everyone feels like that about their PhD years, but your comments seems to suggest that it is not universal.