I lived by the beach outside Silicon Valley. My neighborhood gradually became more and more deserted. Executives would buy houses in the neighborhood, and just stop by once or twice a year to use it. At some houses I never saw anyone ever - over many years.
I think the real issue is that we let market forces determine housing prices and housing strategy. Housing is, like medicine, something that is poorly served by markets - since markets distort basic resources away from the common good.