Affordability can still matter for rich people. Isolated rich areas end up being very artificial, which is maybe not the environment they want to live in. Or maybe they're raising families. They might want to be somewhere their children can afford once they become adults without having to rely on their parents.