The question is whether it's going to be new construction that they occupy or existing construction. If you're not well-off you'd want that decision to end up with "new construction" so you can move into "existing construction" at a lower rent/mortgage than if new construction didn't exist.
I am in favor of building new housing, but I'm even more in favor of reducing wealth inequality. I think we can do both, but we need to be deliberate about it.
The greatest tool we have to reduce wealth inequality is make it so people can buy homes - and the biggest levers we seem to have there are making supply available in general, and making jobs available where there's already supply.
Which is exactly what people are afraid of happening, and what they mean by the crisis of housing affordability.
You can go and buy or rent a cheap house today in probably 90% of the localities of the USA. Of course, if you already live in those places, you probably don't have the money, because good work is very hard to find.
The problem of housing is that regular people want to live in NYC, and Chicago, and LA, and all of these places, and relatively near to where their jobs are, and they're seeing the rich own more and more of the space that could have allowed them to do so.
It's not clear in your phrasing, but it sounds like this is a casual correlation. It's not... Many people only want to live in these cities because it's the ONLY way to get work, given a choice they would quickly move out.
... And that's why nothing gets done.
That's the point, the "dumpy old apartment" was never going to be the thing stopping the rich person from moving in.
Due to overlap in preferences amongst a multitude of people, the rich person would free up the housing they like by buying it from whoever, who would then free up slightly-less-good housing in the area by kicking out someone else (with money, to be clear) and so on down the chain until you get to the "dumpy old apartment", whose rent now rises because there are more people interested in moving into it due to lack of other options.
New housing, even if it's new 'luxury' housing, breaks this chain of housing migrations in the area before it gets to buying people out of their dumpy old apartments.