Well the big thing is that most people can't just move if they want to. They have their jobs, and (especially if they're older) it might be really hard for them to get an equivalent job elsewhere. So moving will already cost them money.
The second part of this is that it costs a lot of money for these people to move/be without income. So for some, they might just be too poor to move, and suffer with diminishing disposable income as the price of everything else around them rises.
I think the core of the argument is that it's not like these rich people came into a barren land and built giant apartments. There are many people in NYC that simply live there, and why let these billionaires come in and make everything more expensive?
I think that's the crux of the counter-argument. I think that a combination of proper tax brackets, loose zoning regulation allowing for high-density housing + commercial mixes, and primary-residence rules can solve the problem.
I think the central moral argument is that having giant empty houses in the middle of a city with a community trying to do otherwise isn't exactly useful for society as a whole (see Detroit) , so we should disincentive it as much as possible.