Locally, when I see people complain about luxury housing being built, it's because the developer is building high-spec townhomes and selling them for $800k+. Instead of lower-spec townhomes at $500k. The developer can't fit appreciably more homes in the same footprint (they're all 3-4 stories tall here - reducing size likely means one fewer stories). And the market will bear $800k+, so that's what we get.
The real answer is build even more homes. Or allow high-rises on the same lots.
In theory, the family that buys the $800k townhome came from somewhere. The smaller house they vacated is now open for somebody else (who in turn moved out of a studio flat that's now open, and on down).