Even if that were true, that's not an argument to not build. People need houses, the pricing is reflecting a shortage, denying house is denying people housing and there's no way to spin that reality.
So what you'll be left with is even more wealthy folks coming in, and never anything affordable for the blue collar folks.