It may not be a zero sum game, but it at least approximates/approaches zero sum. How exactly would you change downtown Manhattan such that anyone & everyone who wanted to live there could?
I'm not arguing against development, I just think a lot of the pro-development folks imagine that cities are making things zero sum with their policies, when real property will always have that element.
That's not what we should be striving for. We should be striving to allow people to comfortably live and work outside of just downtown Manhattan. The biggest obstacles to that are zoning laws and a decrepit subway system that makes traveling between boroughs outside of Manhattan practically impossible.
Better public transportation would also relieve pressure on the city center.