> Every X number of poor people in a city, represents a subtraction of an equivalent number of rich people
No such logic like this is required - the more people who live in a city but don't contribute their share of taxes, the higher the taxes are for everyone who does. I'm talking strictly about people who are not net contributors - I think you'd be hard pressed to find people in low-income housing projects who are, the housing alone has significant cost and upkeep of it often falls on the city.
That's not to say it has no benefits of course, bringing people out of poverty can be powerful and useful economically, but if I were to state the opposite is true - that we should be decreasing taxes by building more low income housing, you'd look at me like I'm crazy unless I presented some seriously compelling evidence.