Why is it a problem when poor people want to move in but not rich people?
Does some homeless dude from Boise degrade Austin any more or less than some programmer from SF? Both are seeking what they think is a going to be a better life. Both are ever so slightly changing the character of their destination by showing up.