The issue is the SROs, which are legally protected as housing for the semi-homeless/insane.
I don't think there's an inherent right/need for people to live in a specific area of the city. I'd far prefer to see the land used for market-rate developments and then some of the profits taken as fees to fund programs to de-fuck these people somewhere else.