I realize that landlords have been vilified in the past, but who is really benefited by suppressing the mobility of the population? It's businesses (whose workers can't leave), it's politicians (who don't have to worry as much about people packing up and leaving in response to bad policies, corruption, etc.), and it's banks -- most people pay at least the value of their home in mortgage interest, and are effectively enslaved to the bank, an unrelenting landlord who can inflict the equivalent of modern-day debtors prison in the form of foreclosure credit destruction and confiscation of any equity earned since purchase.
How many people are tremendously afraid of losing their job because they might lose their house (their biggest investment)? How many people put up with unfulfilling, soul-sucking work just to make that house payment?
Consider how this has all changed -- how many more debt slaves were created -- when the housing market tanked. Home ownership is not the American dream, nor is being part of a community for the long term, nor is having the same job for 30 years and then retiring with a gold watch and a pension.
True freedom is in being able to take a better offer and being able to reinvent one's self whenever one wants. But our entire financial system, it seems, was based on the idea of encouraging long-term debt (mortgages) and the stasis that accompanies it.
So I think the ideal that you are striving for with not uprooting people from their rental apartments is not only illusory but destructive. Price changes contain information, and preventing price changes simply hides information, keeps people in the dark, etc.
Maybe being uprooted will (metaphorically) allow the plant to catch flight and land somewhere better... Maybe rental real-estate prices are so high because tons of rich people are moving in and they all plan on sending their kids to private school, leading to the destruction of the public schools -- this is the case in SF where most people with money would never send their kids to SF public schools even though it costs $2K/month to rent a dumpy 1BR apartment and so objectively it's a "rich" area.
But if you agree on the major point that the social engineering geared at getting people to live/work in one place for a long time is on the wrong track, then our differences seem focused on rent control.
I should mention that in my last building in SF there was a guy who had lived there in a rent controlled apartment since the 1970s. After the first year I lived there, a letter arrived in the mail informing me that rent would increase by exactly 1.5%, the maximum increase allowed by law, and that the landlord was pushing for "utility cost pass-through", a rent-control exception that allows slightly bigger increases in cases of utility price jumps. So I was a beneficiary of rent control.
Back to the guy. His apartment had a view of the bay and golden gate bridge from every window, including the bathroom. It could probably be rented for $3500+/month because of the amazing views. He was a wealthy lawyer, arch-enemy of the landlord and heavily involved in the tenants union, and completely not in need of any help paying rent. With rent control he was probably paying under $1000/month for his apartment and had been for years. He will probably live there until the day he dies.
I'd have been far less skeptical of rent control if there had been an immigrant family living in that apartment or someone who worked at the local Walgreens.
Why does the debate always veer back to renovation of the buildings? Do you really think that adding stainless-steel appliances to an apartment is a more desirable goal than not displacing established residents?
As for your personal experience with the wealthy lawyer, here's the balancing anecdote: I live in the Mission, a part of town where rents are rapidly increasing (I'm sure I'm guilty of driving up prices). There are plenty of people down here who can only afford to stay in their homes because of rent control, and stories of tenant intimidation are plentiful.
In the mission, I would prefer if a subsidy was just given to the poor via tax dollars. Say it's a $200 per household subsidy. Maybe it makes sense for that family to pay it to the landlord, but maybe it makes sense to find a cheaper apartment and spend it on a TV. If it turns out that the TV was the chosen decision, that helps drive down the price of rent b/c maybe the landlord will have a slightly harder time raising rent prices. If, on the other hand, the tenant chooses to allocate the money to housing, it will help drive up housing prices. The upside of driving up housing prices is that it now becomes more worthwhile for someone to convert their house into apartments, which in turn increases supply and would tend to help keep prices affordable.
Direct payment like you're advocating is a political non-starter. As paternalistic as it is, taxpayers don't like the idea that their money could be spent on anything, they want it to go only to the things they approve of.
I agree that the public would rather "force evil landlords to do x" than achieve the social goal in a minimally economically distortionary way. It similar to how politicians give handouts to people who drive hybrid cars while at the same time fighting a hugely expensive war to prevent oil prices from increasing to the point where people would just buy electric cars because they were cheaper!