You can't have a freedom that's conditional on somebody else's forced labour.
And if your answer is to take all the existing houses and give them, one to each family that needs a place to stay, that's great in the short term. In the long term, though, who's going to build a new one when it will just be taken from them? And if nobody builds new ones, where are new ones going to come from, as the population continues to grow?