There's no shortage of builders. You can just compensate those workers with taxes. If you think taxes are also a form of slavery, that's a different discussion.
Providing housing most of the time is just a program. Providing housing all of the time with no exceptions, like you would expect from a human right, involves some serious distortions of other values. That's the problem, people are using the word human right when they mean high priority. It's a high priority that I do my laundry, but if society were organized so that I did my laundry at all costs, if we were all convinced that a society where I did not do my laundry did not deserve to exist, then things would get pretty crazy.
This isn't some paperclip maximalization requirement you adhere to the social contract, and under the wrong conditions will force people into house-building labor camps. Human rights are how you reason about resource allocation. You have a right to a trial by a jury of your peers, but that obviously doesn't mean a judge must preside over your case that same day by threat of force. We can recognize physical realities, while still understanding that if we wanted to, we could provide a social safety net that includes a place to sleep.
> If your country ran out of money, if the builders all started working for foreign companies
If your country runs out of money, many human rights are probably in danger.