There are more people born each day (400,000 a day) than there are houses built each day (4,657 a day). So I'd say no, houses are harder to replace than people.
And then there are all the "houses" built from salvaged materials in poor countries. Not sure if you'd want to count those. A person living in such a structure would still be considered homeless in many places.