People are harder to replace than houses.
It would be more accurate to say 'People are more emotionally attached to people than houses'.
And then there are all the "houses" built from salvaged materials in poor countries. Not sure if you'd want to count those. A person living in such a structure would still be considered homeless in many places.