People also care if you are not well groomed, wear clothing with holes, are sitting in public for too long. It just so happens that the two correlate a lot.

> going to steal,

Think about the thought process that leads someone to believe you are "going to steal", and how much prejudicial stereotyping that entails.

> unshowered, drunk/high, schizophrenic/screaming,

All things that people who have houses are free to do without being punished. So yeah, anybody does care if you have a house, practically speaking.