That can be a very different scenario though - a family using one (or even multiple) properties as investment vehicles that they are themselves residing in (even if its just part of the year) does not seem unethical at all to me. Foreign investors who are not planning to even live on the property nor rent it out at all? Definitely a crapshoot. It is a small overall percentage of the pie but every bit counts. And giant corporations scooping up tons of single family homes and small apartment complexes is maybe even scarier.
Lets face it though - there's a separate problem and it is pretty simple - we just need more housing in places where the demand is high. Personally I think it would be especially helpful in the short term if there were big incentives for individuals to ADU's or inlaw units and things like duplexes, triplexes, or "bungalows" on one property. Easy sell for the public and would open up all of the single family home zones to much more housing and maybe even incentivize families across generations to live together again.