it largely does not. in the high-dollar markets like vancouver and toronto the cost to purchase has little to no relation to the actual houses-to-live-in market, and the cost to own is many times higher than the cost to rent because the housing market is purely a speculative financial market.

many of the properties are left empty for extended periods of time because the owners value liquidity over rental income, and having a renter makes it more difficult to sell quickly when market conditions change (or if you simply want to shift your portfolio).

A lot of these properties apparently are bought by wealthy foreigners purely speculatively, and sit vacant. If they were buying them to rent them out, I'd agree with you, but as it is, they end up reducing the housing stock that's available to be lived in while the number of people stays the same, so they do drive up price.