That's the problem. This was recently pointed out by the Globe and Mail investigation into real estate tax evasion in Vancouver.

Many people are claiming a property as their main residence whilst not staying in said property for more than a few weeks a year (main residence for these people is abroad).

The unpopular solution would be to charge capital gains on all property, irrespective of being your primary residence.

That would certainly cool a speculative market but I doubt such a measure would be supported by Canadians.