The tax is based on residency, not ownership. If nobody lives there as their primary residence, it's subject to the tax.
They can also split the property into multiple independently owned properties that just so happen to be right near each other, each just under the limit.