It's the same critiques that come up with taxing the rich. "Oh people will just evade it". Cool then nail them then. Or if they all want to move to tiny tax shelters then tax them when they move their money there.
They want you to believe it's not possible to restrict them so that you don't try.