The point, I argue, is not just the leaving aspect, which is obviously untrue. It's the fact that without taxing them, they tend to meddle in politics and turn a reasonably good society into a social and political chaos. By taxing them we, the society, exert a form of very necessary political control whilst also funding essential services that make life liveable for the individuals.