On the other hand, if companies (and individuals) were not able to shop around for the tax regimes they prefer, there would be nothing to prevent extortionate taxation -- companies and people wouldn't be able to avoid it by going somewhere else, and governments would be able to do whatever they wanted.
On balance, I prefer competition in the regulatory and taxation space over the alternative. It limits the ability of governments to use taxation as a weapon or a form of expropriation. I'd rather live in a world where governments need to attract investment rather than coerce it.
Some might say that such competition would result in a race to the bottom, but we already have that competition and taxes are still quite high in many places -- a race to the bottom would have happened by now if it was going to happen.