VAT is a consumption tax. If you consume goods in a place with lower consumption taxes then the tax is doing exactly what it is meant to do.
If you buy consumer goods in a low-VAT place but transport them to a high-VAT place and consume them, then that is clearly illegitimate for the same reason that it is legitimate for VAT to be applied to imports and refunded to exporters.