Let's apply that same reasoning on a different scale.

Rome violently occupied most of Europe and forced its culture on to those peoples. Therefore, most Europeans may eat as much pizza as they like. But if an Italian so much as looks at a croissant or wears pants, they've done A Cultural Appropriation.

Seems sensible to me!