I suppose it's like calling someone a Neanderthal, but with racial implications. Interestingly, we're learning Neanderthals were not as cognitively lacking as perhaps some supposed.
It's like saying, hey, you are hairy... Right, we're all hairy (bald or not) with very few exceptions. It's a weird thing.
[1] In Indonesian lang. people and apes share the "orang" root. Oran, orangutan, orangnakal, etc.