2+2=4 is arbitrary. Humans decided base 10 makes sense because we have ten fingers. It's arbitrary in the sense that it's something we picked out of convenience. Base 10 isn't some real thing existing in reality that you can touch or sense. Language is also arbitrary. Humans came up with it! Wasn't like Proto-Indo-European was something objective and real that we discovered. We settled on it out of convenience and necessity.
If someone is allergic to peanuts, they could definitely say it's poison. Categories aren't objective.