Unfortunately, whether or not a term is really offensive is a combination of what it is, who said it, and when/where (at least in the common-sense definition). Unfortunately, because this is directly opposed to our (at least in the US, and in most countries rooted in liberalism) sense of fairness which says that rules should be applicable universally, across all people and in all contexts.
Which is to say… policing speech is a problem best avoided!