Most importantly, the fact that the word is offensive to someone does not invalidate the racist double-standard. That's what context means: that the use of a word or phrase has multiple properties. On the one hand, use of the word by a white person is offensive. On the other hand, saying a white person can not use that word is racist. Both of these things are true at the same time.
You can attempt to gloss over these facts by saying "oh, historical use" or "oh, it's offensive" and many other reasons, but that does not remove the racism inherent to the double standard which is based on the color of the skin of the speaker. If the word was not offensive, it would still be a racist double standard.