I wholly disagree with the use of that word as being offensive.

That said, you should talk to the person in question if you are uncomfortable so they have a chance to "correct" their behavior when around you. That's the generally accepted best practice.