Homo is the diminutive of 'homosexual'. A diminiutive of something inherits its qualities. Saying that 'homo' is derogatory is saying that 'homosexual' is derogatory.
When you say in French "Il est homo", people don't take it the wrong way. But it would be really weird if you say "Il est homosexuel".
I feel the people it pisses off the most are the White Knights who go to great length to prove how open minded they are, to the point of jumping right to the jugular :).
And I don't "know gay people", some of my friends are gay. It's not "knowing" in the distance like "I know Hillary Clinton".
And take a stand as you want. As long as people don't see how ironic, really, being intolerant towards homophobic for their..well, intolerance, this will continue to be an issue.
"Either you think the way we do, or you're a bigot".