Now, I'm sure some men (or women) can successfully use that word to describe their colleagues and still work productively with them, but quite a few men can't. For an egregious example, see this profile about Sheryl Sandberg: http://www.newyorker.com/reporting/2011/07/11/110711fa_fact_...
"Dina Kaplan, the co-founder of Blip.tv, says that when she met with angel investors to raise funds she dressed nicely, and in a meeting with a potential funder he told her, “Here’s what we do, Dina. We’re going to spend half the meeting with you pitching me, and half the meeting with me hitting on you!” “I felt nauseous,” she says. “I tried to laugh it off. I asked, ‘Of all the things you’re working on, what most excites you?’ He said, ‘Seeing you naked tonight.’"
When you've got men like that out there describing their female coworkers as sexy, you're going to have to work pretty hard to convince women not to lump you in with them. It's unfair, but that's how stereotypes work.