Sexual harassment? Sure. Sexist? Sorry, no, that's just silly. By your logic, if a male manager only propositions female subordinates, it's sexist, because he isn't also propositioning male subordinates. That's not sexist, that's a side effect of the male manager being heterosexual.
By your logic, every person that only dates people of a particular sex is "sexist", which is completely ridiculous.