For example, I consider "heterosexual" and "homosexual" mostly as categorization words, as they tell you almost nothing about a person.
On the other hand, if we used words like "penis-loving" and "vagina-loving", which are descriptions, we'd get along much better. I'm pretty sure I have more in common with vagina-loving women (i.e. bisexual women and lesbians) than with only-penis-loving men (gay men).
I think that would just be confusing, as you're conflating very different things. I'm gay in terms of who I'm actually attracted to, but I find both penises and vaginas sexually interesting. Similarly, gay men aren't necessarily particularly into penises and straight men aren't necessarily particularly into vaginas. (With regard to the latter, witness all the nasty misogynistic things straight men are liable to say about them.)
I'd also point out that as a gay man I'm far more attracted to transguys, who generally have vaginas, than I am to transwomen, who generally have penises.
So in short, describing my sexuality in terms of which genitalia get me going would be very misleading, and I'm not an isolated case.
How many of either do you know and / or call as friends?