Well, trans is not really a hard factor for attraction. Fully cis-passing trans women exist, even in intimate relationships, and I'd wager not all things that go into that even matter to most people.
Feel free to exclude based on physical criteria common in trans people, be it from masculine features, to larger frames to a set of genitals you're not attracted to, but "trans people" as category is way to broad. Because they can functionally be cis women. Unless you have a breeding fetish I guess.
Insisting on a separate category feels like people are holding on to bioessentialist feeling that trans women must be men, which would make a cis man gay. Pair that with toxic masculinity and you have one of the reasons for violence particularly against trans women.
(I have no personal stake at this, like many trans people I mostly date other trans people, less complicated and your partner has less of a hard time relating to you)