Why can't it be both?
It would be like saying a buffer overflow is sexist, or ctrl-alt-delete is sexist.
And calling the act sexist would serve to shame an activity that we all engage in due to primary biological programming.
Compare it to the discouragement we have to "slut shaming". If tit staring is basic human programming, something wired in to our brains, what are the benefits of shaming that activity? And if it is just an neurological tick we all have, is it accurate to call it a sexist behavior that presumably men and women can be weaned off of?
I occasionally feel compelled to beat the living shit out of someone. This is probably an evolutionary impulse. Those who cross me in the jungle must be eliminated before they eliminate me, etc. But do I go around beating the hell out of people? Not a chance, absolutely never. Why? Because I don't want to live in a society where people solve differences with violence, and I don't feel violence on my part is necessary.
That being said, I definitely "notice" attractive individuals of the opposite sex. I don't harass, stalk, or assault them in any way, but I do notice them. I don't believe this is sexist. However, going up on a stage in front of people and making a joke about "staring at tits" might very well be construed as sexist because it goes well beyond the act of "noticing", and involves a mixed audience who may not all appreciate the concept.
If it's an unconscious tick over and done with, much like a blink of an eye than it's not comparable to your beating someone up which would take this hypothetical compulsion of yours and render it into an act spanning minutes.
"We are not evolutionary automatons, we can override our programming at will."
Let me know how successful you are to not blinking, or not having goosebumps, or not feeling ticklish, not having a reflex, not being averse to certain odors, ...