In radical feminism (the ideology that the term "feminism" most frequently refers to in contemporary usage), virtually the entirety of human interaction is viewed from the lens of the patriarchy model, the notion that society is governed by "a system of power that organizes society into a complex of relationships based on the assertion that male supremacy oppresses women" (Wikipedia). Such feminists define any act as "sexist" if it reinforces, or even if it is the product of, what they see as the patriarchy. Similarly, liberal sociologists and critical race theorists define "racism" as the characteristic of any action that results from the oppressive power the dominant group (most contiguous with the class of straight white males, at least in Western society) has over minorities. (This is why those on the far left often make statements like "there's no such thing as reverse racism/sexism." They're right based on their usage--it's just a matter of definition.)
So anyway, when feminists use the term "sexist", they're not usually using it in the colloquial sense of "you're a bad person" but rather in the sense of "you're saying or doing something that results from the patriarchal society that governed your upbringing, and it would be really good if you start questioning the assumptions that led you to this action." In fact, I've heard quite a few feminists describe an action of their own doing as "sexist" and immediately set out to unlearn whatever caused them to say or do that. If only more feminists were as open to entertaining discussion regarding alternatives to their patriarchy hypothesis, but I digress!