Surely not, surely it's the reason for the rudeness - if the reason is based [at all] on the sex of the person one is being rude to then it can veritably be called sexism. Otherwise it's just being rude to another person.
There clearly isn't any easy way to measure "aggregate rudeness" to be able to prove that women receive a higher quantity of "just being rude", but that doesn't necessarily mean it isn't there.
I wasn't suggesting that it was a necessarily conscious thing either; just that it either is or isn't sexist based on other things than quantity. Aggregate rudeness won't even do, if you happen to see a woman you don't get on with every day and are rude to them then your aggregate rudeness towards women could be huge without you being sexist; equally you could hate and avoid women and so have a very low aggregate rudeness towards them.
It's like the people who get arrested while walking home from work because they happened to enter a protest area. The reason for their being there is totally unrelated to the protest yet the effect of their being there is to increase the size of the crowd by one.
We can argue until we're blue in the face that this is unjust and the truth is that it is: it's unjust for everyone involved. It creates further entrenchment and higher barriers between people.