If his intent wasn’t sexist then his failure to perfectly negotiate the complex social way those points have to be expressed isn’t a moral failing on his part.
Edit: I’d rather posters took the time to argue against rather than downvote. My point is just that ‘socially colourblind’ people probably shouldn’t be punished for lacking perceptual and expressive subtlety. It doesn’t mean they can’t be corrected or educated, of-course.