You are part of the problem.
You are correct that there's no reason for other people to apologize for him, because he needs to do that himself, if he can be convinced that his sexism is a problem. There is also no need for other people to be defending his sexism.
Here's a hint: a complaint about "annoying people" should aim to be more specific than 50% of the population.