Gender, race, sex preferences, or political beliefs are very different from being anti-vax.
Being anti-vax is evidence of low intelligence. So no, I would not want someone with low intelligence to handle or assist my baby's delivery.
But the most obvious reason is that they directly endanger mothers and babies in the delivery ward because unvaccinated people are more contagious and carry higher viral loads.