There are many other medically valid reasons to not take a vaccine. There are even some (a very small number) of people whose religious convictions are a valid reason to get an exemption.
The issue is that the overwhelming majority (>99%) of people who choose not to vaccinate, do so for bad reasons, which amounts to making an uninformed medical decision, and one that adversely impacts other people in addition to themselves. These people are what the controversy is actually about.
Anyone bullying people who have medical issues is a bad person and needs to chill.