I've never known anyone that was against only the fluoride. There are a whole range of chemicals that you can find, in your local water quality reports. Many of which will be sitting at the safety limits, if it's anything like my city.
Related, my tap water kills my ferns. I have to used distilled/reverse osmosis water on them, which I also switched to after realizing this.