How many users apply their own style sheets? I remember this was popular among a relatively few people, and now I don't know anyone who uses things like Stylish or Greasemonkey/Tampermonkey anymore.
It's probably worth noting that reader-mode -- another accessibility tech -- does seem to be fairly popular.
I wish more developers had your perspective.
My Dad uses Windows' high contrast mode. It's less that "literal dozens" use this feature and more that this feature transforms all that desktop-shaped-material into a computer for the vision impaired.
Isn't the biggest performance enhancement the transition from not-working to working? I view accessibility as the ultimate performance enhancement for a given slice of users.
Now, if those 0.1% of users have some sort of legitimate accessibility issue, we can talk about it. If they just have highly idiosyncratic preferences about their user experience, the axe it shall be.
Moreover, whatever the portion of users who currently have accessibility issues is, the portion who will have some is much much larger.
Finally: human accessibility frequently seems to dovetail with machine accessibility. Engineering done with accessibility in mind seems to be less likely to result in silos and more likely to present an interface more amenable to automated interaction.
It's still a question of whether a significant amount of people use it within the affected group. If you need to enlarge or increase contrast, there are standard accessibility tools for that which don't rely on HTML at all.
I have also written various user styles for myself for particularly bad sites, over time, though I only have one such installed at present (on my phone, actually, to turn a poorly-implemented dark mode into a more complete and more actually-black dark mode). The progressive diminution has occurred mostly due to me ceasing to use bad sites.
I have more user styles on my browser, doing things like hiding the tab bar in favour of the Tree Style Tab extension, and the sidebar’s title.