If you make your site completely unusable for 30% of people, no matter how many try to use it, they'll end up making up a miniscule portion of your user base, which you'll then use as justification for ignoring them.
Accessibility is a different issue of course, but to illustrate my point: I have never translated my websites to Chinese, making them unusable for 1/7th of the world population. I don't think it's necessarily a failure on my part.
You didn’t intend it for a blind audience, but they’re certainly capable of reading the text. Did you avoid making the text actively toxic to screen readers?
Many of the whiz bang features on new sites, ostensibly because “users want features!”, hinder even this functionality. It’s fair to criticize this as a regression.
I use a browser extension that lets me click links and buttons from the keyboard. The more bleeding edge they make the sites, the less I can use this feature. That’s not an improvement.
The issue is that so many projects have not been designed with these ideas from the outset, that it indeed usually does take a mammoth amount of re-engineering and dev time to make these kinds of changes and optimizations after the fact.
To continue with the mediocre carpentry analogies floating around: a (competent) carpenter doesn't build a sturdy table by first building a shitty table and reinforcing it later.