Yes! Why should a web site specify fonts at all? The user should set their browser to a default that they like and most sites should be happy with that. People just use web fonts because they can. If there's a particular reason you need to override the users preference then fine.
If this was actually a decent part of a browser's configuration UI, it might have caught on (or... still might). Give people tools to manage this easily, and they may choose to do it - the 'reader mode' in some browsers shows some people want to get at content regardless of fonts, and giving them easier and more 'up-front' ways of managing this would bring the issue front and center.
Why did we end up with legislation about cookies? I think, in part, it's because it was essentially black magic for years, hidden in layers of nested "advanced" menus, instead of a button right in the damn menu bar saying "clear cookies".
"Why doesn't anyone use personal certificates?!" (I hear from tech colleagues now and then) - because the UI/management for it is horribly useless for 99% of people.
Same with this font stuff. Put "font choice" front and center, as part of browser setup - don't hide it behind layers of "accessibility" stuff. Give people some standard themes they can switch between easily - high contrast, B&W, etc - and have it be something that's a button press away (harder to do on mobile, I'd think).
Web fonts are better than the stuff people did to do this in the old days. Crazy abominations like siFR [1] or rendering out a gif of your text and using that instead of just delivering some ASCII text that screenreaders can deal with? Web fonts aren't perfect, especially on slow connections, but they're the least horrible solution we have for people who would like to control the finer aesthetic details of how their sites look.
Because design happens to be one avenue of human expression. If you keep denying that, or label users enjoying it as superficial, you'll end up with something like North Korea, where everyone gets to choose from two approved hairstyles, walks around in "male shoe" or "female shoe" and goes home to eat soylent.
(Many people in this thread have said they do not value consistent layout. Nor do I, but that is the usual reason.)
Less so nowadays, where HTML is used for layouts closer to say, PDFs.