Like fine, let's put aside somewhat intentional things like hinting and grid-fitting with accumulating error for a minute. Some FreeType configurations dramatically fuck up the visual weight of fonts, making the regular style in a type face look fairly bold. The damn font looks wrong. It's not "wrong" as in I disagree with what the designers intended for the type face, it's wrong as in it looks nothing like the designers intended and it looks nothing like the parameters you put in to render the font. There is basically no perspective where this output is desired, it's just a bad rendering.
There's definitely a bit of subjectivity in exactly where to draw the line, but there is definitely still a line you can cross that just goes into blatantly wrong territory. The relative visual weight of a glyph is not supposed to be influenced by its size on screen.
But OK, other than just being incorrect, does it matter? Many people don't have proper color management in their software and it's usually fine. Well, yes, sometimes it matters. For one thing, this issue really screwed up scaling in Win32 and even GTK+2, because if you tried to render dialogs with different font sizes it would completely change the UI and screw up some of the component sizing. OK, though, you can fix that by just not using a fixed layout. However, you still run into this problem if you want to render something that actually does have a specific layout. The most obvious example of how this can be a serious problem is something like Microsoft Word that is meant to give you a WYSIWYG view of a document on paper, but the paper is 300+ DPI and the poor screen is only 96 DPI.
Maybe most importantly, this is all pointless! We don't actually have to settle for these concessions for Latin script text on 96 DPI screens. Seriously, we really don't. I recommend this (old) article for a dive into the problems caused by non-scalable font rendering and how it could've probably been solved all along:
https://agg.sourceforge.net/antigrain.com/research/font_rast...
(Though to be fair, there are still problems with the approach of vertical-only hinting, as it does cause distortion too.)