Kind of both. Community notes and moderation are both emotional humans making the best decisions they can.
There is no “fix”. There’s no magic. It’s just us in here.
Notes usually go too far in the other direction, for instance you will rarely see notes on mainstream politicians even when they are speaking objective falsehoods. This is because that system requires an agreement from people who usually disagree for a note to show up.