I think this is by design - Downvoted comments clearly show community disagreement, but one can still read them with minimal effort.
Not to mention that so long as they're displayed on the page, someone will come up with a userscript to display them.