Is that a recent change? I'm pretty sure some of my less popular remarks earned double digit negative points.
Back when comment scores were public via the API, I retrieved the lowest-rating comments for each month. which was always -4: https://docs.google.com/spreadsheets/d/1IfbSDYVBXiHZCuMdHXgp...
Bonus histogram of comment scores calculated in 2014: https://minimaxir.com/img/hn-comments/distribution_comment_p...
I think the limit probably discourages long winded "all these downvotes are a badge of honor" comments. People who crave downvotes only get four. People hurt by downvotes only get four, too.