I have noticed this too and have a humble suggestion: down-votes should require a comment from the down-voter. This would have the following advantages:
- The extra effort will slow down causal unthoughtful down-votes.
- If there is no justification, the down-voter can be down-voted in turn presumably by a greater degree.
- Any pattern of abuse can easily be seen in user profiles.