Possible fix: Put the up-vote graphic to the left side of a comment (e.g. where it is now), and place the down-vote graphic to the right side (spaced a little bit away from the "link"). That makes it pretty hard to accidentally press one when you mean the other.
EDIT (example): http://dave-gallagher.net/pics/hnVote.gif