when you submit a comment to a thread, other people can vote your comment up or down. when you're voted up, your score goes up, and it goes down with a vote down. your score can also go up based on your submissions, which can't be voted down.

based on your comment history (http://news.ycombinator.com/threads?id=sscheper), you made some remarks that people didn't like, so they voted you down.