There is social stigma. You'll be seen as petty and somebody that micromanages unimportant parts of life.
If you're going to spend time on Facebook at all, it's worth the time every now and then to make the time better spent. They even have a page called "edit friends" where you can just go down your friends list ticking the X button to unfriend them.