The main issue is forcing that protection down the throat of whomever doesn’t want to be protected. Or in other words, believing that the government knows better than you on how you should live your life.
It's not that "the government" knows better than you how you should live your life, but it sets the ground rules that we all must adhere to in order to participate in society.
And for better or worse, because our society dictates that we won't simply leave crash victims who weren't wearing a seat belt to die in the median of the highway (whether they want to be left or not) we mandate that everyone wears a seat belt.