This is actually a very deep question.
To oversimplify, yes, the difference is that you've consented to the rules. Imagine you and your friends decide to order a pizza and agree to vote on the toppings. That's fine. But it's not fine for a friend to force you to pay for the pizza when you didn't agree to do so.