Society as a whole agrees that taxes are generally a good idea. In today's complicated democratic systems it may be harder and more tedious to prove how exactly, but the reason taxes are levied is that society does not want to change this.
Imagine you are in a bar with friends, and they agreed, that you should pay the bill. Can they collect your money by force? Clearly the answer is no.
To your example, specifically, if you and your friends had an explicit system of law governing your night on the town, then yes, they could agree that you should pay the bill and collect the money by force.
In modern states, you don't have a lot of choices either. But at least, even though you are "forced" into a society, you eventually get the right to participate politically.