HOA's can be formed based on a premise that democratic procedures govern the creation/modification of new rules.
To oversimplify, yes, the difference is that you've consented to the rules. Imagine you and your friends decide to order a pizza and agree to vote on the toppings. That's fine. But it's not fine for a friend to force you to pay for the pizza when you didn't agree to do so.
Your friend consented to the first pizza, but he didn't consent to any pizza after, except insofar as you can say that his remaining a member of the pizza-owners' association represents actual consent.