This is true sometimes, but if you are upset at a specific HOA, look at your docs to see what it actually manages. HOAs exist to pay for community obligated costs; sometimes that is a pool, sometimes it's "water and gas bills", and sometimes it's an entire golf course.
Further, if you're talking about a condo building (which, yes, is technically a COA, but the two get combined all the time), it is literally paying for your roof and elevators to work.
Buying into a "little club" and agreeing to pay your share of expenses only works when everyone participates. If not, your neighbors are just paying upkeep on your property; why should they have to do that?