Correct, workers do. Unions don't. They are fundamentally parasitic, which isn't necessarily a value judgement.
So fighting against unions is the same as fighting against people's right to associate since any group can have the same shortcomings unions do.