I mean, it is because you can look for another job or start your own business if that's the hill you want to die on.

But in general society has rules and we must abide by them if we want to participate. For instance I'm sure most people wouldn't appreciate if I walked around naked, which does not harm anyone unlike needlessly having an increased risk of contracting and/or spreading COVID.