"You have the privilege to drive a personal car, but if you make yourself a danger to others by drinking and driving, the privilege will be revoked" is a more apt comparison.
Exercising your rights is fine until doing so risks the health and safety of others. Other people have rights too.