And that's why we have driver licences, clean air laws, NRC.
You can't just subject other people to risk because you want to, since it is the community who is going to pay the price, it is the community that decides how to restrict the rights.
(Granted, the process for making that decision is not always the best, but unfortunately this is the only one we have)