There is a saying in French, "La liberté des uns s'arrête là où commencent celle des autres".
I'm not sure if it's known in English, I never read it online that's for sure, so I just roughly translated word by word :
The freedom of someone ends where the freedom of others begins.
Basically saying that living in a stable society is like signing a contract to restrain your liberty of doing anything when it's proved harmful for the others.