Since when did we restrict people's ability to do things?
This question is not the obvious winner you think it is. To me, and I am sure many, it sort of undermines your argument.
Even in the most ‘free' cultures, society has _always_ restricted people’s individual ability to do things that it collectively deems harmful to the whole society.