You think taking what little money poor people have from them via allowing them to spend it on lottery does not have social consequences ?
If a person or group of people does not act in a way that benefits you, they still have not harmed you. It simply means they have not chosen to conform to your expectations.
On one side you have the rights of the individual and on the other side you have the rights of the community.
Some people tend to lean the scale toward individual rights and others toward community rights.
Smaller government tends to handle the balance by staying out of the way until it's necessary that they step in. Bigger government tends to handle the balance by marking clear (or blurry) lines and moving them when necessary.