I would say morality and the legal system are entirely separate things and I made no argument about the latter. I merely am invoking the golden rule. Is it fair to hurt someone who has hurt no one?
Because your view is a naive reduction in an attempt to trivialize the issue by claiming nobody was specifically harmed when the matter at hand is the protection of society as a whole.
Protection of society as a whole is usually just special interest in disguise. See drug war, child pornography distribution, copyright extension etc.