I still don't see any connection.

Lots of what I would call "victimless crimes" are outlawed because of what the proponents of criminalization see as focused harm. Prostitution, for example, is seen as either harming the prostitute, or harming the patron's family. For drugs, it's often considered that the harm is focused on the user, not diffuse.

And again, lots of crimes with diffuse harm are generally agreed on to not fall into the "victimless" category, like dumping toxic materials.