Consider substances like weed or alcohol that can ruin lives but usually don't. How should our laws treat these substances? I'm not sure but there's nothing absurd about thinking they should be illegal.
Also consider: what are laws for? To some degree they're about discouraging certain behavior. It would be nice to think that we could discourage weed/alcohol without making them illegal but it's not clear that we can do that.
Whether laws are here to protect and/or ensure prosperity of society, or whether laws are there to maximize freedom seems to be a huge source of contention in our modern society. Ultimately I think the two ideals, while overlapping many places, are incompatible. Hopefully we can rewind the consolidation of federal powers to allow people with different ideals to move to different states to achieve something closest to their desired way of life. Federal drug laws controlled by unelected beuracrats are an example of making this selection impossible.