Individualism shouldn't be the only lens we look through when considering these problems.
I think debilitating substances that are potentially addictive and don't have social acclimatation (unlike alcohol) are a very high risk. I don't think they should be banned outright if they have useful properties, but I don't think they should just be allowed in society without proper care.
Just looking at it from a "I do what I want to my adult body" completely ignores the condition some of these adults will find themselves in, and the risk they will impose on others as a result. And it also completely ignores how we are all interdependent on each other in a modern economy.
We all require a village of people to function and thrive and defend our rights. Something that potentially harms a large percentage of this village will also harm those in the village who don't partake. And then when your village is helpless, a belligerent neighbouring one comes knocking.
Maybe you think that's an exaggeration, but it's literally what brought down China in the 1800s. It turned from one of the oldest and most grandiose powers on earth into an incapacitated corpse pillaged by western powers.
So you know, being careful with these substances seems just a bit prudent.