There are (in my limited understanding) too many confounding issues to be able to say that its ok to get off your head regulars with drug X but not with drug Y.
In my (again limited) understanding, saying "all these are bad for us long-term (to wildly varying degrees), but it seems likely that other factors (childhood, mental illness, social position, economic, family and social) all play as important if not more important roles.
Lets unban them if they have some medicinal purposes, but lets not pretend that having mushrooms and amphetamines marketed by the same socially-minded people pushing Marlboros for decades is going to make the world a better place, and please lets not think that if we strike a blow for freedom, its ok to stop worrying about all those other confounding issues. (But I concede its a lot easier to legislatively fix banned psychedelics than to fix childhood abuse, poverty and mental health. But that's why we pay government the big taxes, to take on the big jobs. )