People want to believe that the thing they like is safe, but every drug that can do anything is dangerous. That's the whole idea, a drug is a lever to make change, not a magical "make everything better" solution. You push the lever too far and you get things you don't want.
It's not just temporary psychosis (which sometimes isn't so temporary). It can trigger major changes in personality, or latent mental illnesses which tend to only manifest with a push (schizophrenia is one).
It's not a coincidence that psychedelics have been used in religious ceremonies throughout human history. They can give people religion, and without direction they can turn people into troubling zealots for just about anything (and often the drug itself). People who experienced this and became cultural icons are part of the reason they got banned. (turn on, tune in, drop out)
There is even a considerable group that want to re-name psychedelics to "entheogens" which has a rough etymology of "to put god into".
I have personally felt this and it scared the fuck out of me.