If you believe there are no downsides, FYI, around a century ago heroin was marketed as a "no downsides" cough suppressant. With time (very soon) the downsides became apparent. This may happen to the psychedelic you now consider safe.
The idea to alter one's mind with substances to have more pleasant / interesting / euphoric / hallucinogenic experience is vicious in principle, even if there were no other downsides.