According to David Nutts study psylocibin is regarded as one of the least harmful drugs. I have also seen a study showing that use of psychedelics was associated with lower rates of mental health issues later in life.
http://journals.plos.org/plosone/article?id=10.1371/journal....
Sounds to me like it put a crack in your ego rather than a crack in your sanity.