Though I've never been diagnosed and live a normal life, I've had mood swings and extended family history of schizophrenia. Because of this, and the knowledge that there are certain triggers that can cause lifelong neurological issues, I'll never try acid or other hallucinogens.
The value of knowing that "underlying craziness" can exist is in inherently reminding us of the actual risk. Let's say someone predisposed toward neurological issues is on the fence, but ultimately decides to follow the reasoning in the comment. What liability are you going to bear for it?