Another annoying aspect of LSD is that it's impossible to say anything bad about LSD without being jumped on by people who have made LSD part of their own identity, the same way being a Democrat is part of their identity. Try saying anything bad about either one, and you're in for a bad time.
It's possible that the silent majority simply feel that spending a bunch of time in an altered LSD state is a crazy thing to do. People claim they are profoundly different after tripping, but if you look at what they do, not what they say, they don't seem much different. They seem like pretty much the same person, if you look at their accomplishments before and after.
That's not to discount the therapeutic properties. If LSD can help someone in pain, whether physical or emotional, then it's a great tool. But that's a very different situation from what most LSD proponents seem to imply: that everyone should do it for no particular reason except curiosity. And curiosity is nice, but we should at least be willing to admit what it is: an interesting experience, the same way taking a vacation to a remote tropical island is an interesting experience. But you'll still be pretty much the same person before and after your vacation, though more relaxed afterwards.