Mushrooms get this "harmless" reputation because, y'know, they're "natural" etc... There's now a big push to get them legalized somehow here in Canada, and full-on illegal stores popping up around that the police don't seem to be all that interested in shutting down.
But having been down that road... I'd say the side-effects from them were always more paranoia and reality-destructuring to me (and not usually in any kind of pleasant way) than LSD or other things I tried back when I was young and unwise.
The one advantage I suppose would be that they wear off after 4-5 hours. Not so LSD. Takes days to become normal after an LSD event. No thanks.