Disclaimer: I'm pro-legalization.
I think the only downside of legalizing it will be increasing the number of users, and therefor the number of possible bad trips/stupid actions while high.
Being that we let ~88K die in the US due to alcohol, I think we can handle that a couple dozen misadventures from shrooms as well.