Acute toxicity level 4 for Oral ingestion
https://www.ilo.org/legacy/english/protection/safework/ghs/g...
Defines Toxicity level 4 as ~ 2000 to < 5000 mg/kg
The most potent strain of "magic mushrooms" are at most ~2% by weight(Psilocybe azurescens)[0]. So according to all the de facto standard literature, a potentially fatal dose of magic mushrooms is over 3 ounces of dried mushroom per kilogram of body weight. At 68 kg I would need to ingest over 5 kg to come up to the lower bound of the LD50.
No one without an axe to grind could realistically argue toxicity is a real valid concern.
[0]https://www.shroomery.org/8700/Psilocybe-azurescens-taxonomy...