> I've read that it would be incredibly difficult to abuse psilocybin mushroom because humans quickly build up a tolerance. It's also impossible for a human to consume enough psilocybin to kill them.

"Abuse" has a much larger definition than daily use or fatal overdose. Someone ignorant of the effects or with poor judgment might take psilocybin while operating a gas stove, taking care of children, or operating machinery. It's also possible to take too much, take it too frequently, or take it in inappropriate settings and cause real harm to the user. All of which behaviors probably fall under substance abuse.

EDIT: I'm pro-legalization, but I think statements understating risk are just as harmful to society and statements overstating risk.