That's a good point, and I think it is because it wasn't as powerful as other drugs. Cannabis today is far stronger than even 20 years ago if you go by old hippie lore.
We do know people ate magic mushrooms and used psychedelics all the time, and that alcohol and tobacco were used widely as soon as discovered.
Maybe that's a truly modern thing. The 420 art.