I read that book, but don't remember what he had to say about psilocybin. Could you provide a tl;dr? I only remember that Feynman consumed weed recreationally and liked to use it to enhance his creativity and thinking in sensory deprivation tanks, but he never tried LSD, fearing it would permanently damage his brain.
He tried LSD once, when it was still legal, at the urging of Dr. Timothy Leary. He then thought he'd solved a hard problem in physics he had been working on. While going to give a talk on the solution, he realized that he had not solved the problem, but had hallucinated that he had solved it. He was very angry. Never tried LSD again. "I like to think, and I don't want to break the machine", he wrote.
As a dabbler myself, it seems like this was The Classic™ "user error."
Imagine what Feynman could have produced if he had used LSD correctly! He could have been as productive as Jack Kerouac!
Instead of widening the perceptual horizons as traditionally believed, psychedelics rather seem to 'switch off' certain sections of the mind - particularly the cautious area that frets over the idea of dabbling with such compounds. Those that are household names have a knack for quieting this voice of doubt first. Paradoxically, the subjective experience of liberation is born from selective self-silencing.