I believe many who have taken psilocybin for medical reasons would agree: they would never take it again. I did not find it not habit forming. (edited to make clear my opinion and not facts)
Caffeine is a mind altering experience. Eating too much sugar is a mind altering experience. Drinking alcohol is a mind altering experience. I have had extremely pleasant and also terrifying psilocybin trips. They have all been constructive experiences for me.