It only has that effect when they gave it to people in test trials in terrible settings like mental institution s and prisons which is why Dr Leary was adamant about set and setting
External stimuli may be one of the few contributors of the experience that has a significant temporal persistence, possibly giving it substantial influence.