I'm heavily hooked to cigarettes, however, walking away from caffeine was absolutely trivial. I was told to stop due to panic attacks, so I was lucky having sick leave to deal with the initial abstinence.

Long term, zero caffeine is like being on caffeine all the time. It's worth ditching.